InvFusion: Bridging Supervised and Zero-shot Diffusion for Inverse Problems
Authors: Noam Elata
Hyungjin Chung
Jong Chul Ye
Tomer Michaeli
Michael Elad
Venue: NeurIPS 2025 poster ()
Keywords: Image Restoration
Inverse-Problem Solving
Diffusion Models
Primary area: deep_learning

Abstract: Diffusion Models have demonstrated remarkable capabilities in handling inverse problems, offering high-quality posterior-sampling-based solutions. Despite significant advances, a fundamental trade-off persists regarding the way the conditioned synthesis is employed: Zero-shot approaches can accommodate any linear degradation but rely on approximations that reduce accuracy. In contrast, training-based methods model the posterior correctly, but cannot adapt to the degradation at test-time. Here we introduce InvFusion, the first training-based degradation-aware posterior sampler. InvFusion combines the best of both worlds - the strong performance of supervised approaches and the flexibility of zero-shot methods. This is achieved through a novel architectural design that seamlessly integrates the degradation operator directly into the diffusion denoiser. We compare InvFusion against existing general-purpose posterior samplers, both degradation-aware zero-shot techniques and blind training-based methods. Experiments on the FFHQ and ImageNet datasets demonstrate state-of-the-art performance. Beyond posterior sampling, we further demonstrate the applicability of our architecture, operating as a general Minimum Mean Square Error predictor, and as a Neural Posterior Principal Component estimator.

InvFusion: Bridging Supervised and Zero-shot Diffusion for Inverse Problems
Noam Elata"
Hyungjin Chung" EverEx Scoul, South Korea
Technion
Haifa, Israel
noamelata@camput technion.ac. ac il
hj wwwnggeverex....mmmmmm ac ac.kr
Jong Chul Ye KAIST
Tomer Michaeli
Michael Elad
Technion Haifa, Israel
Technion Haifa, Israel
Daejeon, South Korea
jong. ye@kaist. kc.kr
tomer m@ee. technion. ac. il
elad@cs. www.neen.cnn...e ac
Abstract
Diffusion Models have demonstrated remarkable capabilities in handling inverse problems, offering high-quality posterior cettrior-sampling. -aase solutions. Despite signif- icant advances, a fundamental trade-off persists regarding the way the conditioned synthesis employed: Zero-shot approaches tion but rely on approximations that reduce accuracy. In contrast, training-based methods model the posterior correctly, but cannot adapt the degradation at test- time. Here we introduce InvFusion, the first training-bass based degradation-awe awaare posterior sampler. InvFusion combines the best of both worlds the strong per- formance of supervised approaches and the flexibility of zero-shot methods. This is achieved through a novel architectural design that seamlessly integrates the degradation operator directly into the diffusion denoiser. We compare InvFusion general-purpose
accommodate any linear degrada-
against existing general purpose posterior samplers, both degradation awaree zero- shot techniques and blind training-based methods. Experiments on the FFHQ and ImageNet datasets demonstrate state-of-the-art performance. Beyond posterior sampling, we further demonstrate the applicability of our architecture, operating general Minimum Mean Square Error predictor, and as Neural Posterior Principal Component estimator!
Introduction
52 28 have emerged as a leading class of generative machine learning Since their inception, diffusion models have gained significant traction in solving complex inverse problems, such as super-resolution (7) or in-painting (37), where the goal is to reconstruct or estimate an underlying image from partial or degraded observations.
Diffusion models 49
techniques 16 4
Diffusion models can be trained to solve inverse problems using simple conditioning framework, Although powerful, these training- based methods are typically restricted to handling a limited set of degradations per trained network, as they must learn the connection between clean images and their degraded versions. Moreover, existing training- based models cannot accept the degradation as an input and must rely on the measurements provided at test-time infer which among the possible degradations restore. This forces blind
in which the measurements supplied to the network |6 47
"Equal Contribution
Code implementation available httttp////gttt...cmm///aaaaaaaas
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Bly
InvFusion
H'y
InvFusion
H y
InvFusion
Patch in- In-aainting
Strided Motion Blur
Figure 1: Examples for posterior samples from our degradation-awann diffusion model. A single model can restore multiple degradations, such as in-painting, de de-blurring and super-resolution with high image fidelity and realism, by integrating the degradation operator into the model's 's architecture.
restoration setting, which is inaccurate when multiple degradations fit the measurements. These fundamental limitations significantly reduce the applicability of trained conditional diffusion-based methods, as no single model can accommodate a wide range of inverse problems. Conversely, zero-shot methods [10
33 51] utilize unconditional pre-trained diffusion models for solving inverse problems, taking into account the particular degradation process from which each input suffers, enabling a more precise non-blind setting. They are thus remarkably flexible in handling various degradation scenarios. However, these approaches use approximation that reduce accuracy and suffer from computational inefficiency.
11
32
Here, we introduce InvFusion, an architectural framework that combines the strengths training- based and zero-shot methodologies. Our approach is the first training-base method that can be informed of the degradation at test-time. The InvFusion architecture is conditioned on the degradation operator, enabling the network make full use of the provided measurements. Since the set of all possible degradations is too large to be used directly as model input, we propose integrating the degradation operator into the architecture's 's design, implicitly enabling the network to infer whether its intermediate features correspond to the measurements. This novel architecture allows the network adapt wide range of degradations while simultaneously maintaining the high performance of training-bast based networks. Our method represents a paradigm shift, offering more versatile, accurate, and computationally efficient approach to handling complex inverse problems.
Our network's 's architecture is inspired by unrolling frameworks [54 61 D. in which the layers mimic an optimization process. In our layers using attention mechanisms thus enable applying the degradation operator all network activations. We use HDiT [13] as base architecture, which is particularly effective facilitating operations on higher-resolution images, shown in Fig. [
integrate the degradation operator within the network's s The attention layers preserve dimensionality. and
[57 17 45
Accordingly, our method bypasses the approximations of zero-shot methods and solves degradation-ambiguin of existing training-based method, offering significant advantages in ac- curacy. computational efficiency, and flcxibility. In particular, our method can be used with any sampler designed for unconditional diffusion models [35] 36 30), including Classifier-Free Guidance and related techniques.
(CFG) 27 12
Our experiments on FFHQ (29 and ImageNet 15 evaluate various approaches for addressing multiple restoration tasks with a single trained model. InvFusion out-performs existing inverse problem solvers, both in the training-based and zero-shot categories, establishing a new of the-art (SOTA). Beyond posterior sampling, we illustrate how our architecture can also be used for Minimum Mean Square Error (MMSE) prediction, as well as for training a Neural Posterior Principal Component (NPPC) 42 predictor, enabling uncertainty quantification a wide variety of restoration tasks with single model.

---

2 Background and Related Work
2.1 Diffusion Models
generate high-quality images through sequential Gaussian noise removal process. Given a source data distribution p(xo), a forward diffusion process constructs a Stochastic Differential Equation (SDE) dx, - - + [3()dw,, where w. is standard Wiener process and (()) is deterministic function of i [0, 1 taken such that = 1, the final marginal distribution p(x1) becomes approximately a standard Gaussian. Each marginal distribution along this SDE, p(x:), can be constructed directly by adding Gaussian noise to the source data distribution, x = v (a(()xx + /11 - a())e, where a(!) = e JJ 8(s)da and is standard Gaussian random vector. To generate a sample image within this framework, one starts with sample of white Gaussian noise and employs an appropriate SDE solver [35
Diffusion Models 44
52 28
30 for the reverse SDE 4 39 53
The latter requires knowing the score Vx. x. (xg()(x;)) 52
50
53
By exploiting the connection
(il-a(t)) x, log(p(x))) (ā) X [x,] a denoising network trained to approximate E Xg [x|] can be used in place of the real score function.
between the score and the MMSE predictor [58 2
Training such network done using a regression loss,
E Wt [x - xoll
where my is the model being trained, X ~ p(xo), and w. is some weighting function.
2.2 Inverse Problems
Inverse problem solvers attempt to reverse a degradation process that corrupted an image x R.D and yielded measurements _R Many common degradations, such as those encountered in the super- resolution, deblurring, denoising, and in-painting tasks, have a linear form. These degradations are often written as y = Hx where H E RdxD and (O o is white Gaussian noise added to the measurements. Noiseless measurements can be formulated with on = 0. When attempting to design system that can handle multiple degradations, H can be considered random matrix drawn from some distribution of possible corruption operators.
A popular approach for generating reconstruction & of x based on the measurement y and on knowledge of H, is to draw from the posterior distribution, Xpust ~ (x H). Methods aiming for this solution are called posterior samplers and are the main focus of this work. Another popular approach to generating a reconstruction, is aiming for the Minimum Mean Square Error (MMSE) predictor, ÊMMSE = E [xly, H). which, as its name implies, achieves the lowest possible squared- error distortion. Obtaining an approximation of the MMSE predictor can be done by training a regression network my(y) to minimize L = ()) x||
When the degradation operator is not known, or is known but cannot be provided to the model, the relevant posterior distribution is
p(xly)= p(xly, Hogly,H)ppue H)
This posterior a weighted average of possible posterior distributions (for all possible degradations H). each weighted by the corresponding y.. This posterior may generally encompass significantly larger uncertainty regarding x. One key limitation of taining-based posterior samplers (like Palette [46]) is that they currently lack mechanism to condition the model on H. Therefore, even if H is known, it cannot be provided to the model at test time, so that the model is unavoidably tasked with solving blind restoration problem. In this work, we introduce degrallaioo-awwrre model architecture, enabling the model to incorporate information about the degradation H. This allows solving the precise non-blind restoration task whenever the degradation is known at test time.
2.3 Diffusion Restoration
In recent years, diffusion models have become the leading approach for posterior sampling. By modifying the probability distribution into a conditional one, p(x,ly), solutions to the inverse problem can be sampled using the score function of these conditional distributions. The most straightforward
3

---

approach to leverage diffusion models for posterior sampling is to alter the training to accommodate the partial measurements y 46 47.. modifying Eq.
f
Wt [x xol
(3)
where y = Hxg for some desired degradation H. It common implement the conditioning on y by concatenating y to the noisy network input H²y, y, or a similar operation that shapes the measurements back into the image dimensions can be used instead, but efficient implementations of both H and H exist for many practical degradations. Such methods typically achieve good results, and have proven to be accurate. Yet, such methods are not conditioned on the degradation
operator H. limiting them to a single or few degradations per trained model [46 47 Several works extend these frameworks to flow matching [14] and Schrödinger bridges kd Adapting to these
methods are beyond the scope of this work, but expect architecture to perform equally well for these different training regimes.
An alternative approach is to use an existing unconditional model mo t) and knowledge of H to model the conditional score Xx, log(p(x;ly H)), using the Bayes rule
Vx, log(p(x,ly, H)) Vx, log(p(x.)) + Vx, log(p(y)xe Xx, H)).
Such methods are referred as zero-shot [ 10 33
51 59 60]. for their use of pre-trained 32
diffusion models to solve inverse problem tasks. These methods are highly flexible, for a single trained model can be used to solve any inverse problem for which Vx. log(p(y/xx, H)) can be approximated. Yet, these methods are often inaccurate, slow, and computationally expensive, due to the challenge of obtaining the correctness guarantees of training- based models [5], zero-shot models cannot sample from the posterior with ideal denoiser [22
accurate approximation for Vx, log(p(y|xx, H)). In contrast to
2.4 Algorithm Unrolling
Algorithm unrolling, also known as deep unfolding, represents a paradigm where iterative algorithms inspired by classical optimization utilize repeated application of neural networks. This approach, first introduced in Gregor and LeCun (2T]. had emerged into image processing [54, [ 40). In inverse problem solving, many algorithm unrolling methods apply H and H (or H between network evaluations, and are trained in an end-to-end manner. Deep unrolling algorithms are typically trained for distortion reduction and do not accommodate posterior sampling, unlike our method, which is diffusion model. Also, while the InvFusion architecture is inspired by the same principles, it uses very different execution. Instead of approximating an iterative algorithm, InvFusion uses the attention mechanism to learn where to utilize the knowledge of the degradation.
2.5 Attention for Conditional Generation
originally developed for language processing tasks, have emerged dominant architectural paradigm in image processing applications. The fundamental operation in attention layers involves matching queries and keys, enabling the network identify and combine corresponding features across different data streams, and output relevant values. This architectural approach has found widespread adoption in image generation tasks, particularly in conditional generation scenarios. A prominent example is text-to-image synthesis 47). where cross-attention mechanisms facilitate interaction between textual inputs and internal diffusion model activations.
Attention mechanisms [57].
The versatility of attention mechanisms extends beyond text-to-image applications, demonstrating significant utility in various domains, such as image editing 25 56 9 novel-view-synthesis [55] 18
48 and many more.
3 Method
The integration of degradation operators directly as input to neural networks presents significant computational and architectural challenges. The complexity arises from the vast dimensionality of potential degradation representations. Even when considering only linear degradations, each row of
Ht is the Moore Penrose pseudo-inverse of the operator H.

---

the degradation matrix is as large the input image itself, creating an overwhelming input space that would render traditional network architectures computationally intractable and practically infeasible.
To address this fundamental challenge, our approach incorporates the degradation operator within the network architecture itself. We introduce novel Feature Degradation Layer, which applies the degradation operator to the internal network activations, and compares the result to the provided measurements, as shown in Fig. 2
Specifically, in this layer, feature representations undergo the following operations: deep features first rearranged in the shape of stack of images, to which the degradation is applied. These degraded features are then concatenated with the measurements, and are transformed with a learned linear operator, incorporating the information from both the measurements and the degraded features. Finally, the transformed degraded features mapped back to the feature space through the degradation's 's pseudo-inverse. This process can be described formally as
Muttihead Joint Altention
InvFusion Block
(Taaaa y)),,
(5)
H
Feature Degradation Layer
v
kv
where Dxe are the inputs the layer, and To R(++1) + R is learned trans- form and activation operating on the chan-
Feature Degradation Layer
Concat
nels axis
H
We interweave the output of our Feature Degradation Layers back into the network's processing stream using joint-attentior fl as illustrated in Fig. 2 These complete blocks, which we name InvFusion Blocks, are de- signed to replace some or all standard at- tention blocks in existing Transformer or UNet architectures. The above operations require that the network's 's deep feature res- olution match the input image dimension D. This constraint is readily accommodated in many Vision Transformer 17 43 3
Activations
y
Activations
Figure 2: A diagram of the InvFusion Block. Our block contains Feature Degradation Layer, which in- corporates the operator H into the architecture by ap- plying H on the activations and comparing them with the measurements y Hx.
based networks by using number of hidden channels that is divisible by the original image channels (which does not compromise performance). Internal network activations can then be un-patched back to the original image shapes to apply the degradation H.
Finally, using our complete InvFusion architecture, diffusion model can be trained directly by conditioning on both the degradation and the measurements,
C E [|mo L, H) xol1?
(6)
where H is sampled from some pre-determined distribution over possible degradations. Additionally, also find it is beneficial to concatenate Hty to the noisy network input.
Sampling from InvFusion maintains fundamental compatibility with standard diffussin model sam- pling procedures, offering significant advantage in terms of flexibility and implementation. This compatibility means that any established sampling technique developed for conventional diffusion models can be directly applied to InvFusion.
3.1 MMSE Estimator
Our InvFusion architecture can be used for more than posterior sampling. Having an architecture that makes use of the degradation H enables training a single model with
= E [|me (y, H x]]]
(7)
to produce an accurate MMSE predictor for many different degradations. Alternatively, the MMSE predictor can be approximated by diffusion model trained with Eq. (6) by applying single denoising
Jointt attentiot differs from cross-atenttin by using the same queries for both the "self" and "cross" keys
5

---

Table 1: Comparison of inverse-problem solving methods FFHQ64. Best method in bold.
Strided Motion Blur PSNR FID CFID
In- In-Painting
Matrix Operator PSNR1 FID CFID.
NFEs
Method
PSNR FID CHID
Zero-shot Methods
DDRM 33
22.5
23.2
23.2
12.6
53.6
53.6
10.8
72.7
72.7
25
DDNM 59]
24.0
42.5
42.3
16.2
54.6
54.6
11.9
131.4
131.4
100
DPS I
18.3
9.7
17.7
10.1
17.0
67.5
9.6
26.7
35.9
1000"
HIGDM
51
22.1
6.3
6.3
15.2
12.8
12.8
10.7
19.1
19.1
100"
DAPS 60
17.5
143.5
60.3
16.3
16.9
19.1
12.0
56.3
87.2
1000
MGPS
20.5
7.7
9.7
13.9
10.5
24.3
11.7
8.2
29.0
1764
Training- -based Methods
Palette 46
21.0
4.7
5.8
17.0
5.3
5.5
12.5
6.6
25.5
63
InDI [14]
21.4
21.6
22.8
17.7
22.6
25.0
13.7
35.5
58.7
50
InvFusion
22.7
4.2
4.2
17.1
5.0
5.0
15.1
7.1
7.1
63
MMSE predictor (Training-based)
Palette [46]
23.5
19.5
15.5
1
InvFusion 25.0
19.8
17.9
Table 2: Comparison of inverse-problem solving methods on ImageNet64, Best method in bold.
Strided Motion Blur PSNR FIDI CFID
In-Painting PSNR FID CFID
Matrix Operator PSNR FIDJ CFID
NFEs
Method
Zero-shot Methods
DDRM 33
21.2
26.1
26.2
12.2
30.5
30.5
10.6
32.9
32.9
25
DDNM 55
22.5
69.5
69.4
15.1
95.3
95.3
11.3
115.3
115.3
100
DPS |I|
18.3
9.0
12.9
10.4
15.2
17.7
9.5
12.8
12.8
1000"
IIGDM
51
20.0
7.1
7.1
14.3
8.7
8.7
10.6
8.9
8.9
100"
DAPS 60
17.3
125.7
62.8
15.4
16.3
19.7
11.7
28.3
37.0
1000
MGPS
19.4
14.2
18.0
13.3
18.5
25.0
11.7
13.8
18.9
1764
Training-brid based Methods
Palette 46
19.6
6.6
7.0
15.6
6.2
6.2
12.3
6.7
11.4
63
InDI [4]
20.6
23.3
24.6
17.1
21.0
21.0
14.5
30.5
40.0
50
InvFusion
20.9
5.6
5.6
15.6
5.9
5,9
14.5
6.3
6.3
63
MMSE predictor (Training-basse
Palette [46]
22.3
18.3
15.3
InvFusion 23.4
18.4
17.3
1
Methods that use the network derivative, using more computation and memory per NFE,
step on white Gaussian noise. That is, for sufficiently small a(t) we have that x; R e, which is independent of Xg, so that
E. [xo y, H = E (xoly, 1 H).
(8)
which also holds for the approximate models being trained. In other words, any conditional diffusion model can be used as an efficient MMSE predictor. We find that this works well (see App. B) and
utilize this in our experiments.
Experiments
We evaluate the advantage of InvFusion over existing inverse problem solvers on 64 x 64 images from the FFHQ (29] and ImageNet [15] datasets. In these experiments, we conduct comparative analysis of InvFusion against alternative methods that operate several types of degradations

---

with single model. This includes zero-shot methods. utilizing an comparable unconditional model. Additionally, we evaluate our approach against a "Palette" -style methodology [46 that incorporates the Hly as input without additional degradation information. We train these models ourselves using the same underlying architecture, to enable a comprehensive assessment of our method's capabilities disentangled from architectural considerations. The models trained on ImageNet are also class-conditional, and use CFG [27] with a factor of 2.0 for sampling.
We train and evaluate our model on several categories of inverse problems; strided motion blur (com- bining various degrees of super-resolution and de- blurrrigg), in in-painting by leaving only patches of various sizes, and general matrix degradation operator. These degradations each have computation- ally efficient implementations for applying both H and H The parameters of each degradation are randomized, to create large corpus of possible degradations (as detailed in App. [A]. All conditional models are also trained on unconditional generation by using a degradation which always outputs zeros. Figure [ shows qualitative comparisons of different inverse problem solving methods for the two datasets. All training- based approaches use the deterministic sampler from EDM 30 along with
identical seeds, highlighting the effects of different models.
For quantitative comparisons, we use PSNR measure the image fidelity, i.e. how close the restored output is to the original. To measure image realism, we use 10K FID [26]. These two metrics are at odds according to the Perception- Distortion- Perception-Distortion T [ and different methods excel at either of the metrics. We find that some restoration algorithms seem to sacrifice consistency with the measurements y to accommodate better image realism, in effect ignoring the inverse problem. To measure the image realism of valid solutions to the given inverse problem, we offer to measure the FID of generated samples that have been projected to be consistent with the measurements y: which we refer to as CFID. In this way, we penalize models for the distance between the generated sample and the nearest valid solution to the formulated problem. We also include LPIPS [62) measures in App. D Tables D and 2 show our qualitative analysis on the FFHQ [29] and ImageNet [5] datasets respectively. Our model achieves the best CFID among all training- along with the best FID in all but a single case, suggesting it is SOTA in generating samples from the posterior for our setting. This superior quality is attained despite using far less computations than nearly all zero-shot methods, as quantified by Neural Function Evaluations (NFEs). Our InvFusion MMSE estimator based on the same model achieves the best PSNR of all models, while the InvFusion posterior sampler is also among the highest in PSNR.
and zero-shot methods,
Interestingly, the gap in performance between InvFusion and Palette highly varies between different degradation families. In problems like in-painting, the degradation is easily inferrable by the network, as the H'y input to the network contains masked regions (see Fig. 3). Thus in in in-painting, we see a small advantage for InvFusion over Palette. On the other hand, in motion-blur and even more so on general matrix degradations, Palette struggles to infer the exact degradation, leading to drop in both PSNR and CFID, sometimes, even lower than some zero-shot methods which are degradation-awar aware. This supports our hypothesis that knowledge of the degradation operator, whether inferred or explicit, is critical for correct restoration.
Previous comparisons focused on smaller images (64 x 64) to enable efficient evaluations of many methods and degradations, some of which are computationally expensive. To demonstrate InvFusion's 's scalability and performance at higher resolutions, we conducted additional experiments on strided motion blur and patch in-painting using 256 x 256 images from FFHQ The complete results are presented in App. D with representative examples shown in Fig.
4.1 Out-Of-Distributiio Degradations and Unconditional Sampling
To measure the generalization capabilities of different methods we evaluate performance on degradation that is slightly outside of the training distribution. In this case, we perform out-painting from a single rectangular area in the image. The results in Tab. 3 show that Palette completely fails to generalize, despite the similarity to the in in-painting task the model has been trained on and the ease at which the degradation operator can be inferred from the input. On the other hand, InvFusion does quite well on this problem. outperforming all zero-shot methods, which are not penalized by the -uut----iitiiittion (OOD) degradation, as they are not trained on any degradation. Notably, the matrix Taas._aadda
degradations demonstrated in Tabs. [nd underssorre the generalization capabilities of InvFusion, as
the range of possible degradations is so huge that near certain that none of the degradations used in test-time have appeared in training. Finally, also evaluate all trained models for unconditional
7

---

Zero-Shot
Training-Based
Hl y
DDRM
DDNM
ΠGDM
DAPS
Palette
InvFusion
GT
Strided Motion Blur
Patch Pa--tnni
Matrix Degradation
Not Seen Training
Figure 3: Examples comparing zero-shot and training- based inverse problem solvers. For each degradation, the top row is an example from FFHQ64 and the bottom row from ImageNet64. Images generated using training-based methods use deterministic samplers and identical seeds, highlighting the subtle effects the different training algorithms.
generation (on which they had been trained, by using H E ROxD). Surprisingly, we find that both conditional models outperform the unconditional baseline, with InvFusion leading with approximately 5% improvement over the baseline (unconditional diffusion). This is despite the conditional models having no additional capacity in unconditional generation, as both the additional model inputs and the Feature Degradation Layer are always output zeros in this setting. We hypothesize that being exposed to partial image information during training helps the diffusion model converge. This indicates that single InvFusion model can be well adapted for both restoration and unconditional generation.
4.2 Noisy Degradations
Following the noiseless linear case explored in earlier sections, we also conduct experiments on noisy inverse problems. We retrain the FFHQ64 model, incorporating additive white Gaussian noise with standard deviation of a, 0.1 in the measurements unchanged, using only the transforms noise at each layer could lead to accumulation of noise. Projection operators not well-defined for noisy degradations, so only PSNR and FID are evaluated for the noisy case. Our findings in Tab. [T] (App. D) demonstrate the advantage of InvFnsio over alternative methods in the noisy settings.
The InvFusion architecture remains and H of the degradation as adding randomly sampled

---

Table 3: Comparison of restoration on degradation that did not appear in train- ing. InvFusion demonstrates strong adap- tation capabilities through its degradation- aware architecture.
InvFusion NPPC
H
MMMSE
4w
2w
*
2w
g
Method
PSNRT
FIDI CFID
Zero-shot Methods
DDRM
11.6
24.9
24.9
DDNM 59
12.7
75.8
75.8
DPS I
10.1
8.8
14.1
IIGDM
51
11.3
7.3
7.3
DAPS
60
12.1
40.8
39.5
InvFusion
Training-basee based Methods
NPPC
Palette 46
11.0
19.9
19.9
MMSE
InvFusion
11.9
5.9
5.9
MMSE predictor (Training-basse
Palette 46
13.6
InvFusion 14.7
x- Aww
X 2w5
8
X 2.2
X fw-ww;
Table 4: Unconditional sampling FID.
FFHQ
ImageNet
Figure 4: Using InvFusion NPPC. By being degradation-awart the model can be trained to pre- dict the MMSE along with several leading principal components w; (left, contrast enhanced) for many degradations.
Baseline [3
6.66
8.98
Palette 46
6.56
8.26
InvFusion 6.36
8.17
Neural Posterior Principal Components
Beyond the improvements in restoration fidelity and realism, using an operator-aware architecture unlocks several capabilities that are useful for down stream applications. A notable example, is the computation of Neural Posterior Principal Components (NPPCs) 422. which are efficient estimations of the posterior distribution's 's largest principal vectors. These direction are meaningful for uncer- tainty quantification among many other tasks 38 [20 and may have high implications for medical analysis [6 I9]. In the original work by Nehme al. [42), network is trained to approximate the posterior principal components for single pre-defined degradation. Using InvFusion, we can use the same approach to scale up NPPC training to multiple degradations at once, enabling flexible adaptation to the degradation at runtime. We experiment with applying the exact same NPPC loss from [42), using the InvFusion architecture to condition the network the degradation operator H for the in-painting of a single area of the image. The examples in Fig. shows that InvFusion meaningfully extends NPPC computation for multiple degradation, making single model viable for a wide range of problems.
Discussion, Limitations and Conclusion
Despite achieving state-of-the-art performance in posterior sampling and MMSE estimation, Inv- Fusion faces several important limitations. A primary constraint lies in the model's training scope although InvFusion demonstrates remarkable adaptability to various degradations, can only be trained on finite set of degradation types, beyond which the model may under uuder--erform. Even in the linear case, the space of possible degradations scales quadratically with image dimensions, creating fundamental constraints that would be impractical to address solely through increased model capacity and training time. Additionally, while our current results focus exclusively on linear degradations, we anticipate that InyFusion could potentially generalize non-linear degradations when appropriately

---

trained, though this remains to be empirically validated. This in turn would enable the use of InvFu- sion for latent diffusion, in which non-linear operators apply or approximate the encoder and decoder. A final consideration is the computational and memory intensity of repeated degradation applications, particularly during the training phase. Although this computational overhead may be relatively minor compared to certain zero-shot methods that require additional steps model derivatives, optimizing these resource requirements represents a key area for future improvement.
This work introduces InvFusion, novel architectural framework that bridges the long-standing gap between training-basse based and zero-shot approaches in solving inverse problems with diffusion models. By incorporating the degradation operator directly into the network architecture through the attention mechanism, our method achieves state-of-the of-the-art performance while maintaining the flexibility to handle diverse degradation scenarios. The empirical results demonstrate superior performance on multiple datasets and across various inverse problems, while offering computational efficiency comparable to many existing methods. Beyond its primary application, InvFusion's 's capability as general MMSE estimator and its potential for NPPC estimation opens new avenues for downstream applications. While certain limitations remain, this work represents significant step forward in the field accurate and efficient inverse problem solving.
Acknowledgments and Disclosure of Funding
This research was partially supported by the Israel Science Foundation (ISF) under Grants 2318/22, 951/24 and 409/24, and by the Council for Higher Education Planning and Budgeting Committee.
This work was supported by the Institute of Information Communications Technology Planning & Evaluation (ITP) grant funded by the Korea government (MSIT) (RS-2025-02304967, Al Star Tellowship(KAIST), and by the National Research Foundation of Korea under Grant RS-2024- 00336454.
References
[1] H. K. Aggarwal, M. P. Mani, and M. Jacob. Modl: Model-based deep learning architecture for inverse problems. IEEE transactions on medical imaging, (8(2):3944405) 2018.
[2] G. Alain and Y. Bengio. What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15(1):3563-3593 2014.
[3] Alkhouri, S. Liang, Bell, Q. Qu, R. Wang, and S. Ravishankar. Image reconstruction via autoencoding sequential deep image prior. Advances in Neural Information Processing Systems, 37:18988-19012, 2024.
[4] B. D. Anderson. time diffusion equation models. Stochastic Processes and their Applicationss 13(3):31--326, 1982.
[5] G. Batzolis, J. Stanczuk, C.-B. Schönlieb, and C. Etmann. Conditional image generation with score-based diffusion models. arXiv preprint arXiv:2111.138 2021.
[6] O. Belhasin, Y. Romano, D. Freedman, E. Rivlin, and M. Elad. Principal uncertainty quantifi- cation with spatial correlation for image restoration problems. JEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[7] M. Binkowski, J. Sutherland, M. Arbel, and A. Gretton. Demystifying mmd gans. arXiv preprint arXiv: 1801. 01400,111111 2018
[8] Y. Blau and T. Michaeli. The perception-distortion tradeoff.. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6228-6237, 2018.
[9] M. Cao, X. Wang. Z. Qi, Y. Shan, X. Qie, and Y. Zheng. Masactrl: Tuning-free mutual self- attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22560-22570, October 2023.
[10] H. Chung, Sim, D. Ryu, and Ye. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems, 35.25683-25666, 2022.
[11] H. Chung, J. Kim, M. T. Mecann, M. L. Klasky, and J. C. Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations, 2023. URL https:///peenrevier net wttp://wwww....oooooeooeeeooooo.
10

---

[12] H. Chung, J. Kim, G. Y. Park, H. Nam, and J. C. Ye. Cfg++: free guidance for diffusion models. arXiv preprint arXiv:2406.08077 2024.
classifier
[13] K. Crowson, S. A. Baumann, A. Birch, T. M. Abraham, D. Z. Kaplan, and Shippole. Scalable high-resolution pixel-space image synthesis with hourglass diffusion transformers. In Forty-first International Conference on Machine Learning, 2024.
[14] M. Delbracio and P. Milanfar. Inversion by direct iteration: An alternative to denoising diffusion for image restoration. arXiv preprint arXiv:2303. 1135,, 2023.
[15] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hicrarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[16] P. Dhariwal and A. Nichol. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[17] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani. M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16
words: Transformers for image recognition at scale, 2021. https:://///// //tp///xxvvv///////////////aaava org/ abs/
2010. 11929
[18] N. Elata, B. Kawar, Y. Ostrovsky- Berman, M. Farber, and R. Sokolovsky. Novel view synthesis with pixel-space diffusion models. arXiv preprint arXiv:241 1.07765, 2024.
[19] N. Elata, T. Michaeli, and M. Elad, Adaptive compressed sensing with diffusion-based posterior sampling. In European Conference on Computer Vision, pages 290-308. Springer, 2025.
[20] N. Elata, T. Michaeli, and M. Elad. Psc: Posterior sampling-base compression, 2025. URL
https://arxiv. org/ ab4/ 09896
[21] K. Gregor and Y. LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th international conference on international conference on machine learning, pages 399-406, 2010.
[22] S. Gupta, A. Jalal, A. Parulekar, E. Price, and Z. Xun. Diffusion posterior sampling is computa- tionally intractable. arXiv preprint arXiv:2402.1277) 2024.
[23] A. Hassani, S. Walton, J. Li, S. Li, and H. Shi. Neighborhood attention transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[24] A. Hassani, W.-M. Hwu, and H. Shi. Faster neighborhood attention: Reducing the o(n" 2) cost of self attention at the threadblock level. In Advances in Neural Information Processing Systems, 2024.
[25] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 2022.
[26] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems, volume 30, 2017.
[27] J. Ho and T. Salimans. Classifier-fre free diffusion guidance. arXiv preprint arXiv: 125988. 2022.
[28] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, $3:6840-6835 2020.
[29] T. Karras. A bayee generator architecture for generative adversarial networks. arXiv preprint arXiv: 049448, 2019.
[30] T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the design space of diffusion-bart based generative models. Advances in neural information processing systems, 35:26565 -26777, 2022.
[31] T. Karras, M. Aittala, J. Lehtinen, Hellsten, T. Aila, and S. Laine. Analyzing and improving the training dynamics of diffusion models. arXiv preprint arXiv:2312.02699 2023.
[32] B. Kawar, G. Vaksman, and M. Elad. Snips: Solving noisy inverse problems stochastically. Advances in Neural Information Processing Systems, 34:21757-21769, 2021.
[33] B. Kawar, M. Flad, Ermon, and J. Song. Denoising diffusion restoration models. In Advances
in Neural Information Processing Systems, 2022.

---

[34] G.-H. Liu, A. Vahdat, D.-A. Huang, E. A. Theodorou, W. Nie, and A. Anandkumar. I2sb: image-to-image schrödinger bridge. In Proceedings of the 40th International Conference on Machine Learning, pages 22042-22066 2023.
[35] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. Dpm-solver: A fast ode solver for diffusion probabilisticc model sampling in around 10 steps. arXiv preprint (xXiv:2206600927. 2022.
[36] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li. and J. Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint wrrXv:22111010990 2022.
[37] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool. Repaint: Inpainting using denoising diffuuson probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages I 11461-11471, 2022.
[38] H. Manor and T. Michaeli. On the posterior distribution denoising: Application to uncertainty quantification. arXiv preprint arXiv:2309. 13598, 2023.
[39] D. Maoutsa, S. Reich, and M. Opper. Interacting particle solutions of fokker-planck equations through gradient-log-dense
estimation. Entropy, 22(8):802, 2020.
[40] V. Monga, Y. Li, and Y. C. Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine, 38(2):18-44 2021.
[41] B. Moufad, Janati, I.. Bedin, A. Durmus, R. Douc, E. Moulines, and J. Olsson. Variational diffusion posterior sampling with midpoint guidance. arXiv preprint arXiv:2410.0994 2024.
[42] E. Nehme. O. Yair, and T. Michaeli. Uncertainty quantification via neural posterior principal components. Advances in Neural Information Processing Systems, 36, 2024.
[43] W. Peebles and S. Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
[44] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pages 10684-10695, June 2022.
[45] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffuusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
[46] C. Saharia, W. Chan, H. Chang, C. Lee, Ho, T. Salimans, D. Fleet, and M. Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[47] C. Saharia, Ho, W. Chan, I. Salimans, J. Fleet, and M. Norouzi. Image super-resolution via iterative refinement. IEEE transactions on pattern analysis and machine intelligence, 45(4): 2022.
[48] Seo, K. Fukuda, T. Shibuya, T. Narihira, N. Murata, S. Hu, C.-H.Lai S. Kim, and Y. Mitsufuji. Genwarp: Single image to novel views with semantic-preserving generative warping. arXiv preprint arXiv:2405. 17251, 2024.
[49] J. SShl--Dicsstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[50] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In International Conference on Leurning Representations, 2020.
[51] Song, A. Vahdat, M. Mardani, and J. Kautz. Pseudoinverse-guided diffusion models for inverse problems. In International Conference on Learning Representations, 2023. URL www.:aareiieei.eteooomuuiiiiioio
[52] Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
[53] Y. Song, J. Gohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[54] Sun, H. Li, 7. Xu, et Deep admm-net for compressive sensing mri. Advances in neural information processing systems, 29, 2016.
12

---

[55] H.-Y. Tseng, Q. Li, C. Kim, S. Alsisan, J.-B. Huang, and J. Kopf. Consistent view synthesis with pose-guidee gose-guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16773-16783, 2023.
[56] N. Tumanyan, M. Geyer, S. Bagon, and T. Dekel. Plug-and-play diffusion features for text- driven image-to-image translation. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1921-1930, June 2023.
[57] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and Polo- sukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Sys- tems, volume 30. Curran Associates, Inc., 2017. URL https:htt////sotteeeeet
wwwrips cc/
paper foper_file//ee
pdf
[58] Vincent. connection between score matching and denoising autoencoders. Neural compu- tation, 23(7):1661-1677 2011.
[59] Y. Wang. J. Yu, and J. Zhang. Zero-shot image restoration using denoising diffusion null-space model. The Eleventh International Conference on Learning Representations, 2023.
[60] B. Zhang, W. Chu, J. Berner, C. Meng, A. Anandkumar, and Y. Song. Improving diffusion inverse problem solving with decoupled noise annealing, 2024. http:://s///rxx org/
0bs/2447.01522
[61] J. Zhang and B. Ghanem. Ista-net: Interpretable cenpaiiiaationnnsiiceee deep network for image compressive sensing. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1828-1837, 2018.
[62] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features perceptual metrie. In CVPR, 2018.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The paper introduces a novel method for inverse-problem solving by condition- ing diffusion model on the degradation operator. This work claims that models trained with this method will have improved results over other inverse-probles solving methods with comparable architecture, well enable the operation on wide range of degradations with a single model. The experiments Sec. 4 provide evidence that these claims hold.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include the limitations in Sec.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Our theoretical analysis includes our assumptions, our formulas are numbered, and our paper does not include proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our experiment setup, model architecture, training and inference hyperparam- eters are clearly defined in Sec. and App. A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
15

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide details and pseudo-code implementation of our architectural changes in the appendix (App.A). We also provide a code implementation in the supple- mentary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
mniissccc/uublic//ee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details provided in the appendix (App. A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: reporting results, we have been following existing literature in this domain, evaluating various methods and metrics over 10000 images and reporting aggregate scores. We use multiple settings for redundancy.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details are provided in the appendix (App. A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :ttpp:::///euurtee nwwrips....../. weeeurrps.cppuuli//Ettitt cc.ppulicetthiiccciiee.........e
Answer: [Yes]
Justification:
This research conforms with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper does not have any socictal impact beyond the broad implications of improving image restoration methods.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The assets used in the paper are properly credited, the licensces included in the appendix when available, and terms of use are followed.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper submission contains the implementation code which is documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This research does not involve LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //appr////eurpp..... http//rrrpsscccboooeeeee/e////// Cc/Coonferences/2225//LI. for what should or should be described.
20

---

Appendices
A Implementation Details
A.1 Architecture and Model Training
Code implementation available
Our experiments data from the FFHQ [29 (CC-BY 4.0 license) and ImageNet datasets. We implement all models using the official implementation of HDiT 113 orrerraannsormeeeeoiisereeooooo
[5
architecture (MIT license). We use default hyperparameters, with changes listed in Tab. [S] The initial patch size is indicated in the table, with the patch size doubling with the progression along the list shown in the 'Depths' column. The attention type column signified which type of attention was used. 'NAttnr' indicates neighborhood attention [23 [24 with a kernel of size 2, which performs attention between patches only an sized neighborhood. 'GAttn' indicates global attention. As described in Sec. 3 InvFusion requires that all layer widths are divisible by the input channels, which this case resolution must be multiple of the input dimension D. In our experiments, this held for all but the deepest layers in the FFHQ256 model. For this reason, we not apply InvFusion to these layers, and used the baseline unconditional attention implementation instead. This provides evidence that our InvFusion architecture works even when not applied evenly on all the network's attention layers. In preliminary experiments, we found joint-attention, in which keys and values from the feature degradation layer are concatenated with the keys and value in the main (self) stream, is preferable to applying self-attention followed by cross-attention.
Also each layer's width times the layer's feature map
Models are trained and evaluated with the official implementation of the EDM2 training script (licensed as CC BY-NC-SA 4.0), using the default P P_mean -0.8, P_std 1.6, and a learning rate of 5, using the default learning rate scheduler. Sampling done with the default Heun scheduler using total of 63 NFEs for sampling. Additional experiment specific-traini hyperparameters can found in Tab. The augmentation implementation and hyperparameters are cloned from EDM [30]. We have used 8 Nvidia A40 GPUs (or equivalent hardware) with 49GB of memory for all experiments. Training our model takes approximately one day for the FFHQ64 models, two days for the FFHQ256 model, and four days for the ImageNet64 model.
A.2 Pseudo-Code
A.2.1
Feature Degradation Layer
The Feature Degradation Layer is implemented with the following steps, as described below; The activations un-patched into the image shape, after which the degradation is applied to them. Next, the measurements are concatenated to the degraded features, a linear function and an activation are applied, and the -nvuereivverse degradation takes the degraded features back into the images space.
Table 5: Architecture Hyperparameters
Experiment
Patch
Depths
Widths
Attention Type
FFHQ64
(2,2)
2, 2,88
[384, 768, 1536 (384, 768, 1536 [192, 384, 768, 1536)
NAttn7, NAttn7, GAttn] NAtt77, NAtt7, GAttn
ImageNet64
(2, 2)
2, 2,
FFHQ256
(4,4)
NAtt7,,
NAtn7, GAttn, GAttn]
Table 6: Training Hyperparameters
Experiment
Iters
Batch
Aug Prob
Label Dropout
Mixed Precision
FFHQ64
226
512
0.12
fp16
ImageNet64
218
512
0.0
0.1
fp16
FFHQ256
218
256
0.12
bfl6
21

---

class FeatureDsgradation (un Module):
def --imit_- (self, chanmels, patch_size, im_channels=3)
super () -..ii___
self.h patch_size(0) [0]
self.w batch_size(i]
self. channells im_channels
self self.deg deg_linear Linear Minear(channel (channels 1, channels, bias=True)
def forward x, degradation, y):
x rearranne (x,
10
"... (nh c)
(h nh) (v nw)",
nh=self.h, aw=self.,, c=self channell)
12
-y He)
13
-y carch,cat(ly,, -yl, -2)
14
-y act deg_linear (_y))
_I EegradatiionH_piiee (-))
_I (_arrange
17
k c (h nh) (v nz) -> h (nh nw k c)",
18
nh-self.he h,, nw-self.w)
19
return
A.2.2 Inv Block
Below is the implementation an InvFusion block applied on global attention. The input x is projected to create queries, values and keys, written as q, k, respectively. The conditional input y. which is the output the Feature Degradation layer, is also projected to make additional keys and values, k', v', which are concatenated to the previous keys and values before the attention is computed. We find that projecting both x and y to produce is beneficial, as the query key matching may be harder to learn solely on the outputs of the Feature Degradation layers. The neighborhood attention of an InvFusion block is similar, using the native neighborhood attention [23 implementation to compute the attention itself.
class SelfAttentionBles (nn. Module):
def __init__. (self, d, d_head, cond, dropout=0.0, joint=False # Whether to apply InvFusion in this layer
):
super (in
self dead d_head n_hads = d_head
self norm AdaRMSNorm (d, cond)
self.jnora AdaRMSNorm(d, (d, cond) if joint else None
self velf.qkv_proj Linear (d,
3, bias-False)
self.jk.proj Linear
2, d, bias=False) if joint else None
11
self.jv_proo = Linear d, bias=False) if joint else None
12
Parameter (torch full (aaell(...nneeeaeeeleee n_hedss), 10.0))
self.pos_emb AxialRoPE (d_head // 2, self n_haads) self dropout nn Dropout (dropout)
15
self selppout_proj zero_init (Linear (d, d, bias-False)
16
17
def forward x, theta, cond, y=None)
18
skip
19
x self norm cond) qkv self qkv (x)
20
21
9, k, v rearrange (qkv, h (t nh e) t nh (h w) e",
22
t=3, aiself.d_heal
24
9, scale(q, k, self scalels None, None], 1e-6)
25
q = rpply_rotary_emb
theta)
26
k apply_rotary_emb(tt, theta)
if self self. self.jk.proj is
None: # Enter into InvFusion block y self jnorm cond)
29
jk selff.. folf.jk_pros jt.proo(torch jv self jv-projy)) jk, (rearrange
(torch. cat ([x, y], -1))
30
31
22

---

"n h (nh e) -> nh (h w) e",
e=self d_head) for in (jk, jr))
jk scale(q, jk, self solfsscaell, None, Nonel, 1e-6)
35
jk apply_rotary_emb_ (jk, theta)
36
k - torch ([k,, jkl, -2) torch. cat jvl, -2)
37
wttpdaooooooctttttttcoooeeeo x rearrange
F.
k, v, scale=1.0)
39
40
"n nh (h w) -> n ahe (nh e) h=skip. shape [-3], w=skip shape (22)) self dropout (x)
41
42
x self out_proj (x)
44
return x skip
45
A.3 Degradations
Below explain the implementation details of each family of degradations used in the paper. Several of the degradations include some form of zero- zero-padding to rows the degradations H (in its matrix form) and therefore the measurements y for practical reasons. This is done such that different degradations can be applied in batched form. This produces identical results, as the linear transform T. operates the channel dimension only, and H would similarly have zero- arro-padded columns.
A.3.1 Patch In-Painting
Patch in-painting is implemented any in in-painting mask where (0, 0.1) of patches of some unvarying size are visible. When creating a new mask, the patch size, and visible patches are randomly sampled until valid mask in which some patches are visible created. We perform this "rejection sampling" to remove masks in which no patches are visible.
The pseudo-inverse of masking operator is itself, thus the computation of Ht is trivial. In practice, we retain zeros in the masked measurements y. which is equivalent computationally, to enable different masks with different number of masked pixels across the batch dimension.
A.3.2 Strided Motion-Blun
This family of degradations implements a strided convolution operator, for which we specifically choose motion-blur kernel. To create a new degradation operator, first sample motion-blur kernel the size of the input image using the motionblur library We then sample a stride size from [2, 4, 8] and smooth the motion-blur kernel by convolving it with isometric Gaussian kernel with standard deviation equal to the stride. The kernels are normalized for numeric stability.
The strided convolution is applied in the frequency space for efficient computation, as this is more efficient for large convolution kernels and simplifies the computation and application of the pseudo- inverse operator. At runtime, the given image or activations are transform to the Fourier domain, where they are multiplied with the pre-computed frequency-space operator.
A.3.3 Matrix Degradation
The matrix degradation is a matrix operator with 128 rows, each which is sampled from a multivariate normal distribution and normalized. The number of rows is limited enable the of different matrices across the batch dimension, which is more computationally intensive. The pseudo-inverse operator is computed directly with pytorch's p_inv function.
As any degradation can be represented by matrix, this family represents all possible degradations in theory. Yet, we find that model trained only randomly sampled matrices as described here does not generalize well to specific tasks such as in- ininnining, super-resolution, or de- blurring.
A.3.4 Box Out-Painting
This degradation is similar to the patch in hh-painting, but instead of masking all but several patches the box out- out-painting leaves only a single rectangle of the image visible. To create this degradation
23

---

Table 7: Comparison of MMSE training and diffusion models used for MMSE.
Strided Motion Blur
In-Painting PSNR
Matrix Operator
Architecture & Loss
PSNR
PSNR
Trained for MMSE (Eq. 77)
Palette [46] InvFusion
23.08 24.66
19.36 19.79
15.63
17.92
Trained For diffusion (Eq. (6)
Palette A6
23.47
19.65
15.51
InvFusion
24.83
19.79
17.60
uniformly sample two different coordinates in the image and mask all pixels outside the rectangle that is formed. The practical implementation details are identical to the patch in- in-painting degradation.
A.4 Zero-Shot Methods
We use our own implementations for all zero-shot methods, using the default hyperparameters unless specified otherwise. We use the variance-exploding notation, and we adjust all sampling algorithms accordingly, DPS [II] and DAPS [60] only require access to the degradation operator H, while DDNM 55 and IIGDM (5I) also make use of the pseudo-inverse operator H In the original implementation DDRM 33 makes use of the complete SVD of the operator H. Because obtaining this SVD is computationally expensive (or even practically impossible in reasonable time) for most of our degradations, we instead implement the DDRM algorithm using only H and H For the noise-less case, this implementation is equivalent to the original implementation. This is not true in the general noisy case, unless all the degradation's 's singular values that are not zeros are equal. Due to the degradation families we choose apply, we assume that most of the singular values that are not zeros are nearly one for the purpose of implementing DDRM in the noisy degradation experiment. For IIGDM, we make a similar assumption to implement the noisy sampling algorithm, to avoid inverting large matrix. For the in-painting case, the assumption holds. Similarly, in the strided motion-blur we make use of the fact that white Gaussian noise remains white Gaussian noise under a Fourier transform, and that the application of the kernel is multiplicative.
B Using Diffusion Models for MMSE
In Sec. 3.1 we explore the of the InvFusion architecture of MMSE estimation. A network be trained directly as degradation-awarr MMSE estimator using a regression loss as seen in Eq. (7). Nevertheless, we notice that we can utilize our existing trained diffusion models (trained using Eq. (6)) for the same task. Intuitively, this is because for sufficiently high noise values, the noisy input to the network is equivalent pure noise. Therefore, The network learns rely only on the information from and H. In practice, we use the value or 100 for the MMSE estimation.
We test the gap between models trained directly for MMSE estimation (Eq. (7) and models trained for diffusion (Eq. (6) on FFHQ64, using the same architecture. The MMSE model's ''s input is Gaussian noise, instead of the noisy image input the diffusion model, keep the architecture identical. The results Tab. reflect that both methods yield approximately the same results, whether trained using the degradation-awart InvFusion architecture and loss without it. We conclude that it is probably more cost-effective to use a single model for both posterior sampling and MMSE estimation task, and do for the experiments our paper.
C Ablations
Table shows the effect of applying the InvFusion block only on a fraction of network layers. this experiment, the InvFusion block has been added or removed selectively to each of the HDiT layers in specified resolution as used in our base architecture. The results show a gradual improvement across metrics with added applications of the InvFusion block.
24

---

Table 8: Ablation of removing the InvFusion block from the architecture different resolutions.
Strided Motion Blur
In-Pat In-Painting
Matrix Operator
InvFusion Resolutions
PSNR FID CFID
PSNR
FID CFID
PSNR FID., CFID
None (Baseline)
21.0
4.7
5.8
17.0
5.3
5.5
12.5
6.6
25.5
Resolution 1
22.3
4.5
4.5
17.1
4.9
4.9
14.6
8.9
8.9
Resolutions & 2
22.7
4.4
4.4
17.1
5.0
5.0
15.0
7.9
7.9
Resolutions 1 (AII)
22.7
4.2
4.2
17.1
5.0
5.0
15.1
7.1
Table LPIPS evaluation
Strided Motion Blur
In-Painting ImageNet
Matrix Operator
Dataset
FFHQ
ImageNet
FFHQ
FFHQ
ImageNet
Zero-shot Methods
DDRM 33
0.095
0.246
0.268
0,458
0.517
0.277
DDNM 59
0.105
0.373
0.171
0,406
0.672
0.300
DPS |
0.136
0.233
0.276
0,462
0.480
0.281
PiGDM 51
0.064
0.156
0.158
0,312
0.470
0.287
DAPS 60
0.683
0.736
0.148
0.327
0.535
0.263
MGPS
0.250
0.441
0.193
0,371
0.085
0.193
Training based Methods
Palette 46
0.068
0.147
0.103
0.215
0.409
0.236
InvFusion
0.053
0.122
0.101
0.213
0.340
0.182
MMSE predictor (Training-baseen
Palette [46]
0.126
0.327 0.278
0.144
0.318
0.606
0.314
InvFusion
0.106
0.143
0.315
0.469
0.237
D Additional Results
D.1 LPIPS Evaluation
We include LPIPS 622 evaluations for experiments in Tab. which correspond to the results shown in Tabs.
and in Sec. InvFusion demonstrated superior performance on LPIPS as well.
D.2 Consistency Comparison
To validate the consistency of different methods directly, we measure the MSE in the degradation range-space for each reconstruction method. The results shown in Tab. demonstratee the consistency of the reconstructions corresponding to the experiment Tab. MSE used to accommodate degradations of differing rank and range, and therefore has different scale for each degradation class. The MSE values confirm that InvFusion achieves consistency comparable to methods that employ explicit projections, such DDNM DDRM. These results offer complementary metric to the FID/CFID analysis.
D.3 Noisy Degradations
Table II shows the results the experiment described in Sec, 4.2 The results show that InvFusion maintains superiority when noise is added to the degradation. We expect this framework to work well across many noise types, not limited white Gaussian noise, as the model learns to treat noise through training, which does not rely on any fundamental properties of the noise. Nevertheless, for high noise levels InyFusion fails, as seen in the case for high levels of Poisson noise shown in Tab. 22
25

---

Table 10: Consistency comparison using MSE in the degradation space.
Method
Strided Motion Blur
In-Painting
Matrix Operator
DDRM 35
6.90
0.00
0.00
DDNM 59
2.28
0.00
0.00
DPS 1
789.63
10.27
0.00
IIGDM [5]
3.22
0.00
0.00
DAPS 660
720765.61
2.89
5.68
MGPS
2087.28
23.55
9.30
Palette
46
115.67
0.23
11.57
InvFusion
5.94
0.01
0.00
Table 11: Comparison of noisy inverse-problem solving methods FFHQ64.
Strided Motion Blur
In-Painting
Matrix Operator
NFEs
Method
PSNRT
FID
PSNRT
FID
PSNR
FID4
Zero-shot Methods
DDRM 33
11.7
137.0
12.2
155.6
11.8
87.0
25
DDNM 59
19.8
97.2
17.3
46.4
11.9
136.9
100
DPS 11
7.0
12.3
8.3
7.8
9.2
22.9
1000"
IIGDM
51
16.5
11.5
15.7
21.5
10.8
20.6
100"
DAPS
60
16.9
46.4
15.5
43.7
10.6
76.8
1000
Training-buat based Methods
Palette [46
17.1
6.0
16.7
5.2
12.6
6.6
63
InvFusion
21.7
4.3
16.6
5.0
14.7
6.2
63
MMSE predictor (Training-barl based)
Palette [46
19.8
18.9 19.4
15.6
1
InvFusion 24.1
17.5
1
Methods that use the network derivative, using more computation and memory per NFE.
D.4
FFHQ 256
Table shows quantitative results for our experiment on 256 x 256 FFHQ images. InvFusion remains the SOTA method measured by CFID the perceptual image quality of valid solutions to the inverse problem.
D.5 Comparison to Algorithm Unrolling
In this experiment, we compare our method to algorithm unrolling 21 unconditional model trained for the experiments in Sec. we unroll the DDRM 33 through 10 steps and train this in end-to-end fashion with MSE loss. Table 14 obtained by DDRM before and after the finetuning, compared InvFusion. The results show that InvFusion is far superior in terms of image quality, and can reach higher fidelity using single NFE using the MSE estimator method.
54
61
Using the algorithm shows the results
40
Table 12: Comparison of inverse-problem with high level of Poisson noise FFHQ64.
Strided Motion Blur
In-Painting PSNR
Method
PSNR
FID.
FID
Palette [46
14.1
17.6
12.6
9.9
InvFusion
11.9
60.0
14.7
11.5
26

---

Table 13: Comparison of inverse-problem solving methods on FFHQ256.
Strided Motion Blur
In-Painting FID
NFEs
Method
PSNR1
FID
CHID
PSNR
CHID4
Zero-shot Methods
DDRM 33
25.3
43.0
43.0
12.8
101.6
101.6
25
DDNM 59
26.2
49.9
49.9
15.2
102.1
102.1
100
DPS [
22.6
15.2
19.7
9.8
24.3
90.7
1000"
H1GDM [1
24.6
9.4
9.4
16.6
26.4
26.4
100"
DAPS 60
9.4
397.8
145.7
17.8
40.1
46.8
1000
MPGS 41
26.0
26.5
25.9
18.2
14.8
15.2
1764
Training-baad based Methods
Palette 44
23.2
12.1
18.2
11.1
12.4
63
InDI [14
24.1
11.3
14.2
20.6
14.0
14.1
50
InvFusion 24.4
8.4
8.4
18.5
11.2
11.2
63
MMSE predictor (Training-base
Palette [46
25.4
20.7
1
InvFusion 26.4
21.1
Methods that use the network derivative, using more computation and memory per NFE.
Table 14: Comparison of our method with algorithm unrolling.
Strided Motion Blur
In-Painting PSNR* FIDI CFIDJ
Matrix Operator
NFEs
Method
PSNR FID. CFID
PSNR FIDI CFID
DDRM (Sampling)
22.2
29.5
29.5
12.2
92.0
92.0
11.0
91.4
91.4
10
Unrolled DDRM
24.1
41.2
40.0
19.6
65.6
65.6
14.1
182.9
182.9
10
InvFusion (Sampling)
22.7
4.2
4.2
17.1
5.0
5.0
15.1
7.1
7.1
63
InvFusion-MSE
25.0
19.8
17.9
D.6 Comparison Deep Image Prior
In this section, we compare our method to recent Deep Image Prior (DIP) method, ASeq-DIP 3 Methods like Deep Image Prior operate the single-image level, making restoration computationally prohibitive task for more than a few images. Additionally, these methods typically optimize for different objectives- fundamentally different architectures, limiting the conclusiveness of direct comparisons. To provide meaningful context nonetheless, we evaluated PSNR, KID 7 and LPIPS 62 metrics for 1024 images restored using DIP 33. and compare it to our methods in Tab. 15 The results demonstrate a clear advantage for our model, despite the substantially cheaper evaluation. We use the default hyperparameters as described in the official implementation, and we find that using larger model or a different number inner and outer steps has a negligible effect on the results.
as MAP estimation rather than posterior sampling quality-ane and employ
D.7 Unconditional Samples
Figures and show qualitative examples for unconditional samples from our model i.e. samples that are conditioned on measurement vector y. The models trained ImageNet are conditioned on the class label, and use CFG [27] coefficient of 2.0 to enhance the image quality.
27

---

Table 15: Comparison of our method with Deep Image Prior.
Strided Motion Blur PSNR1 KID., LPIPS.
In In-Painting
Matrix Operator
Method
PSNR1 KIDI LPIPS..
PSNR KIDJ LPIPS
ASeq-DIP
22.4
51.2
0.48
13.1
75.5
0.66
12.7
247.0
0.93
InyFusion
21.1
6.3
0.13
15.5
7.6
0.24
15.2
10.0
0.34
InyFusion-MSE
23.7
25.71
0.33
18.44
37.9
0.39
18.1
43.8
0.50
Figure Examples class-conditional sam- ples from ImageNet generated using our model. Samples are not conditioned on any measure- ments y, and sampled with CFG of 2.0 on the class conditioning.
Figure 5: Examples of unconditional samples from FFHQ generated using our model.
28


Reviews:
[{"review_id": "okSXq6Xq8z", "reviewer": "Reviewer_Fzwc", "summary": "The authors of this work propose a diffusion model based inverse problem solver that integrates measurement operators inside the underlying diffusion network and train it in that manner. This is done by applying the degradation operator, a linear transformation, and pseudoinverse of the degradation operator to rearranged (in shape of stack of images) deep features. Results are shown for some image restoration tasks on standard datasets.", "questions": "1.\tWhat are the PSNRs of the “not seen in training” results in Figure 3? How do they compare to the PSNRs in Table 3?\n\n2.\tIn Section 4.1, the authors can explain unconditional generation better.  The MMSE predictor model could be explained better.\n\n3.\tThe authors state “Projection operators are not well-defined for noisy degradations” – which is not clear. Projection could also be done onto the set ||y-Hx|| < epsilon with epsilon set based on the noise. Have the authors compared to other recent diffusion models that incorporate measurement noise information?", "limitations": "The authors oversell their approach at numerous places.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n\n1.\tThe approach offers a novel way to integrate measurement operators into the diffusion model without running into the computational cost of explicitly passing them as network inputs.\n\n2.\tResults are shown for a few tasks comparing to training-based and zero-short (i.e., measurement operator is integrated at inference time) diffusion methods.\n\nWeaknesses\n\n1.\tThe approach is novel but ad hoc/ill-motivated without the solid basis in physical consistency that many other image reconstruction models like unrolled networks deploy.\n\n2.\tThe approach requires the deep features to match the image dimension (or be modified for it) – to apply the degradation operator – something not motivated well. \n\n3.\tIn several imaging systems, H and its pseudo-inverse can be expensive to apply rendering the training and application of the diffusion model expensive.\n\n4.\tThe authors mention unrolling methods but the results section lacks comparison to state of the art physics-based supervised unrolled networks as well as recent zero-shot deep image prior (DIP), etc., or combined diffusion-DIP methods.\n\n5.\tThe PSNRs in Table 3 are quite poor. Perhaps, other kinds of out of training degradation testing could be shown for more comprehensive study.\n\n6.\tInvFussion (without MMSE predictor) offers worse performance in several situations than other zero-shot methods compared with. This lacks more detailed analysis. \n\n7.\tThe comparisons to diffusion models is quite limited as there are more recent methods with much better performance than DAPS, etc. \n\n8.\tThere are several typos/English grammar/spelling errors in the manuscript. Please proofread. “preform”, “Bayse”, etc.", "was_revised": true, "final_justification": "The authors responded to my comments and promised to do some more updates in final version. Based on the replies, I increased my score to reflect my opinion of the work.", "rebuttal": "Thank you for your comprehensive review and constructive feedback. We address each concern below:\n\n* **Motivation** Rather than enforcing physical consistency through explicit constraints like unrolling methods, our approach is motivated by the principle of learning-based adaptation to measurement operators. Even though consistency with the measurements is not enforced, we find that our architecture manages to learn to satisfy the consistency constraint quite well.\n\n* **Deep Feature Dimensions** This architectural constraint is readily accommodated in modern diffusion architectures, particularly DiT variants, through straightforward hyperparameter adjustments that do not compromise performance. The implementation involves rounding the hidden dimension in specific layers to ensure the total number of hidden feature elements is divisible by the total image dimensions. Importantly, this constraint need not apply to all layers, enabling architectural flexibility. We will clarify the scope and implementation details of this limitation in the final version.\n\n* **Computation of $\\boldsymbol{H}$** While we acknowledge this computational consideration, it is important to emphasize that this limitation is inherent to many inverse problem solving methods, including established zero-shot approaches. Moreover, the practical applications we target—and many others in the field—benefit from efficient implementations of both $\\boldsymbol{H}$ and $\\boldsymbol{H}^\\dagger$. We will explicitly discuss this computational consideration in the final version.\n\n* **Comparison to Unrolling** While we agree that comparisons to supervised unrolled networks would provide valuable context, direct comparison presents significant practical challenges for our evaluation framework. Methods like Deep Image Prior operate at the single-image level, making a single 10K FID evaluation computationally prohibitive. Additionally, these methods typically optimize for different objectives—such as MAP estimation rather than posterior sampling quality—and employ fundamentally different architectures, limiting the conclusiveness of direct comparisons.\nTo provide meaningful context nonetheless, we evaluated PSNR and LPIPS metrics for approximately 250 images restored using ASeq-DIP [4], a recent DIP method. \\\nDespite ASeq-DIP's significantly higher per-sample computational cost, InvFussion achieves superior performance across all metrics. We will include this analysis in the final manuscript.\n\n|Method        | Deblur    || In-Painting    || Matrix         ||\n|-----------|-------|-------|-------|-------|-------|-------|\n|            | PSNR    | LPIPS    | PSNR    | LPIPS    | PSNR    | LPIPS    |\n|ASeq-DIP [4]    | 22.6    | 0.423    | 12.9    | 0.660    | 12.6    | 0.924    | \n|InvFussion    | 23.4    | 0.278    | 18.4    | 0.315    | 17.3    | 0.469    |\n\n* **Table 3** We acknowledge the reviewer's concern regarding the PSNR values. To address this, we repeat the experiment with a larger constant out-painting mask, yielding the following results below.\\\nThese results maintain the same performance trends observed in Table 3. We emphasize that our primary objective is robust performance on degradations *within the training distribution*, which encompasses a wide and diverse range of conditions.\n\n|Method\t| PSNR\t| FID\t\t| CFID\t\t|\n|-|-|-|-|\n|DDRM\t\t| 14.9\t\t| 9.5\t\t| 9.5\t\t|\n|DDNM          \t| 17.0\t\t| 21.6\t\t| 21.6\t\t|\n|DPS               \t| 11.3\t\t| 10.7\t\t| 15.8\t\t|\n|PiGDM           \t| 15.6\t\t| 3.6\t\t| 3.6\t\t|\n|DAPS             \t| 16.6\t\t| 12.8\t\t| 10.0\t\t|\n|Palette\t\t| 16.0\t\t| 7.6\t\t| 5.7\t\t|\n|InvFussion      | 16.7\t\t| 3.0\t\t| 3.0\t\t|\n|MSE Methods |||\n|Palette\t\t| 18.2\t\t| -\t\t| -\t\t|\n|InvFussion      | 19.5\t\t| -\t\t| -\t\t|\n\n* **PSNR Comparison** The PSNRs reflect the inherent Perception Distortion Tradeoff [1] in restoration methods. Our approach prioritizes generating perceptually realistic samples, as evidenced by consistently superior FID scores, while maintaining competitive PSNR values among methods with comparable perceptual quality. The MMSE variant demonstrates our model's flexibility to achieve higher PSNR when distortion minimization is the primary objective\n* **Other Methods** In our work, we have included comparisons with a representative selection of well-established zero-shot methods that remain dominant in the field. Following suggestions from reviewers 8Xy4 and XFfT, we will incorporate additional comparisons with InDI [2] (training-based) and MGPS [3] (zero-shot), both representing more recent developments. These comparisons demonstrate competitive performance for our proposed method.\n\n|Method   | Deblur   ||| In-Painting      ||| Matrix ||| NFEs |\n|-|-|-|-|-|-|-|-|-|-|-|\n|        | PSNR    | FID    | CFID    | PSNR    | FID    | CFID    | PSNR    | FID    | CFID    |\n|MGPS [3]   | 20.5    | 7.7    | 9.7    | 13.9    | 10.5    | 24.3    | 11.7    | 8.2    | 29.0    | 1764 |\n|InDI [2]           | 21.4    | 21.6    | 22.8    | 17.7    | 22.6    | 25.0    | 13.7    | 35.5    | 58.7    | 50   |\n|InvFussion | 22.7    | 4.2    | 4.2    | 17.1    | 5.0    | 5.0    | 15.1    | 7.1    | 7.1    | 63\t |\n\n* **Typos** We will conduct thorough proofreading to address all typographical and grammatical errors in the final manuscript.\n\n* **PSNR of examples** For the facial image example shown in Figure 3: DDRM: 8.98 dB, DDNM: 11.06 dB, DPS: 13.14 dB, PGDM: 14.85 dB, DAPS: 9.44 dB, Palette: 13.65 dB, InvFussion: 16.06 dB. These values align with the trends observed in Table 3, demonstrating consistent performance across different test scenarios.\n\n* **Unconditional and MMSE** We will improve our explanations of these concepts in the final version. For unconditional generation, we simply omit conditioning on any measurement, allowing the model to generate samples from the prior distribution rather than the posterior distribution given a degraded image. For MMSE estimation, we demonstrate how training-based inverse problem solvers can be adapted for MMSE estimation by leveraging the near-equivalence between the high-noise regime in diffusion training and linear regression problems.\n\n* **Projection for Noisy Degradations** The $\\epsilon$-ball projection suggested by the reviewer ($||\\mathbf{y}-\\boldsymbol{H}\\mathbf{x}|| < \\epsilon$) can indeed be used as a projection. The CFID values with this projection, corresponding to Table 8, are as follows:\n\n|                | Deblur        | In-Painting        | Matrix            |\n|-|-|-|-|\n|DDRM            | 96.9   | 124.7        | 85.4    |\n|DDNM               | 59.5    | 75.2       | 137.9   | \n|DPS             | 12.3    | 20.2       | 25.5    |\n|PiGDM             | 8.7    | 23.8        | 19.2   | \n|DAPS              | 78.1    | 61.9        | 75.7    |\n|Palette          | 4.7    | 5.3       | 6.2    | \n|InvFussion         | **4.3**    | **5.0**        | **6.2**    | \n\n[1] Blau, Yochai, and Tomer Michaeli. \"The perception-distortion tradeoff.\" Proceedings of the IEEE conference on computer vision and pattern recognition (2018).\n\n[2] Delbracio et al., \"Inversion by direct iteration: An alternative to denoising diffusion for image restoration.\" arXiv preprint arXiv:2303.11435 (2023).\n\n[3] Moufad, Badr, et al. \"Variational Diffusion Posterior Sampling with Midpoint Guidance.\" The Thirteenth International Conference on Learning Representations (2025).\n\n[4] Alkhouri, Ismail, et al. \"Image reconstruction via autoencoding sequential deep image prior.\" Advances in Neural Information Processing Systems 37 (2024)."}, {"review_id": "gXhMnDpei2", "reviewer": "Reviewer_8Xy4", "summary": "This paper proposes an architectural innovation for diffusion-based inverse problem solving, where a representation of the degradation operator is incorporated into the architecture. As a result, after training, the model can leverage degradation information in inference time, which may benefit model performance against techniques where the degradation needs to be inferred from the measurement itself. Numerical experiments demonstrate that the proposed technique outperforms zero-shot baselines and a supervised baseline on 2 image datasets with 64$\\times$64 resolution.", "questions": "- How does the method compare to other techniques on larger resolution images? \n- How does performance compare to an end-to-end trained non-diffusion image restoration method with the same training data, such as Restormer? I believe this would provide a strong baseline with highly reduced compute cost compared to diffusion models.\n- How does performance compare in terms of LPIPS? \n- How does performance scale with respect to NFEs? \n- How does compute cost compare across different methods?", "limitations": "Limitations are discussed in the paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The idea of incorporating unrolled networks into diffusion architectures to better capture the degradation operator is interesting. It is a promising way of conditioning on the degradation.\n- The experimental results show clear benefits over zero-shot baselines in terms of performance.\n- The paper is easy to follow.\n\nWeaknesses:\n- The experiments are performed on low resolution images. It is unclear how the proposed technique performs and compares to others on more practical resolutions such as 256$\\times$256.\n- Experiments are missing on compute cost and analysis of performance with respect to NFEs.\n- There are multiple other learning-based diffusion solvers that have not been compared to in the paper, including [1], [2], [3]. \n- I don't see how the proposed method is more \"degradation-aware\" than other techniques that condition on the pseudo-inverse reconstruction of the measurement. On a practical level, I understand that due to the architectural changes, the conditioning may be more efficient/stronger, but sampling is still not conditioned on the operator itself, but only on its action on the input and internal representations. Therefore, on a conceptual level this is not more degradation-aware than other techniques leveraging the action of the operator. A truly degradation-aware method would incorporate some direct representation of the operator explicitly in the prediction. I understand that this may not be practical, but I push back on the claim in the abstract that this is \"the first training-based degradation-aware posterior sampler\".\n- It would be interesting to add LPIPS as an evaluation metric, which measures perceptual quality but is not sensitive to the number of samples.\n- Minor comments: (1) I would use the term cross-attention instead of joint-attention. (2) Why is the method called InvFussion and not InvFusion? Typos on lines 63, 121, 130, caption of Fig 4.\n\n[1] Fabian et al.,  \"DiracDiffusion: Denoising and incremental reconstruction with assured data-consistency.\" Proceedings of machine learning research 235 (2024): 12754.\n\n[2] Delbracio et al., \"Inversion by direct iteration: An alternative to denoising diffusion for image restoration.\" arXiv preprint arXiv:2303.11435 (2023).\n\n[3] Fabian et al., \"Adapt and diffuse: Sample-adaptive reconstruction via latent diffusion models.\" Proceedings of machine learning research 235 (2024): 12723.", "was_revised": true, "final_justification": "Authors added experiments comparing to relevant work and presented an additional metric for comparison. I raised my score in accordance with these improvements.", "rebuttal": "Thank you for your comprehensive review and constructive feedback. We address each concern below:\n\n* **Resolution** We appreciate this important concern about scalability. Kindly note that Appendix C2 reports experiments on 256x256 images. We will highlight these experiments in the main text.\n\n* **NFE Comparison** We have included NFE counts for all compared methods in Tables 1 and 2. Our method achieves superior performance while requiring significantly fewer NFEs than most competing approaches—a key practical advantage for real-world deployment. We will enhance the main text to better emphasize this computational efficiency.\n\n* **Additional Methods** We highlight that the cited works [1-3] train separate models for individual degradation types, whereas our approach trains a single unified model capable of handling multiple degradations. This represents a fundamentally different problem setting with distinct practical advantages. That said, we recognize the value of these comparisons, and we have trained and evaluated [2] on the multi-degradation task demonstrated in Table 1 (results below). Additionally, some approaches like [2] could potentially be combined with our architecture, which we see drastically improves the adaptation to the multi-degradation setting.\n\n| Method | Deblur | | | In-Painting | | | Matrix | | | NFEs |\n|-|-|-|-|-|-|-|-|-|-|-|\n|                | PSNR    | FID    | CFID    | PSNR    | FID    | CFID    | PSNR    | FID    | CFID    | |\n|InDI            | 21.4    | 21.6    | 22.8    | 17.7    | 22.6    | 25.0    | 13.7    | 35.5    | 58.7    | 50   |\n|InDI+InvFussion    | 22.9    | 6.4    | 6.4    | 18.5    | 8.4    | 8.5    | 16.7    | 14.9    | 15.2    | 50 |\n|InvFussion        | 22.7  | 4.2   | 4.2   | 17.1  | 5.0   | 5.0   | 15.1  | 7.1   | 7.1   | 63     |\n\n* **Degradation Aware** We appreciate this conceptual point and the reviewer's distinction. Any method attempting to restore from a zero-filled degradation $\\boldsymbol{H}^\\dagger\\mathbf{y}$ is essentially operating in a blind setting, as no additional information is provided beyond the degraded sample (corresponding to Eq. (2)). In contrast, we use \"degradation-aware\" to denote methods that incorporate some—even partial—information from the known degradation, enabling a non-blind solution.\\\nWe agree with the reviewer that conditioning on the operator's action is not equivalent to conditioning on the operator itself. Nevertheless, our architecture gets to \"probe\" $\\boldsymbol{H}$ several times along the network, each time applying it to different feature vectors. Therefore, theoretically, the network can deduce a lot about $\\boldsymbol{H}$. Both our results and those of zero-shot methods (which also implicitly condition on the action of the operator) demonstrate that incorporating this partial information about the operator's action substantially improves results, as expected from the theory in Eq. (2).\\\nWhile we maintain that our approach provides meaningful degradation conditioning beyond standard pseudo-inverse reconstruction, we will refine our positioning in the final version to more precisely describe our contribution within the existing framework of degradation-conditioned methods.\n\n* **LPIPS** We will include LPIPS scores for all experiments as requested. This will provide additional perspective on perceptual quality alongside our existing metrics. Here are the results of LPIPS on the experiments corresponding to Tables 1 and 2:\n\n|                | Deblur        || In-Painting        || Matrix            ||\n|-|-|-|-|-|-|-|\n|Dataset            | FFHQ     | ImageNet    | FFHQ    | ImageNet    | FFHQ | ImageNet    |\n|DDRM            | 0.095    | 0.246        | 0.268    | 0.458        | 0.517    | 0.277        |\n|DDNM               | 0.105    | 0.373        | 0.171    | 0.406        | 0.672    | 0.300        |\n|DPS             | 0.136    | 0.233        | 0.276    | 0.462        | 0.480    | 0.281        |\n|PiGDM             | 0.064    | 0.156        | 0.158    | 0.312        | 0.470    | 0.287        |\n|DAPS              | 0.683    | 0.736        | 0.148    | 0.327        | 0.535    | 0.263        |\n|Palette          | 0.068    | 0.147        | 0.103    | 0.215        | 0.409    | 0.236        |\n|InvFussion         | **0.053**    | **0.122**        | **0.101**    | **0.213**        | **0.340**    | **0.182**        |\n|MSE Methods   ||||||    \n|Palette           | 0.126    | 0.327        | 0.144    | 0.318        | 0.606    | 0.314        |\n|InvFussion         | 0.106    | 0.278        | 0.143    | 0.315        | 0.469    | 0.237        |\n\n* **Cross-Attention** Thank you for this clarification. The reviewer is correct that \"cross-attention\" is the more standard terminology. Our \"joint-attention\" (also referred to as \"extended-attention\" in some literature) processes queries against both self-branch and cross-branch keys/values simultaneously, differing from the sequential \"self-then-cross attention\" approach. We will clarify this point in the final version. This architectural choice is based on our early experiments; we expect a “self-then-cross attention” architecture to perform nearly as well in all experiments.\n\n* **Typing Errors** We will address all noted typographical errors and consider updating the method name.\n\n* **Comparison to End-to-End Non-Diffusion Methods** We anticipate that diffusion-based approaches will demonstrate superior image quality compared to methods like Restormer, which employs transformer architecture (similar to ours) but is trained to minimize L1 loss. For scenarios requiring low distortion with reduced computational cost, our single NFE MMSE estimation method provides a viable alternative.\n\n* **Scaling With Respect to NFEs** We are not sure we understand the reviewer’s question. As a diffusion model, we expect image quality to worsen the fewer NFEs are used for sampling. Given the computational speedup compared to zero-shot methods, we have kept the NFE count as recommended by [28]."}, {"review_id": "qneusrNePO", "reviewer": "Reviewer_XFfT", "summary": "The paper addresses the problem of solving inverse problems using denoising diffusion models. Existing zero-shot approaches can approximate samples from the posterior distribution without requiring operator-specific training, but they often rely on crude approximations that make them slow and potentially inaccurate. On the other hand, training diffusion models for conditional generation yields more accurate results but is computationally expensive and tied to a specific forward operator. The proposed method strikes a balance between these two extremes by introducing a small, trainable component into a pre-trained diffusion model. This component leverages the closed-form expression of the forward operator, enabling more efficient inference while retaining some robustness to moderate distributional shifts in the operator.", "questions": "see below", "limitations": "I have a few questions regarding the method: \n\n- The InvFussion block requires the pseudo-inverse of the operator. How would you handle non-linear inverse problems for which the pseudo-inverse is intractable? \n\n- It seems that currently this method does not apply to latent diffusion. I see two bottlenecks for doing so. The operator would be the composition of the operator with the decoder and it is not clear how you would evaluate the activations at the decoder. Next, during training, you would need to backpropagate through the input decoder which is very expensive. Am i correct in assuming that these are the main limitations? \n\n- The noisy observation setting is used with a small Gaussian noise. I assume that the method struggles with high noise levels? How would it behave with different types of noise such as Poisson?", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Significance and originality: The idea proposed in this paper is original, meaningful, and likely to inspire further developments in the area of inverse problems with diffusion models.\n\nEmpirical results: The experimental evaluation is, in my view, quite thorough. The proposed method is compared against several zero-shot diffusion posterior sampling baselines as well as Palette. It consistently outperforms these methods, with the improvement over Palette being more modest. However, the authors convincingly demonstrate that their approach is more robust to out-of-distribution (OOD) operators. A notable weakness of the evaluation is the absence of comparisons with more recent and higher-performing posterior samplers, such as MMPS [1] and MGPS [2]. The baselines considered here are either unstable or known to underperform on tasks like inpainting.\n\nIt seems to that there are some weaknesses/limitations for the method but since I am still not 100% sure about them, I list them in the limitations section below.\n\n[1] Learning diffusion priors from observations by expectation maximization. Rozet, F., Andry, G., Lanusse, F. and Louppe, G., 2024. Learning diffusion priors from observations by expectation maximization.\n\n[2] Moufad, B., Janati, Y., Bedin, L., Durmus, A., Douc, R., Moulines, E. and Olsson, J., 2024. Variational Diffusion Posterior Sampling with Midpoint Guidance.", "was_revised": true, "final_justification": "The authors have clarified most of my concerns. The paper is interesting and I have no doubt that it will have a good impact. I recommend it for acceptance.", "rebuttal": "Thank you for your comprehensive review and constructive feedback. We address each concern below:\n\n* **Additional Method** We acknowledge that adding more comparisons to these recent posterior samplers would strengthen our paper. Following the reviewer’s suggestion, we have evaluated MGPS [2] on our experimental setting. The results, corresponding to Tables 1 and 2 in our paper, are presented below:\n\n| Method | Deblur | | | In-Painting | | | Matrix | | | NFEs |\n|---------|---------------------|---|---|-------------------|---|---|-------------------|---|---|------|\n|         | PSNR | FID | CFID | PSNR | FID | CFID | PSNR | FID | CFID |      |\n| **FFHQ** |||||||||| |\n| MGPS    | 20.5 | 7.7 | 9.7  | 13.9 | 10.5| 24.3 | 11.7 | 8.2 | 29.0 | 1764 |\n| InvFussion | 22.7  | 4.2   | 4.2   | 17.1  | 5.0   | 5.0   | 15.1  | 7.1   | 7.1   | 63     |\n| **ImageNet** |||||||||| |\n| MGPS | 19.4 | 14.2| 18.0 | 13.3 | 18.5| 25.0 | 11.7 | 13.8| 18.9 | 1764 |\n| InvFussion | 20.9 | 5.6 | 5.6 | 15.6 | 5.9 | 5.9 | 14.5 | 6.3 | 6.3 | 63 |\n\n* **Non-linear Operators** The reviewer raises an important point about pseudo-inverse tractability. Due to the exposure in training, we believe the model would be able to learn to use the operator even with a roughly approximated pseudo-inverse. The joint-attention scheme can also be applied directly with the transformed degraded measurements, without casting them back to the image space (that is, skipping the application of $\\boldsymbol{H}$). We hope to explore many non-linear operators in future work.\n\n* **Latent Diffusion** The reviewer correctly identifies the main computational bottlenecks for extending our approach to latent diffusion models. We will clarify these limitations in our revised manuscript. However, recent work such as [3] demonstrates promising approaches for handling operators in latent space that could potentially be adapted to our framework. We note that latent diffusion essentially introduces a non-linear encoding/decoding operator that could be handled using similar principles to those outlined above.\n\n* **Noise Level** We have not observed that the method is limited to certain levels or types of noise, we have simply used a single moderate value of $\\sigma$ in our experiment. We expect InvFussion to be more beneficial as the noise level increases, as the blind restoration problem becomes harder. We will add an experiment with a high rate of Poisson noise to the final version to assess our hypothesis.\n\n[3] Raphaeli, Ron, Sean Man, and Michael Elad. \"Silo: Solving inverse problems with latent operators.\" arXiv preprint arXiv:2501.11746 (2025)."}, {"review_id": "s1QGNrP1aH", "reviewer": "Reviewer_21uD", "summary": "This work presents InvFussion, a framework integrating degradation operators into diffusion models via attention layer unrolling to solve inverse problems (e.g., motion blur, in-painting) uniformly. Experiments on FFHQ and ImageNet show InvFussion outperforms zero-shot methods and \"Palette\" approaches, demonstrating superior performance. Furthermore, InvFussion can efficiently handle both posterior sampling and MMSE estimation, highlighting its computational efficiency and versatility.", "questions": "Including additional experiments on generalization to unseen degradation operators would greatly enhance the significance of this work and positively impact my overall assessment. If any of the above points reflect misunderstandings, please clarify them.", "limitations": "The authors provide a discussion of the method’s limitations in the main text.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\nOriginality: \nThis work presents a novel idea by integrating degradation operators via attention layers, offering a novel perspective for designing flexible and high-performing image generation algorithms. Extensive experiments support the effectiveness of the proposed module.\n\nClarity: \nThe manuscript is clearly written, with a well-organized structure and informative figures. The experimental settings are described in sufficient detail, making the work highly reproducible.\n\n**Weaknesses:**\n\nSignificance:\nWhile the motivation is solid, the proposed method falls short of its stated goals. It requires training on a specific range of degradation operators and is only generalized to an outpainting task, which closely resembles patch inpainting. In contrast, task-specific methods offer high performance, while zero-shot approaches provide greater flexibility, lower training costs, and no need for paired data.\nThe proposed method limits its practical applications compared to these two paradigms.\n\nQuality:\n1. Comparison experiments lack adequacy as the proposed method is only compared with a training-based approach published three years ago.\n2. The outpainting task used to demonstrate generalization shares notable similarity with the training task. It would be more convincing to test cross-domain generalization—for example, training on inpainting and evaluating on super-resolution or deblurring.\n3. The paper lacks ablation studies, which would help validate the method’s effectiveness. For example, removing the forward operator $H$ and only inputting $H^\\top y$ into $T_\\psi$ could clarify the contribution of incorporating the degradation operator.", "was_revised": true, "final_justification": "The authors have addressed my concerns about the module’s effectiveness through additional experiments in the rebuttal. However, my main concern remains the significance of the method. If the goal is optimal performance, task-specific approaches may be more suitable. If generalization is the aim, the method seems limited to a narrow family of degradations. I encourage the authors to further investigate which types of degradations the method can generalize to, as this is key to evaluating its broader applicability. Overall, thank you for the rebuttal, and I decide to keep my score unchanged.", "rebuttal": "Thank you for your comprehensive review and constructive feedback. We address each concern below:\n\n* **Generalization Capabilities** We acknowledge this limitation and appreciate the opportunity to clarify our contribution's scope. The reviewer correctly notes that our method operates within a trained distribution of degradations, which differs from the unlimited flexibility of zero-shot methods. However, our approach occupies a valuable middle ground with specific practical advantages. \\\nWithin the trained degradation family, our method demonstrates strong generalization - for instance, in our random matrix experiments reported in Tables 1 and 2, the degradation space is so vast that test-time degradations are virtually guaranteed to be unseen during training, yet our method significantly outperforms alternatives. Additionally, Section 4.1 demonstrates generalization to tasks slightly outside the training distribution (outpainting, which shares structural similarities with inpainting but involves different spatial patterns). \\\nThe practical value lies in scenarios where: (1) a known family of degradations is encountered frequently enough to justify training, and (2) computational efficiency and quality gains are important for deployment. This represents a meaningful trade-off between task-specific methods and zero-shot approaches.\n\n* **Alternative Approaches** We acknowledge that our current training-based comparisons are limited. Following the suggestion of reviewers 8Xy4 we include a comparison to InDI [1], a flow-based restoration method, in our multi-degradation setup. Preliminary results corresponding to Table 1 can be found below.\n\n| Method | Deblur | | | In-Painting | | | Matrix | | | NFEs |\n|-|-|-|-|-|-|-|-|-|-|-|\n|                | PSNR    | FID    | CFID    | PSNR    | FID    | CFID    | PSNR    | FID    | CFID    | |\n|InDI [1]            | 21.4    | 21.6    | 22.8    | 17.7    | 22.6    | 25.0    | 13.7    | 35.5    | 58.7    | 50   |\n|InvFussion        | 22.7  | 4.2   | 4.2   | 17.1  | 5.0   | 5.0   | 15.1  | 7.1   | 7.1   | 63     |\n\n* **Ablations** We will include comprehensive ablation studies in the appendix. Specifically, we address the suggested experiment of removing the forward operator $\\boldsymbol{H}$ and providing only $\\mathbf{y}$ as input to $T_\\psi$, which we have implemented for the FFHQ64 experiment and include below. This will help isolate the contribution of our mechanism.\n\n|Method            | Deblur    ||| In-Painting        ||| Matrix |||\n|-|-|-|-|-|-|-|-|-|-|\n|                | PSNR    | FID    | CFID    | PSNR    | FID    | CFID    | PSNR    | FID    | CFID    |\n|Only $\\mathbf{y}$     | 21.2    | 4.6    | 5.5    | 17.0    | 5.1    | 5.2    | 12.7    | 6.1    | 24.0    |\n|InvFussion    | 22.7  | 4.2   | 4.2   | 17.1  | 5.0   | 5.0   | 15.1  | 7.1   | 7.1   |\n\n[1] Delbracio et al., \"Inversion by direct iteration: An alternative to denoising diffusion for image restoration.\" arXiv preprint arXiv:2303.11435 (2023)."}]

Author rebuttal:

Meta-review: The average rating is 4, with reviewers mostly recommending acceptance (4, 3, 4, 5). The discussion has been constructive and improvements have been made to the paper, to the satisfaction of most reviewers. Concerns remain about the significance of the contribution. Given the otherwise positive reviews and the constructive discussion, I tend to recommend acceptance, as significance is often difficult to assess a priori. I encourage the authors to address the reviewers' comments and to implement the modifications discussed with the reviewers in the final version of the paper.
Decision: Accept (poster)