Regularization by Texts for Latent Diffusion Inverse Solvers
Authors: Jeongsol Kim
Geon Yeong Park
Hyungjin Chung
Jong Chul Ye
Venue: ICLR 2025 Spotlight (2025)
Keywords: Inverse problem
Text regularization
Diffusion model
Primary area: generative models
TL;DR: We propose text regularization for inverse problem solving.

Abstract: The recent development of diffusion models has led to significant progress in solving inverse problems by leveraging these models as powerful generative priors. However, challenges persist due to the ill-posed nature of such problems, often arising from ambiguities in measurements or intrinsic system symmetries. To address this, we introduce a novel latent diffusion inverse solver, regularization by text (TReg), inspired by the human ability to resolve visual ambiguities through perceptual biases. TReg integrates textual descriptions of preconceptions about the solution during reverse diffusion sampling, dynamically reinforcing these descriptions through null-text optimization, which we refer to as adaptive negation. Our comprehensive experimental results demonstrate that TReg effectively mitigates ambiguity in inverse problems, improving both accuracy and efficiency.

Published as a conference paper at ICLR 2025
REGULARIZATION BY TEXTS FOR LATENT DIFFUSION INVERSE SOLVERS
Jeongsol Kim", Geon Yeong Park', Hyungjin Chung, Jong Chul Ye
KAIST
*: Equal Contribution
[jeongsol, pky3436, hj. chung, jong wew.yeekkkitttcc. ac. kr
Suppr-recolutionxx155
Gaussian Debiur
Measuremen
Photography
'Monet'
Watercolor
Measureman
Measurement
Curl
Image Inpainting
Non-linear problem
Measurement remmen
Blue eye"
Green eye
Messsurement
LDPS
Tee
Figure 1: Representative solutions obtained by TReg for various inverse problems. TReg optimizes both data consistency and the semantic alignment of the solution with textual cues, by reducing the solution space with text-conditional latent regularizer. This serves as an effective semantic guidance throughout the reconstruction process.
ABSTRACT
The recent development of diffusion models has led to significant progress in solving inverse problems by leveraging these models as powerful generative priors. However, challenges persist due to the ill-posed nature of such problems, often arising from ambiguities in measurements or intrinsic system symmetries. To ad- dress this, here we introduce novel latent diffusion inverse solver, regularization by text (TReg), inspired by the human ability to resolve visual ambiguities through perceptual biases. TReg integrates textual descriptions of preconceptions about the solution during reverse diffusion sampling, dynamically reinforcing these descrip- tions through null-text optimization, which we refer as adaptive negation. Our comprehensive experimental results demonstrate that TReg effectively mitigates ambiguity in inverse problems, improving both accuracy and efficiency.
INTRODUCTION
Consider given forward measurement process:
= A(x) +
(I)
where : I + R describes forward measurement operator, y E Rn and æ E R" represents the measurement and the true image, respectively, and E R" denotes the measurement noise that follows the Gaussian distribution (00 I.).). Then, an inverse solver attempts to recover from the measurement y. Unfortunately, most inverse problems are ill-posed, meaning that many different visual inputs can produce identical measurements due to significant information loss during the measurement process. For example, consider Fourier phase

---

Published as a conference paper at ICLR 2025
the phase information of signal from its Fourier intensity measurements. Because of intrinsic system symmetries for shifts, rotations, and flips, it causes an imput to be mapped to multiple symmetry-related outputs when inverting such systems.
Traditionally, regularization techniques have been extensively studied to address the ambiguity in inverse problems. Methods such as sparsity (I1 norm), total variation (TV), and regularization by denoising (RED) have been widely employed, leveraging the statistical properties of natural images (Boyd al., 2011; Venkatakrishnan al., 2013; Romano et al., 2017). With the advent of diffusion models, a new class of Diffusion Inverse Solvers (DIS) (Kawar et al., 2022; Wang et al., 2022; Chung al.. 2022a; Mardani et al., 2023; Chung et al.. 2023d; Rout et al., 2024) has emerged, offering superior reconstruction performance. The main idea is to leverage the diffusion model, which learns the score function of the prior distribution, to mitigate the ill ill-posedness of inverse problems.
While diffusion models have significantly advanced the solution of inverse problems, it is important to acknowledge the unresolved ill- po-posdddeess in certain challenging imaging systems. For instance, in Fourier phase retrieval, Diffusion Posterior Sampling (DPS) (Chung et al., 2022a; Rout et al., 2024) recover solutions more effectively than traditional methods, but it still cannot fully overcome the intrinsic symmetry, shown in Figure 6. This limitation arises because the diffusion prior is based on image statistics, which alone may be insufficient to break the symmetry and guarantee a unique solution. A similar challenge is in systems with severely degraded measurements. Therefore, inverse problem solvers must incorporate additional cues to reduce ambiguity and fully resolve the problem.
In search of additional cues, we focus text descriptions which may provide contextual information to resolve ambiguities during the sampling process, enriching the reconstructed solutions with relevant semantic content. While text conditions play crucial role in the success of latent diffusion models, current DIS approaches based on LDM often do not fully leverage this descriptive conditioning to resolve uncertainties in recovered solutions. For instance, P2L (Chung et al., 2023d) leverages prompt embeddings as additional parameters for fine-tuning to improve alignment between solutions and measurements. However, this method primarily focuses on data consistency and thus lacks robust alignment with textual prompts. Thus, there remains a significant gap between how human perception resolves ambiguities by linguistic conditions and how DIS manage them. Bridging this gap may lead to more effective solver for addressing inverse problems.
To this end, we introduce novel concept called Regularization by Text (TReg), implemented through latent diffusion models. TReg leverages textual descriptions that encapsulate the preconceived description of the desired solution during the reverse sampling process, framing this as a latent optimization problem to mitigate the ill-posedness. Ideally, the concepts described in the text should be exclusively reflected in the outcomes. However, we observed that manually crafted descriptions can introduce noise, leading blurry outcomes artifacts. To avoid this, we propose innovative null-text optimization technique called adaptive negation, applied throughout the reverse diffusion sampling process. This method dynamically adjusts the influence of the textual guidance, ensuring that only the intended concepts align with the evolving state of the reverse sampling. As a result, TReg, when guided by a regularization prompt such as "photography of face," " can break symmetries in Fourier phase retrieval and consistently produce a unique, true solution, as demonstrated in Figure 6. Furthermore, our method functions as zero-shot inverse problem solver, robustly applicable across various domains, as illustrated in Figure 1.
2 BACKGROUNDS
2.1 LATENT DIFFUSION MODEL
Image diffusion models that operate on the pixel space are compute-heavy. So the latent diffusion model (LDM) (Rombach et al., 2022) has emerged as class of iiffusion-based generative mod- els (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2020b), where the diffusion process is operated on low dimensional latent space instead of the pixel space. Specifically, the LDMs are first trained as variational autoencoder by maximizing the evidence lower bound (ELBO) (Rombach et al., 2022; Kingma& & Welling, 2013). Thus, the latent is represented as:
z == E = E/S (x) - Eg (x) OE, e~ 10,,1,
(2)
2

---

Published as a conference paper at ICLR 2025
where denotes the seaneent-by-leemeee multiplication, and 2. parts of the encoder that outputs the mean and the variance of the encoder distribution. In this paper, we assume that Eg (z) is isotropic, i.e. Eg (xx) - 1g.. The resulting pixel domain representation can be obtained by
a = (z(=)
(3)
where Dep is the decoder.
Then, the forward difffusion is defined given the latent representation of the clean image =o = Ex(x) R which perturbs as 2t V0 vI-ā, are following forward VP-SDE. Accordingly, the residual denoiser (c)()) is trained to estimate the noise e from ž₁ by epsilon matching:
min ee) [[|c - co (zz, 1) 0
where Eo is often parameterized as time-conditional UNet (Ronneberger et al., 2015) in practice. Notably, the optimal solution eg of (4) serves as an alternative approximation of a score function as 2z, log p(z,) - (z,, l)/t (Song et al., 2020b). Based on this, the generative reverse sampling from the posterior distribution po (2t-1 Eos 20) can be performed as
(ze - v1 - 0460 (z,, Va/ - 0 20 v1 16-1 (zt,t)
001 =E [+]
(5)
(6)
which corresponds to single iterate of deterministic DDIM sampling (Song et al., 2020a).
For cext-conditional sampling, Classifier Free Guidance (CFG) (Ho Salimans, 2021) is widely leveraged based on the sharpened posterior distribution. Let @ denotes null-text embedding and c represents target text embedding. Then, we have Eğ (z1, c, = E0 2,t) + w(e c, (z 2, t)) where w is scale for the guidance. For the brevity, will use Eø(z, 22 Eg. (z(2, c, t) = éc. and eg (z, c, = c. For the details, please refer to Appendix A.6.
2.2 DIFFUSION INVERSE SOLVERS
Recently diffusion models have been emerged as powerful generative priors for inverse problems (Kawar et al., 2022: Song et al., 2022; Wang et al., 2022; Chung et al., 2022a; Mardani et al., 2023; Chung et al., 2023b). Earlier techniques in inverse imaging relied on an alternating projection method (Song et al., 2020b; Choi al., 2021; Chung et al., 2022b), enforcing hard measurement constraints between denoising steps either pixel or measurement spaces. More advanced strategies have been proposed approximate the gradient of the log posterior within diffusion models, broadening the scope to tackle nonlinear problems (Chung et al., 2022a). The field has seen further expansion with methods addressing blind inverse problem (Chung et al., 2023a), 3D (Chung et al., 2023c; Lee et al., 2023), and problems of unlimited resolution (Bond-Taylor & Willcocks, 2023).
Traditionally, these methods have utilized image-domain diffusion models, but shift has been observed towards latent diffusion models such as latent DPS (LDPS) with a fixed point of autoencoder process (Rout et al., 2024), LDPS with history update (He et al., 2023), Resample (Song et al., 2023), and leveraging prompt tuning to improve the reconstruction (Chung et al., 2024). Despite these innovations, the use of text embedding for regularization is often overlooked, missing an opportunity to fully leverage the power of multi-moda latent space an aspect we aim to address in following.
3 MAIN CONTRIBUTION: REGULARIZATION BY TEXT
In this section, we present TReg, an iterative refinement process for solving inverse problems by fully leveraging informative text conditioning. Our framework is based on a text-conditioned latent optimization objective, which, when minimized during the reverse sampling process, progressively improves both data consistency and the alignment of the estimated solution with the provided textual cue. This regularized sampling process produces solution that aligns to human perceptual priors. For textual alignment, our framework incorporates two key components: (1) cext-conditional latent regularization and (2) adaptive negation. Specifically, the text embedding is jointly optimized to align with the current latent representation, mitigating the unintended signal components introduced by hand-crafted text descriptions. We begin by outlining the overall latent optimization framework.
3

---

Published as a conference paper at ICLR 2025
3.1 LATENT OPTIMIZATION FOR TEXTUAL CONSTRAINT
Consider a probability density of posterior distribution p( gy, 2) where = denotes latent vector of clean image æ and y denotes the measurement. By the Bayes' rule, we can see that
p( y. =) x plyl,, 1 2)p(2z))) x p(z|æ, y y)p(yy))r))))))) ())
where we assume p(at z) == 8( (z)) given a well pre-trained autoencoder as in (3). Then, the objective of the maximum posteriori (MAP) problem is defined
(=)P(z) log p(=)pp(22))) y) pogp(y/Dp(z)) ||
0 (P)(2))
1 A(D
20
20
where the second equality holds due to the noisy measurement in (1) and VAE prior in (2).
While optimizing (8) during reverse sampling approximates posterior sampling. it does not guarantee semantic alignment between the estimated solution and the textual descriptions. To address this, we propose lext-conditional latent regularizer which drives the sampling trajectory towards clean manifold that optimally aligns with the text condition c:
TRegg (z) I Zuju 11
where the input latent is conditioned on the condition c, and Znj refers text-conditioned denoised estimate åg: = 2 (zt ( vI / derived by the Tweedie's formula (Rob- bins, 1992; Efron, 2011). Thus 20 serves as pivot, preventing the sampling trajectory from deviating significantly from the proper text-conditioned sampling path. By combining (8) and (9), the resulting proximal optimization framework is defined follows:
min 2,2
(MAP(=) + n (=) (=A + 7|1= 20jell
(10)
s.l. x = Dp()),
where we initialize the input latent 2a as Enμ, the initial clean estimate of the sampling process at time t. While one may use off-the-shelf optimizers, we aim to solve (10) by leveraging variable splitting inspired by the altermating direction method of multipliers (Boyd et al., 2011).
Namely, using the decoder approximation and setting æ = (z), the optimization problem with respect to becomes
E 1 - A(x))1 ||
min
A|I= Dz(()) nn
(11)
20²
20
Here, for simplicity, we set dual variable 17 as a zero vector and do not consider its update. Then, from the initialization with 20tc, we have
1 - 1(x)
2 arg min
A|
(
(12)
2x²
which can be solved with negligible computation cost such conjugate gradient (CG) if A is linear operation. Subsequently. using the encoder approximation and setting z E/ (z) with = 0, the optimization problem with respect to z is derived as
4
gema arg min <|] 2o (v)ll 7/|= éojel = - (0( + (1 ār 1
ā.-
(13)
where the second equality is a closed-form solution with the initialization of 2o(g) == (ăo (1)). Here, 7 are empirically chosen to satisfy ā;-1 </(C + +)) to hold the second cquality of (13). Specifically, we set the interpolation coefficient to prioritize En(y) during the final phase of reverse sampling, ensuring fine-grained refinement for improved data consistency. Meanwhile, the text- conditioned pivot relatively dominates the early stages the sampling process, establishing the coarse layout and improving text adherence. Note that the estimated solution a is obtained by the interpolation between initial text-conditioned estimate Enjt and En (y) derived with the measurement y. Finally, by integrating the updated latent gema single iterate of the resulting DDIM sampling (Song et al., 2020a) at t reads:
2/-1 V
Gema + / a
(14)
1
4

---

Published as a conference paper at ICLR 2025
(8)
Concept Emphasize
Concept Nagation
Adaptive Negation
CUP similarity
CUP siilarty
CUI similartty
concapts
Desired concept
Undesired concept Co
Undesired concept Co
(b)
Messurement
Without Adaptive Negation
With Adaptive Negation
otlggio
photography baby
Figure 2: (a) Concept of adaptive negation. Compared to concept emphasize and negation which targets specific concept, the adaptive negation tries to suppress concepts except the desired one. (b) Adaptive negation is crucial to avoid artifacts on reconstruction.
where é; denotes the total noise given by
V ât-1 11-1-011111 17 nB.e
é,
(15)
VI a1-
In (14), 17 [0, is parameter controlling the stochasticity of the update rule. Empirically, find that v/,3e /a:-1 /I dit1 results in robust performance. other words, the total noise is computed by é, == VI 011168 + Vāc-16. For the convergence analysis, please refer to the Appendix B.
3.2 ADAPTIVE NEGATION FOR TEXTUAL CONSTRAINT
The primary component of the proposed optimization framework includes 20/+. pivot for sampling paths conditioned on text embeddings, which effectively narrows the solution space. While con- ditioning on well-chosen embedding crucial for semantic alignment, relying on error-prone, hand-crafted prompts result in suboptimal reconstructions, as illustrated in Figure 2(b).
To mitigate this, we propose the joint optimization of the text embedding during the reverse sampling process in adaptive manner based image representations. Specifically, employ concept negation (Ho & Salimans, 2021) to suppress invalid concepts which relatively enhances the intended semantic contents, shown in Figure 2(a). We prioritize concept negation concept emphasis via direct prompt tuning (Lester et al., 2021). as the latter may disrupt the original intended semantic contents embedded in the text prompt.
Here, the null-text embedding ca CFG regarded as a representation of concepts to be suppressed. The goal of adaptive negation is to update cœ exclude concepts that is already captured by the latent through latent optimization. During the reverse sampling, we update Co to minimize the following similarity in the CLIP (Radford al., 2021) embedding space:
E (ce) (Ting (wo( ca).
(16)
where Timg denotes CLIP image encoder, and (y) denotes clean estimates pixel space which is the solution of (12). By optimizing (16), Cg is "adaptively" updated to encode concepts that are not prominent in the image representations, thereby allowing it represent complementary concepts.
The computational cost of this optimization negligible as shown in Table 4, since only involves the optimization of eg without backpropagating through the denoiser.
5

---

Published as a conference paper at ICLR 2025
3.3 LATENT DPS WITH UPDATED NULL-TEXT
Based on the defined proximal optimization on clean latent domain, we found that alternating latent DPS steps further improves the data consistency, especially for the image inpainting and phase retrieval problems, In this case, we obtain the standard LDPS gradient with null text embedding cæ. and update the intermediate noisy sample:
2 z Pe Zz, (lA(p((2))))(
yll
(17)
where Pt denotes the step size that weights the likelihood, similar (Chung et al., 2022a; Rout et al., 2024). In image super-resolution and deburring, we bypass the DPS update for computational efficiency. Nevertheless, our algorithm can effectively solve the problem and shows comparable performance with DPS update as shown in Figure 7. To sum up, the proposed algorithm is described as in Algorithm
4 EXPERIMENTS
The goal of text regularization (TReg) is to refine the solution space and reduce ambigu- ity by leveraging text prompts as additional cues. We assume that both the measure- ment and a text prompt describing the so- lution are provided for the inverse problem. To highlight the effectiveness of ambigu- ity reduction, we evaluate TReg under ex- treme measurement conditions, in contrast to other inverse problem solvers. Our evalu- ation focuses on two key aspects: (1) the ef- fectiveness of TReg in resolving ambiguity through text, and (2) the accuracy of the re- sulting solution, which includes alignment with both the text and the measurements. For further details on the experimental set- tings, please refer the Appendix.
Algorithm 1 Inverse problem solving with TReg
Require: Pretrained LDM 6g, Null text embedding Cz+ Text embedding c, VAE encoder &g,, VAE decoder D.p. Forward operation A, Measurement y. CLIP image encoder Timgs Update Range r" Initialization 2/ ~ I0
for I E [7, 1/ do
Soje (2, VT Vot
ē; + Compute noise using (15)
if E I' then
// Latent Optimization
áo < D(2oje)
oo Equation (12)
Eo(y) t E(
20 geme + -0-1220 ( + (1 āz-1)
21-1 + vog-120 eeaa
+ VI ac-1ct
I/ Adaptive Negation
4.1 EXPERIMENTAL SETTINGS
Cg f- C2 - V (Timg (0o ca)
else
Forward models. For linear inverse prob- lems, we select bicubic super-resolution with scale factor 16, Gaussian deblur with kernel size 61 and sigma 5.0, and box in- painting where the masked reason is de- signed to encompasses the eyes and mouth of either animal or human. The forward operators are defined by following Kawar et al. (2022); Wang et al. (2022); Song et al. (2022); Chung al. (2023b). For non-linear inverse problems, we choose Fourier phase retrieval and gamma correction. For details of the forward operators, please refer to Appendix. For all tasks, we add measurement noise that follows the Gaussian distribution with zero mean and noise scale o = 0.01.
21-1 f- vuk-12oje + VT 10--16
end if
end for
"Please refer to Appendix A.4.
Dataset and text prompt. We prepare measurement-text sets where the text describes solution. The simplest way is to leverage the ground-truth class label as a text description. However, to further explore the capability of TReg reducing solution space according to given text description, we provide perceptually estimated monn-roonnd truth) classes as text cues. Here, we should carefully set proper class for measurement avoid ignoring the provided guidance.
We decide to use Food-10 101 dataset (Bossard et al., 2014) for quantitative evaluation since it contains complex patterns, leading to high ambiguity. Also, given that all images fall under the 'food' category, it is more straightforward to select an appropriate e aligned with the original caption. Specifically, we
For example, "dog" " for the mcasurement generated by haby image cannot effectively reduce solution space.

---

Published as a conference paper at ICLR 2025
"Bear"
Reconstrution
P
Recondtiucton
Reconsttrution
Unoersiity map
Westomment
69
Unceraiity
WW Th0
Figure 3: TReg effectively reduce ambiguity of solution with regularization. (a) Given measurement and text description. (b) Multiple reconstructions and pixel-wise variance without and with text regularization. (c) Variance measured over white dotted line on uncertainty map.
leverage 250 images from each of the "fried rice" and "ice cream" classes. For the case that given text description different from the original classes, we use "spughetti" for "fried rice" and "macarons" for "ice cream"" For the qualitative comparison, additionally AFHQ, FFHQ and LHQ datasets. Used text prompt is provided with reconstructed samples.
validation set of ImageNet,
Baselines, As the first attempt solve inverse problems using text-driven regularization, we establish baselines by combining 1) data consistency-preserving methods and 2) text-guided editing methods. Notably, these baselines involve sequential processing, while the proposed method stands out for its efficiency. eliminating the need for pre/post-hor techniques. For data consistency methods, we utilize the measurement itself and PSLD (Rout et al., 2024) which is LDM-based inverse problem solver. Regarding text-guided editing methods, we opt for state-of-the-art techniques, namely Delta Denoising Score (DDS) (Hertz al., 2023) and Plug-and Plug-and-Play diffusion (PnP) (Tumanyan al., 2023). Since PnP involves an inversion process, we handle it as two-stage approach. Also, we use PSLD with text guidance as our baseline by computing estimated noise via CPG. For the inpainting task, we utilize Stabie-Inpaint, a fine-tuned stable diffusion model for the inpainting task, and Repaint (Lugmayr et al., 2022) as our baseline. Finally, the Fourier phase retrieval task, we compared the results with and without the text prompt using our framework to emphasize the effect of text guidance in breaking symmetry. other words, for the baseline, we provided null text for the text condition.
4.2 EXPERIMENTAL RESULTS
Ambiguity reduction. To assess the effectiveness of TReg in reducing ambiguity in reconstruction through text regularization, we quantify pixel-level variance across multiple reconstructions from blurry measurements with and without text descriptions (Figure 3).² Specifically, for the reconstruc- tion without text description, we use a null-text Cg as a text description for the solution. All other components such as latent optimization (Section 3.1) and adaptive negation (Section 3.2) are retained.
As shown in Figure 3(b), TReg leads to consistent solutions corresponding to the given text description, while reconstructed images exhibit multiple solutions, such as a car the background or a bedroom, in the absence of text regularizaiton. This discrepancy is clearly observed in pixel-level variance in Figure 3(c). While solutions with TReg are successfully refined, some residual uncertainties observed, especially at head positioning or mouth shape. However, it is noteworthy that those residual uncertainties do not violate the data consistency and text description, and TReg is effective in resolving ambiguity via text.
Accuracy of obtained solution: use true class as c. TReg is an inverse problem solver that uses text description of solution refine the solution space. To evaluate the accuracy of obtained solution via TReg, we first prepare set of measurement-text pairs where the original class of measurement is given text. For both SR and Gaussian deblur tasks, we report PSNR and FID of reconstructions
"Specifically, repeat reconstruction 10 times with different random seeds for single measurement. The image sampled from ImageNet validation set.
7

---

Published as a conference paper at ICLR 2025
Measurement (Deblur)
TReg
Ground Truth
Measurement (9R)
TReg
Ground Truth
OS
Figure 4: Reconstruction by TReg where the original class is given as text description: "A photo of
<class>."
Class: Ice-Cream
SRx16
Deblur
Class: Fried-rice
SRx16
Deblur
Method
PSNR
FID 4
PSNR
FID L
Method
PSNR
FID 1
PSNR f
FID +
DDRM
16.98
268.5
17.41
223.2
DDRM
16.71
244.0
17.08
195.3
PGDM
14.95
260.5
13.87
295.8
PGDM
14.58
246.6
14.01
276.2
PSLD
17.73
280.1
21.89
213.6
PSLD
17.32
320.3
21.09
187.3
PSLD CFG
16.47
176.1
20.76
192.9
PSLD CFG
14.97
150.8
20.49
168.3
Our (wlo AN)
22.05
144.4
23.91
131.2
Our (wlo AN)
21.26
132.0
22.99
133.4
(w/ AN)
21.09
124.2
23.73
120.6
AN)
Table Quantitative result for text "ice cream".
Table 2: Quantitative result for text "fried rice".
Fridd
Speghem 15it126)
Ice Cream
Macaron (WW)16)
Fried
Spaghetti (Chblirr)
Ice Cheam Macaron (AAbbiur)
stages
Method
LPIPS L
CLIP sim
MSL
LPIPS
CLIP sim
LPEPS
CLEP sitt
MMSE
LPIPS
CLIP sim 1
MSE
Ohrs
I:760
0,313
RUNS
0.721
0.314
0.004
0.737
0.300
0.013
0L743
0.312
IU811
P21
8.756
0.265
0.020
0.743
0.203
0.000
0.708
0.274
0.113
0.750
11353
RU111
PSLD
0.756
0.263
80020
0.745
0.293
0:029
0.798
0.274
0.013
0.700
01252
0.011
PhP
0.826
0.251
0.808
0.239
0:005
0.298
0.259
0.015
0.252
012.88
0.011
PSLD+DDS
0.783
0:247
0.000
0.722
0.328
0.247
0.014
0.283
0.300
0015
D+PP
0.201
0.305
0.261
0.016
0.511
0.319
Table Quantitative evaluation of SRx16 and Gaussian Deblurring task. Mean values are reported. Bold: the best score, underline: the second best.
in Table and 2. Evaluation demonstrates the effectiveness proposed method compared to other diffusion-based inverse solvers. The proposed solver without adaptive negation tends to achieve higher PSNR than with adaptive negation, since it obtains blurry images with missing details. Thus, FID score improved with adaptive negation which is aligned with qualitative comparison in Figure 2. Also, TReg effectively reconstruct solutions for other dataset such as ImageNet (Figure 4) which implies the robustness of the proposed solver various image domain based on powerful diffusion prior. For more analysis, please refer to Appendix section G.
Accuracy of obtained solution: use different class as
Using TReg, we can find solution of inverse problem according to text prompt. To examine this property, we solve the aforementioned problems where the given text is different with the original class of the measurement. The proper solution should satisfy 1) data consistency with measurement and 2) alignment with given prompt. We evaluate the quality of obtained solution based on three metries: LPIPS, the mean squared error on measurement domain, namely y-MSE', and the CLIP similarity between reconstruction and given text prompt following the prior works image editing (Tumanyan et al., 2023; Hertz et al., 2023; Park et al., 2024). Note that there is no ground truth in this case, we do not evaluate PSNR or SSIM.
This equivalent data consistency loss, y A(zz)
8

---

Published as a conference paper at ICLR 2025
text
Super resolutioo (x18)
Gouction Doalur
Figure 5: Reconstructions when given text prompt differs from the original class.
Table 3 shows that TReg achieves superior performance compared to baselines. Specifically, TReg find solution with high quality while preserving data consistency, evidenced by lower LPIPS and y- MSE values. For some cases, PSLD gives lower LPIPS but solutions lose data consistency and exhibit large y-MSE values, which is also shown in the fourth column of Figure 5. TReg is comparable to or outperforms baseline methods, emphasizing that TReg is better than a straightforward application of image editing algorithms solving text-guided inverse problems. For more analysis, please refer to Appendix section H.
Direct application of the editing algorithm to the measurement, referred to as PnP, yields sub-optimal performance across all metrics. In the case of y-MSE, PnP achieves lower value due minimal alterations made to the measurement, as depicted the last column of Figure 5, Also, the performance of sequential approaches shows that reconstruction error is accumulated each stage, resulting in substantial errors in the final outcome. In contrast, TReg effectively integrates the given text prompt while preserving robust data consistency, thereby validating its efficacy. Meanwhile, we also compare the results of our method without text prompt. When text is replaced by null-text (third column in Figure 5), it provides a solution that aligns well with the given measurement but generates arbitrary structures. This result highlights the impact of TReg that reduces the ambiguity of inverse problem solving. PSLD with CFG guidance (fourth column in Figure 5) also fails appropriately reflect given text prompt, as there no consideration for text-guidance in solving inverse problems.
4.3 ADDITIONAL RESULTS ON NON-LINEAR INVERSE PROBLEMS
Although we propose solving latent optimization (10) using variable splitting for computational efficiency, any off-the-shelf optimizer could be employed to address, for example, non-linear inverse problems. We demonstrate in Figure that TReg improves the accuracy of solutions for non-linear problems through text regularization. Specifically, the results for Fourier Phase Retrieval emphasize that current diffusion-basee inverse problem solvers, such as Latent DPS, are still insufficient for breaking the symmetry caused by the loss of phase information. In contrast, text regularization via TReg successfully breaks this symmetry and consistently produce unique solution across multiple trials. Additionally, for non-uniform deblurring with gamma correction, TReg enhances reconstruction quality compared to Latent DPS. Notably, text conditioning through CFG in the latent DPS introduces unintended noisy structures, which are absent when using TReg.
5 RELATED WORK
Leveraging text prompts for inverse problem solving has been studied in several recent works. TextIR (Bai et al., 2023) incorporates text embedding from CLIP during the training of an image restoration network constrain the solution space in the of extreme inverse problems. DA-

---

Published as a conference paper at ICLR 2025
Reconstructions via TReg Iphotography face"
Ground Truth
Measurement
Reconstructions via LDPS
Fourler Phase Retrieval
Measurement
PSLD
TReg ('ice cream")
Ground
Non-unitorm w/ gammaa correction
MILA
MELA
MILA
MelA
Figure 6: Reconstructions for non-linear inverse problems: Fourier Phase retrieval and Non-uniform deblurring with gamma correction.
CLIP (Luo et al., 2023) takes similar approach of leveraging the embeddings and applies it to diffusion bridges. P2L (Chung et al., 2023d) proposes to automatically tune text embedding on-the-fly while running generic diffusion inverse problem solver (More discussions in Appendix C). Notably, the latter two methods (Luo et al., 2023; Chung et al., 2023d) are focused on improving the overall performance rather than constraining the solution space. TextIR has similar objective to TReg, but requires task-specific training. In this regard, to the best of our knowledge, TReg is the first general diffusion- inverse problem solver that does not require task-specific training, while being able to incorporate text conditions to effectively control the solution space in general inverse problems.
CONCLUSION
This study introduced a novel concept of latent diffusion inverse solver with regularization by texts, namely TReg. The proposed solver minimizes ambiguity in solving inverse problems, effectively reducing uncertainty and improving accuracy in visual reconstructions, bridging the gap between human perception and machine-based interpretation. To achieve this, derived LDM-based reverse sampling steps to minimize data consistency with text-driven regularization, consisting of adaptive negation. Specifically, to effectively integrate textual cues and guide reverse sampling, the paper introduced a null text optimization approach. Experimental results and visualizations demonstrated that text-driven regularization effectively reduces the uncertainty of solving inverse problems, and further enables text-guided control of signal reconstruction.
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation of Korea under Grant RS-2024- 00336454 and by the Institute for Information & Communications Technology Planning & Evaluation
10

---

Published as a conference paper at ICLR 2025
(IITP) grant funded by the Korea government (MSIT) (RS-2019-11190077 Artificial Intelligence Graduate School Program, KAIST).
REFERENCES
Yunpeng Bai, Cairong Wang, Shuzhao Xie, Chao Dong, Chun Yuan, and Zhi Wang. Textir: A simple framework for text-based editable image restoration. arXiv preprint axXiv23322.47336 2023.
Sam Bond- Taylor and Chris G Willcocks. co-diff: Infinite resolution diffusion with subsampled mollified states. arXiv preprint arXiv:2303. 18242, 2023.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative compo- nents with random forests. In European Conference on Computer Vision, 2014.
Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends@ in Machine learning, 3(1):1-122, 2011.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Condi- tioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.02935
2021.
Hyungjin Chung, Jeongsol Kim, Michael T. McCann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. CoRR, abs/2200.14687,, 2022a.
URL
485500/arXiv... 2209. 14687.
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye.
sellose--diffus--faeee Accelerating conditional diffusion models for inverse problems through stochastic contraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12413-12422, 2022b.
Hyungjin Chung, Jeongsol Kim, Sehui Kim, and Jong Chul Ye. Parallel diffusion models of operator and image for blind inverse problems. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6059-6069, 2023a.
Hyungjin Chung, Suhyeon Lee, and Jong Chul Ye. Fast diffusion sampler for inverse problems by geometric decomposition. arXiv preprint arXiv:2303.0m 055554 2023b.
Hyungjin Chung, Dohoon Ryu, Michael T McCann, Mare I. Klasky, and Jong Chul Ye. Solving 3d inverse problems using pre-trained 2d diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22542-22551, 2023c.
Hyungjin Chung, Jong Chul Ye, Peyman Milanfar, and Mauricio Delbracio. Prompt-tuning latent diffusion models for inverse problems. arXiv preprint arXiv:2310.011 2023d.
Hyungjin Chung, Jong Chul Ye, Peyman Milanfar, and Mauricio Delbracio. Prompt-tuning latent diffusion models for inverse problems. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pp. 8941-8967. PMLR, 21-27 Jul 2024. URL https: / http://doooceeeeee///ee/// mlr. press/v235/chung222 html.
Bradley Efron. Tweedie's i formula and selection bias. Journal of the American Statisticul Association, 2011.
Linchao He, Hongyu Yan, Mengting Luo, Kunming Luo, Wang Wang, Wenchao Du, Hu Chen, Hongyu Yang, and Yi Zhang. Iterative reconstruction based on latent diffusion model for sparse data reconstruction. arXiv preprint arXiv:2307.12070 2023.
Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. In Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 2328-2337, 2023.

---

Published as a conference paper at ICLR 2025
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop Deep Generative Models and Downstream Applications, 2021. URL https::/oppenrevieww forumm1dd::WAAKxYYYI.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration
models. Advances in Neural Information Processing Systems, 35:23593-23666 2022.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes.
arXiv preprint
arXiv: 14, 2013.
Suhyeon Lee. Hyungjin Chung, Minyoung Park, Jonghyuk Park, Wi-Sun Ryu, and Jong Chul Ye. Improving 3d imaging with pre-trained perpendicular 2d diffusion models. arXiv preprint arXiv:2303.08440. 2023.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint SxXiv:2104008665 2021.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11461-11471 2022.
Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Controlling vision-language models for universal image restoration. arXiv preprint arXiv:2310.01018 2023.
Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective solving inverse problems with diffusion models. arXiv preprint arXiv:2305.04331, 2023.
Geon Yeong Park, Jeongsol Kim, Beomsu Kim, Sang Wan Lee, and Jong Chul Ye. Energy-based cross attention for bayesian context update in text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevieee ew. net/forum? JN
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Herbert E Robbins. An empirical bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory, pp. 388-394. Springer, 1992.
Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (red). SIAM Journal on Imaging Sciences, 10(4): 1844))18044844, 2017.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp
Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241. Springer, 2015.
Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning, pp. 2256-2265. PMLR, 2015.
12

---

Published as a conference paper at ICLR 2025
Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving inverse problems with latent diffusion models via hard data consistency. arXiv preprint arXiv:2307.08123, 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guiee guided diffusion models for inverse problems. International Conference on Learning Representations, 2022.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020b.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1921-1930, 2023.
Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. In 2013 IEEE global conference on signal and information processing, pp. 945-9488 948. IEEE, 2013.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. CoRR, 10,022,,,,0490, 2022. URL https: //to////oiioooo www.ii.rr/1.... 48550/ ar Xiv. 2212. 2019.00490.
13

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
In this section, we provide further details on implementation of TReg. The code will be available to public on https://gtt http:://///ttbu.cooo////////// hotp://///Ree--neeeee//reeeee
A.1 FORWARD OPERATIONS
Box inpainting In the case of the box inpainting task, our intention is to employ universal mask that encompasses the eyes and mouth either animal or human. For this, we generate a rectangular mask based on averaged images across all data points within each dataset.
Fourier phase retrieval For Fourier phase retrieval, we obtain the Fourier magnitude by ap- plying the Fourier transform to the image. In the formula, the forward model is expressed as ~ N N(yl F P .gT), where F denotes the 2D Discrete Fourier Transform (DFT), represents oversampling, that is implemented by adding 256 zero- sero-padding to each side, and Xo denotes the clean image.
A.2 STABLE-DIFFUSION AND CLIP
We leverage the pre-trained Latent Diffusion Model (LDM), including auto-encoder and the U-net model, provided by diffusers. Due to the limitation of its modularity, we implement the sampling process for inverse problems by ourselves, rather than changing given pipelines. The Stable-diffusion v1.5 is utilized for every experiment this work, and the ViT-L/I4 backbone and its checkpoint is used for CLIP image encoder for adaptive negation.
A.3 CONIUGATE GRADIENT METHOD
In this work, apply the CG method to find solution the following problem:
Ax Aalæ (20je))
min
(18)
20
As the objective function is convex function, the solution x" should satisfy
-A (y- Axr) + A(x (D)())))) = 0,
(19)
where the coefficients are absorbed to A. Then, can formulate it as linear system
(AI + A AT A)x" xDi((oj1) + A
(20)
where A AI+ and (2)r) + +AT y. Thus, we can solve (20) by CG method. this work, we use 5 iterations of CG update with A le 4 for each time step if not explicitly stated otherwise.
A.4 CG UPDATE RANGE
The data consistency update with CG algorithms is applied for a subset of sampling steps as described by I in Algorithm of the main paper. Our empirical observation show that this partial data consistency update achieves better trade-off between the image reconstruction consistency and the latent stability. In the absence of the data consistency update, employ the deterministic DDIM sampling step by setting n78 0 or apply DPS gradient to improve the reconstruction quality. Overall, for inverse problems except the Fourier phase retrieval, we set the Network Function Evaluation (NFE) 200 and design I' (tlt mod 3 0, t 850)*, where mod denotes the modulo operation. For the Fourier phase retrieval, we set r = [t]t mod 10 = 0).
*The pre-trained Stable-diffusion uses T = 1000. Thus, 850 is equivalent to the sampling step with
NFF 170.
14

---

Published as a conference paper at ICLR 2025
Without DPS
With DPS
Measurement
A{%)
20
A(R))
Figure 7: Regardless of DPS guidance, we can obtain proper solution through TReg for super- resolution and deblurring. Original class: "ice cream", Given text prompt: "macaron", 2g represents obtained solution and A(ăo) denotes simulated measurement with Øo.
Algorithm 2 Inverse problem solving with TReg with DPS
Require: cB, Ø D.c.&. C, D, y. Tamg 2T N0, I)
for / E [T. do
if l r then
Zoje (zt VT arē (z:))/vā,
ē; Compute noise using (15)
Zo + D(20j))
Co(y) + Equation (12)
En(y) E(Eol(y)
20
sema + ā.- ( + (1 1211)
b Latent Optimization
2 + - sema + / 11-16
20
Cø f- Co 17 V sim (Togg ( cØ)
b Adaptive negation
else
201 (z, - VT are (z. /) oiv.
ē; Compute noise using (15)
2 + vā + VT 01116
2t-1 + 2/211 P: 2., AID (20):)) -
end if
end for
A.5 TREG DPS UPDATE
In the main manuscript, we only describe the pseudocode of TReg without DPS update for the readability. However, the algorithm could readily conduct additional DPS update as follow. The pseudocode could described as Algorithm We have used this additional DPS step for the box inpainting and Fourier phase retrieval task with step size Pt Va--1. For box inpainting task, set CFG scale we for DPS gradient to O, while we set CFG scale W2 = Wj in Fourier phase retrieval task. For other tasks, we found that reconstructions quality is promising even without DPS as shown in Figure 7.
A.6 CFG GUIDANCE SCALE
The classifier-free guidance (CFG) is defined as
ég E0(21, &. w(Eo(Zz, c, E0(21,
(21)
15

---

Published as a conference paper at ICLR 2025
where wi is scale for the guidance. In other words, w could be interpreted magnitude for the negation. Thus, we leverage w to control the extent of enhancement applied to a given text prompt. In fact, we can regulate the same feature by adjusting the null-text update strategy, including null-text update frequency and number of optimization iterations per time step. However, the CFG guidance scale is simpler and more intuitive than adjusting the null-text update strategy. Hence, we have used proper CFG scale while fixing the null-text update schedule l'.
For the main experiments with the Food101 dataset in the main paper, we use the default scale 7.5 for all results. For the Fourier phase retrieval problem, we set the CFG scale other results, CFG scale among (3.0, 4.0, 5.0, 7.5} provide robust performance for various datasets including FFHQ, AFHQ. LHQ (landscape dataset), and ImageNet.
4.0. For
A.7 OPTIMIZATION PROBLEM FOR FOURIER PIIASE RETRIEVAL
In contrast other inverse problems with linear operation, Fourier phase retrieval involves non-linear operation so we cannot leverage CG to solve the optimization problem in pixel space. Thus, we use Adam optimizer with learning rate le and 31 = 0.9, 32 = 0.999 to obtain the solution of
y - A(x) A|| zo
min
(22)
20
by setting À O. This induces additional computational costs solve the optimization problem. However, as analyzed in Section D, these costs are comparable to those of other baseline algorithms, while consistently achieving the reconstruction of a unique solution.
B CONVERGENCE ANALYSIS
In section 3.1, we optimize the full latent proximal objective by leveraging alternate variable splitting, where the data consistency (12) is optimized in pixel space and the text-conditional proximal opti- mization (13) is mainly solved in latent space. Here, we analyze the convergence of the proposed refinement process for a better understanding.
A direct theoretical convergence analysis is challeng- ing due to the complexity arising from the transition between pixel and latent spaces. However, an informative empirical analysis by evaluating the pro- gression of two objectives during the reverse sampling process: (a) data consistency measured in pixel space and (b) diffusion training objective (also known as the denoising score matching loss) with a given text prompt.
0.40
provide
0.35
606
0.30
Dale Consistency
500-
6.5
a00
0.00
a00-
6355
In formula, we plot (a) |/y A( and (b) ||e ca(2,, c. t) as in (24). The first objective indicates how well the solution aligns with the given measure- ment. The second objective represents how well the solution follows the marginal distribution of the pre- trained diffusion model, conditioned on the given text. We present results for two cases the main paper: where the given text prompt matches the true label and where it differs from the true label. For both cases, we plot the mean and standard deviation of these objectives, computed across 100 samples. Figure 8 and 9 shows that both objectives decrease significantly during the reverse sampling process, demonstrating that TReg empirically promotes convergence. Notably, the data consistency of TReg improves particularly in the later sampling phase, which in line with the interpolative constant setting (ā,-1 = C/(C +)) in (13).
200
0.30
0.05
0.00
3000
C00
000
400
200
Uiffusion time
Figure 8: Data consistency and DSM loss during reconstruction. Mcan + Std for 100 samples are plotted.
C COMPARISON TO P2L
Here, we compare TReg with P2L (Chung et al., 2023d) in detail and highlight the key differences. As mentioned in Sec. 1, note that P2L. is orthogonal to approach, despite their apparent
16

---

Published as a conference paper at ICLR 2025
TReg
TReg
1-B, Ie Eg(6..
A(ann) )II
600
400
200
1000
800
600
400
200
1000 800
600 400
200
Diffusion
Diffusion
Figure 9: Data consistency and Text-conditioned score matching loss for TReg and P2L. Mean + Std for 100 reconstructions are plotted. Task: Gaussian Deblurring. Original Class: "Risotto", Target Text: "French Fries". P2L initializes the prompt embedding with the target text ("French Fries").
similarities. Specifically, P2L. treats the (null-text) prompt as an additional parameter to improve the data consistency, optimizing it as:
Ci = arg min yyy
(23)
C
which corresponds to cq.( of Chung et al. (2023d).
Unlike TReg, P2L does not guide outputs toward a specific mode aligned with semantic linguistic conditions. It lacks support for Classifier-Free Guidance (CFG) and relies solely conditional scores. While both methods leverage prompt embeddings, they do so in fundamentally different ways: P2L focuses data consistency, whereas TReg reduces the solution space by incorporating rich perceptual estimates of noisy measurements in the form of linguistic descriptions.
Several empirical analysis verify these distinctions (see Sec. G). First, while P21. initialize null text embedding in general, for the comparison, we adapt P2L. as a target text-based baseline by providing ground-truth textual descriptions. Quantitatively, TReg outperforms P2I. even under these ideal scenarios (see Sec. G for more discussions). We remark that P2L requires twice the neural function evaluations (NFEs) compared to TReg, as it recomputes Tweedie estimates after each prompt update (refer to Table 4 and Sec. D for details).
Fig. 9 further highlights the fundamental gap between TReg and P2L. Specifically, we monitor the normalized text-conditional score matching loss (i.e. score matching distillation loss (Poole et al., 2023)) throughout the reverse sampling for both TReg and P2I.:
1 0i ||e co(Vorz V1 o,
((z)
(24)
1 at c0(2,, 1)12.
(25)
This evaluates how well the intermediate solutions progressively align with the clean data manifold and the corresponding text condition c. As shown in Fig. 9 (right), TReg consistently achieves better progressive alignment with textual conditions, particularly during the early stages sampling. These early stages are critical for the overall spatial layout, eventually establishing semantic textual alignment. Overall, TReg demonstrates qualitative and quantitative advantages over P2L by achieving more efficient sampling, better data consistency, and textual alignment.
D RUNTIME ANALYSIS
We propose text regularization by formulating a proximal optimization problem in the latent space. Thus, additional time solve the optimization problem is required compared to reverse diffusion process. Notably, TReg does not require excessive computation time although we introduce two- step optimization approach. This is because employ accelerated optimization method, like
17

---

Published as a conference paper at ICLR 2025
PSLD
PSLD w/ AN
P2L
TReg
Linear
1.47
1.47
2.00
0.26
Non-linear
1.12
1.12
1.09
Table 4: Wall-clock runtime (min) each algorithm for NFE=200.
Measurement
"a car
photo of cat"
photography oi cat"
Figure 10: Reconstruction by TReg with various text prompts. Super-resolution (x16) task on AFHQ-cat.
conjugate gradietn (CG), and derive a closed-form solution for subsequent problem. As a result, TReg demonstrates superior efficiency compared the baseline algorithm, shown in Table 4. To evaluate the efficiency, we measure the runtime for Gaussian deblurring task and Fourier phase retrieval with PSLD and TReg, as representative tasks for linear and non-linear inverse problem. Specifically, we did not use the fixed-point constraint of PSLD for the non-linear inverse problem, as AT is not properly defined in this case. Table 4 shows that TReg requires significantly shorter time to obtain solutions for linear inverse problem. Also, for the non-linear problem where the off-the-shelf optimizer is leveraged, TReg faster than PSLD since the optimizaion variable is clean latent estimate so it does not require back-prop proppaaaaonatoonn through denoising UNet. Furthermore, the optimization variable for adaptive negation is limited to text embedding, which incurs minimal additional computational cost. Consequently, the runtime for adaptive negation is negligible.
E ABLATION STUDY ON TEXT PROMPT
We conduct an ablation study on text prompts to assess whether the reconstructions vary based on the given text. Specifically, for the AFHQ-cat validation set, we examine the solutions for super- resolution task obtained by TReg using the prompts "a cat," "a photo of a cat", and "a photography of a cat." As shown in Figure 10, the reconstructions are not highly sensitive to grammatical variations in the text prompts.
F ROBUSTNESS TO JPEG ARTIFACTS
A practical scenario for inverse problems involves measurements that include additional noise beyond Gaussian noise, such as artifacts introduced by JPEG compression. In this section, we evaluate the robustness of TReg to JPEG artifacts. Specifically, we apply a JPEG compression algorithm² to
'We use the forward operation defined by DDRM (Kawar al., 2022), http:://gituub.omm. fhhjat-kwwwr/ddrmm jpeg.
18

---

Published as a conference paper at ICLR 2025
Goussian Buur
Peecn
Recon
Rlacon
Recon
Figure 11: Reconstructions in the presence of JPEG compression artifacts applied after Gaussian blur on AFHQ-dog validation set.
FFHQ (SRx16)
FFHQ (Deblur)
FFHQ (Inpaint)
AFHQ (6Rx16)
AFHQ (Deblur)
AFHQ (Inpaint)
Method
PSNR
FID 4
PSNR
FID
PSNR
FID
PSNR
FID
PSNR
FID
PSNR
FID
PSLD
20.01
142.8
24.82
59.41
19.76
60.97
16.48
113.4
20.52
125.5
16.93
104.7
15.67
22.61
90.72
20.58
125.3
P2L
21.94
72.02
23.03
91.15
16.84
85.32
19.99
121.7
20.96
85,80
16.07
138.4
TReg
22.60
82.71
24.82
40.24
19.95
66.93
19.60
37.13
21.13
35.47
17.39
51.97
Table 5: Quantitative evaluation on FFHQ and AFHQ validation sets. The ground-truth class labels
given as text descriptions. Bold represents the best and underline denotes the second best.
the measurement and then add Gaussian noise. For JPEG compression, vary the quality factor (QF) at 50, 30, and 10, where a higher quality factor indicates less compression. note that do not include JPEG compression in our forward operation, the purpose of this study to assess TReg's robustness to measurement noise. For the prompt, we use "a photo of dog". As shown Figure 11, TReg demonstrates robustness up Q30, providing nearly identical reconstructions. At QI0, however, significant noise is introduced into the measurement, resulting in degraded reconstructions. For such extreme JPEG artifacts, incorporating JPEG compression into the forward operation could be considered.
important
G ADDITIONAL EVALUATION WITH GROUND-TRUTH CLASSES
The main experiments focus on practical scenarios where perceptually estimated descriptions simulate real-world usev-provided cues better reconstruction. this section, for completeness, we also evaluate additional scenario with 'ground-trutee truthh textual descriptions of the noisy measurements. Using benchmarks like FFHQ, AFtQ-cat, and AFHQ-dog, where ground-truth class labels (c.g.. 'dog", 'cat') are available, P2L is adapted text-based baseline by initializing prompt embeddings with these ground-truit -trth label deseriptions.
Table 5 and shows that TReg outperforms other baselines, including the computationally expensive P2L. Fig. 10. 11, 12, and 13 further illustrate the qualitative advances. These results highlight that TReg utilizes textual information
effectively and could achieve even better performance with reliable descriptive inferences of noisy measurements.
ImageNNet (8Rx16)
ImageNet (Dehlur)
ImageNet (Inpaint)
Method
PSNR 1
FID
PSNR t
FID J
PSNR
FID )
PSLD
18.04
170.5
20.97
115.9
17.41
90.13
P2L
18.62
141.1
19.58
117.0
15.94
119.3
TReg
19.71
69.65
20.66
55.92
18.11
50.67
Table Quantitative evaluation on ImageNet Ik validation sets. The ground-truth class labels are given text descriptions. Bold represents the best and underline denotes the second best.
19

---

Published as a conference paper at ICLR 2025
Measurement
Ground Truth
TReg
P2L
PSLD
PSLD+CFG
Figure 12: Qualitative comparison for super-resolution (x16) task FFHQ
H ADDITIONAL EVALUATION WITH ESTIMATED CLASSES
To further demonstrate whether TReg reliably and generally reconstruct "perceptually plausible" solutions from a single noisy measurement with text description, we conduct additional evaluation on ImageNet validation set which contains 1000 classes.
Specifically, we construct a validation set from ImageNet comprising image-text pairs where the text description differs from the original class. First, we constructed 1k ImageNet validation encompassing all ImageNet classes, following the approach used in P2L. For each validation sample, the ground-trus class known. Next, to identify "perceptually plausible" text description, we measured pairwise I.PPS for all noisy measurements simulated from the 1k ImageNet validation set and selected the target text description corresponding to the class label with the lowest I.PIPS samples. This process resulted in 1k ImageNet validation set with both original and target classes (e.g. "great white shark" and "albatross", see Figure 16). Finally, we solved the inverse problem on this validation set using the experimental setup outlined in the main experiment.
Table demonstrates that TReg effectively reconstructs solutions aligned with the given target class while satisfying data consistency, evidenced by improved FID, CLIP similarity scores, and yMSE. In contrast. baseline methods exhibit significant trade-offs among these metrics. For example, PSL.D achieves the best FID in the deblurring task, but its reconstructions fail align with the
20

---

Published as a conference paper at ICLR 2025
Measurement
Ground Truth
TReg
P2L
PSLD
PSLD+CFG
Figure 13: Qualitative comparison for deblurring (Gauss) task on FFHQ
given text description, as indicated by low CLIP similarity. Similarly, PSLD CFG the SR task achieves higher CLIP similarity but fails to maintain data consistency. In comparison, TReg delivers better or comparable performance across metrics. These results highlight the versatility text regularization with TReg in solving inverse problems for wide range of natural images, extending beyond categories like person, cat, and dog.
I ADDITIONAL RESULTS
I.1 TEXT-GUIDED BOX INPAINTING
We conduct qualitative comparison for box inpainting task. As illustrated in Figure 17 (a), TReg reconstructs image properly according to the given text prompt with higher fidelity compared to baseline methods. From Figure 17 (b), repeatedly observe that text regularization enables the discovery of solutions as intended. It allows the discovery even rare solutions, such as cat with green eyes. Additionally, in Figure 17 (c), TReg demonstrates its capability to solve the inpainting problem by composing concepts. other words, TReg can narrow down the solution space by utilizing not only singular concept but also multiple concepts simultaneously.
21

---

Published as a conference paper at ICLR 2025
Measurement
Ground Truth
TRag
P2L
PSLD
PSLD+CFG
Figure 14: Qualitative comparison for super-resolution (x16) task on AFHQ
Furthermore, we provide uncurated reconstruction results in Figure 18 to demonstrate the robustness of the TReg. We provide two box inpainting scenarios: rectangular inpainting task on the AFHQ- dog validation set with the text prompt "a photo of a dog with glasses", and square inpainting task on the AFHQ-cat validation set with the text prompt photo dog". For both cases, TReg successfully fills out the masked regions according to provided text descriptions.
1.2 SUPER-RESS RETERRRESOLUION AND DEBLURRING
TReg is zero-shot inverse problem solver based on diffusion prior trained on large scale of text-image dataset, implying its robustness to diverse of texts and image domain. Thus, we further investigate the robustness of TReg in reducing solution space based on given text prompt. Figure 19 illustrates reconstructions with various text prompts. Regardless of data domain, TReg effectively finds the solution by preserving data consistency and reflecting text prompt. Furthermore, the result directly shows the existence multiple solutions that satisfied given forward model, which is an evidence of the ill-posedness nature of the inverse problem. In addition, examine the reconstruction for inverse problem defiend on FFHQ dataset with text prompts "baby face" and "adult face". As shown in Figure 20, the proposed method successfully reconstructs images according to the given prompt, while the baseline method (sequential approach of PSLD and DDS) shows inferior reconstruction quality.
22

---

Published as a conference paper at ICLR 2025
Measurement
Ground Truth
TRag
P2L
PSLD
PSLD+CFG
Figure 15: Qualitative comparison for deblurring (Gauss) task on AFHQ
LIMITATION
While TReg demonstrates performance in breaking the symmetry in the Fourier Phase Retrieval problem, its effectiveness is contingent upon the text prompt. For instance, when the true image features a complex or colorful background, a simple text prompt such as "a photography of a face" may not suffice to reconstruct unique solution without residual symmetry.
our scenario, we assume that the user can access the category of the dataset (e.g., FFHQ for faces). However, in real-world scenarios, it may be challenging to derive more informative text prompts solely from measurements with severe degradation. In general, finding suitable text prompts remains open problem in text regularization for solving inverse problems.
23

---

Published as a conference paper at ICLR 2025
Clean Image
Messurament
PSLD GFG
True class: "magple"
"borza"
"slbeerooss
True clsas: "dalmation'
"oardigen"
"Inriisha"
Figure 16: Reconstructions for deblurring task when given text prompt differs from the original class. ImageNet validation set (1k images) is leveraged.
(
Mescurement
mlg
Shablelnpent
Sable-Rapahnt
PGI D-CFG
10
Messursment
"Blue eyes"
"Groen eves"
Glassoo"
Figure 17:
Representative results for box inpainting task. (a) TReg is better finding solution according to text with high-fidelity. (b) Text regularization helps to reconstruct as intended. (c) TReg possible to reconstruct based multiple concepts.
24

---

Published as a conference paper at ICLR 2025
Measurement
Reconstruction ("a photo of with glasses")
Measurement
Reconstruction ("a photo of dog')
Figure 18: Uncurated results for box inpainting task. (Top) Inpainting of eye region of AFHQ-dog with the prompt "a photo of dog with glasses". (Bottom) Inpainting face region of AHFQ-cat with the prompt "a photo of dog".
25

---

Published as a conference paper at ICLR 2025
Gaussian Debluiring
Supar-resokuiinn
Veesurement
"Waffee
"Pizza"
"Tacos"
Measurement
"Whoee grain pasta'
"Baked beans"
"Cherry muffin'
Food
"Chcose plattor"
"Panro pasta'
Spaahoott
Jally boans
"Granela"
"Pino cones
"Pacans"
"Cookins"
Measurement
"Photography"
"Monet"
"Watercolor"
Messurement
"Photography"
"Monel"
"Watercolor"
Landscapo
Messurement
"Goats"
"Trees"
'Dens'
Measurement
KFC
"Shop"
"Pizza
ImageNat
Measurement
'House"
"Cows"
"Rock
Messurement
"Fox"
"Dog"
"Worr
Figure 19: Reconstructions for diverse domain.
26

---

Published as a conference paper at ICLR 2025
Super-resolution (bicubic x 16)
Debluringg (Gaussian)
Measuremeee remmeeeeenttt
Measurement
"Adult face
"Adult face"
TReg
TReg
"Baby face"
"Baby
"Adult face
"Adult
PSLD DDS
PSLD + DDS
"Baby lace
"Baby lace
Figure 20: Reconstruction result for SR (x16) and Deblur (Gaussian) task on FFHQ dataset. Our
method outperforms the existing methods (PSLD+DDS) in terms of the reconstruction quality.
27


Reviews:
[{"review_id": "QYVoIEo9hH", "reviewer": "Reviewer_CtoC", "summary": "This paper introduces a novel method for solving inverse problems using diffusion models, with a focus on reducing ambiguity in the solutions.\nThe authors propose a latent diffusion inverse solver that incorporates regularization by texts (TReg), which applies textual descriptions of the expected solution during the reverse sampling phase. \nDuring training, this text guidance is dynamically adjusted through null-text optimization for adaptive negation.\n\nOverall, I think this is a good paper.", "questions": "please refer to weakness part.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* this paper introduces an explicit regularization term during training via text-driven regularization (TReg)\n* the proposed TReg effectively addresses the challenge of ambiguity in inverse problems\n* this paper further introduces an adaptive negation to dynamically adjust the influence of textual guidance\n* the paper is well written and easy to follow, extensive in main text and supplementary demonstrate the effectiveness of the proposed method for diffusion inverse solvers", "weaknesses": "1. lack of overall results over whole dataset. all the experiments results are shown in visualizations or subset results of specific classes.\n2. i notice in both table 1 and table 2, PSNR scores are higher without adaptive negation, do authors have any analysis or intuition about this results?\n3. (this might not be a weakness, just naturally curious) in the experiments, the text regularization is only tested with class name, how about the results using some natural captions (such as generated with BLIP or GPT)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kai5foOJKq", "reviewer": "Reviewer_vYBr", "summary": "This paper introduces TReg (Regularization by Text), a method for solving inverse problems in image processing using latent diffusion models by conditioning on text.  TReg leverages user provided textual descriptions to guide the image reconstruction process.\n\nPrior diffusion-based inverse solvers often struggle with ambiguity, as multiple different images can produce the same degraded output. TReg addresses this by incorporating textual prompts as a form of regularization. The authors frame the problem as a text-conditioned latent optimization, where the goal is to find a solution that is both consistent with the measurements and aligned with the provided text description.\n\nThe authors introduce the following:\n* A text based regularization term that encourages the reconstructed image's latent representation to be close to a text-conditioned denoised estimate. This helps guide the sampling trajectory towards solutions that are semantically aligned with the text.\n* To further refine the textual alignment, this method uses a novel \"null-text optimization\" technique. This dynamically adjusts the influence of the textual guidance during the reverse diffusion sampling process, suppressing unintended signal components.\n\n\nThe authors also discuss limitations, particularly the challenge of finding suitable text prompts for some challenging inverse problems where the problem is extremely ill posed.", "questions": "See above", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall the paper is well written, provides clear motivation, and the authors perform reasonably comprehensive experiments. The authors do a good job in their logical flow and provide pretty clear explanations of the proposed method and experimental setup. Figures look nice.\n\nIn terms of novelty, they authors introduce a reasonably novel approach to solving inverse problems by incorporating text-based regularization into latent diffusion models.  This addresses a significant limitation of existing solvers, mostly their struggle with ambiguity, and mitigates the need for task specific training. \n\nThe proposed method is technically sound, with a clear mathematical formulation and well-defined optimization procedures.  The integration of LDPS further enhances the method's capabilities.", "weaknesses": "* `Line 071` In my view the connection to the human brain is quite weak, not sure why the authors put this in the paper...\n* I would like to see additional, none cherry picked outputs. In my experience using diffusion models to recover images can often fail in very strange ways. I am specifically interested in image inpainting results, which nearly always result in boundary artifacts.\n* I would like to see experiments in the presence of measurement noise, for example with JPEG compression artifacts after gaussian blurring.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vMK17s4KYL", "reviewer": "Reviewer_2SWh", "summary": "This work uses diffusion models with text guidance in order to solve inverse problems. Inverse problems (e.g., in-painting or deblurring) have been recently tackled using diffusion models, however, their ill posedness nature makes resolving some ambiguities still challenging. The authors propose to leverage a user bias via a text prompt in order to guide the diffusion process towards a solution that is aligned with the description provided. The proposed framework uses a latent diffusion model with classifier-free guidance as a core framework with two innovations compared to previous works: (1) a term that induces alignment between the latent being optimized and that conditioned through the text prompt, and (2) a dynamic null-text adaptation. (1) is the distance between the latent embeddings that are being optimized and those obtained with the text used as guidance (in a classifier free guidance algorithm). (2) The null-text (which usually is the representation provided without any guidance) is updated dynamically in order to be aligned with the CLIP embeddings of the text provided by the user, incentivizing in this way the embeddings to continue to learn new concepts by moving the null-text closer and closer to the text description. \n\nThe authors show results on super resolution, deblurring and Fourier Phase Retrieval problems and compares with some baselines such as DDRM, PGDM, PSLD, and PnP.", "questions": "Main questions (answering these questions could affect the recommendation)\n\n- Could the authors provide information about convergence of the alternate direction method? \n- Could the authors provide quantitive evaluation averaged across full datasets (as done in state of art works, e.g., P2L) and include relevant baselines, as mentioned above P2L should be included. Mokady would also be a good comparison for inpainting results. \n\nOther questions (this are more curiosity driven questions, it is unlucky the answers here will change the recommendation)\n- In figure 2 authors show that thanks to the adaptive negation strategy the generated image is less noisy. It is unclear however why this would be the case. Could it be that without adaptive negation one could reach a similar results with more iterations?\n- Due to (18) does it mean that the VAE needs to have the same dimensionality of CLIP embeddings?\n- “The proposed solver without adaptive negation tends to achieve higher PSNR than with adaptive negation, since it obtains blurry images”. Could the Structural similarity index measure (SSIM) be a better metric in this case?\n- For the phase retrieval experiments: the error in the LDPS reconstruction seem to go beyond the ambiguity of symmetries… Do the authors have any intuition about why the quality of LDPS reconstruction is so low compared to TReg?\n- How sensitive is this frameworks to slight variation of the text? This would be an interesting discussion of the paper. The authors use “A photography of …” how much do results change if instead they used “A photo of xxx” or simply “xxx”?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper tackles an interesting problems with many potential applications. \n\nWhile the individual ingredients are not completely new, even in the context of inverse problems (the use of prompts and the idea of solving the problem using alternate direction method have been explored by Chung et al in “PROMPT-TUNING LATENT DIFFUSION MODELS FOR INVERSE PROBLEMS”; the optimization of the null-text has been explored by Mokady et all in “Null-text Inversion for Editing Real Images using Guided Diffusion Models”) the overall framework and the way these ingredients are used is novel and seems to lead to compelling results.", "weaknesses": "The main weaknesses of this work are in the lack of quantitative evaluations on full datasets, lack of some important baselines and the clarity of the writing could be improved. See below for more detailed comments.\n\nEvaluation: all tasks presented are obtained on specific classes rather than being averaged across full datasets as it is done in other the state of the art works (e.g., P2L). Specifically:\n- Ambiguity reduction: it is unclear if the quantitative evaluation in Figure 3 is an average over a dataset or simply the results of the image shown (bear). It is also unclear from which source this image comes from. Maybe it is ok since this is an interesting experiment but not the core? In any case more clarity is needed.\n- Accuracy of obtained solution: use true class as c. Results in  Table 1 and Table 2 show quantitative measurements (great) for two tasks (super resolution and de-blurring) but only on two classes “Ice-cream” and “fired rice” rather than averaging across full datasets.\n- Accuracy of obtained solution: use different class as c. Results shown in Table 3 are for only two classes using two different C labels (fried rice to spaghetti, and Ice Cream to Macaroons). It is understandable to show selected classes in Figures but the quantitative evaluation should be reported on a larger datasets (with mean and standard deviations).\n- Results on non-linear inverse problems. These results are only qualitative shown on 4 different images (Figure 6).\n\nThis work shares a lot of similarities with P2L (Chung et al.) so I was expecting P2L to be among the baselines. In the intro when talking about P2L the authors say “However, this method primarily focuses on data consistency and lacks robust alignment with textual prompts.” It would be great to support this statement with quantitative results.\n\nFor the alternate directions method there should be some additional information about convergence. Being an iterative method one would expect to see some analysis about it but I did not find any information. \n\nThe work of Mokady et al also shares the idea of adaptive negation (but with different application and implementation) so for Inpainint experiments (which are suitable for Mokady) this would also be a useful baseline.\n\nWriting: The paper is not alway easy to follow. The readability of the manuscript would benefit from some re-writing. For example \n- Background session could be summarized and moved to the appendix since these are all known concepts.\n- The additional space gained should be used to provide more details about the proposed method. Here are some examples of things that could be clarified but there are more throughout the text:\n    - Probably the most important: authors should mention that the proposed update is scheduled to be performed every so many iterations  end not always. This is somewhat hidden in the “range” argument of algorithm 1 but it becomes clear only reading the appendix. I strongly encourage the authors to bring this discussion into the main paper.\n    - Consider moving the inpaiting results (with comparison with Mokady) in the main paper.\n    - In Eq (13) and (14) there is the term A(x). Given that A is an “imaging system” but it is not clear the role it plays in practice when solving a specific problem. I would recommend the authors to provide some information about the role of the forward operator when discussing section 3.1.\n    - “Here, ζ,γ are empirically chosen to satisfy¯αt−1 = ζ/(ζ+ γ).” Please provide an explanation why satisfying this equation is important.\n    - …\n- Since Figure 1 is even before the abstract the reader should be able to understand it without reading anything but the caption and looking at the figure itself. Consider increasing the details in the caption or moving the figure in a place where the reader has sufficient information to understand all of the examples.\n- I find the abstract lacking important details. It should encourage a reader to continue the reading but in order to do so the more details  need to be presented. Here some potential improvements:\n    - ``ambiguities in measurements or intrinsic system symmetries.’’ consider adding a concrete example of these problems.\n    - define what adaptive negation is  or avoid adding a non-defined term.\n    - ``Our comprehensive experimental results’’ consider telling the reader which are the most important quantitive results so that it encourages the reader to continue.\n- In the experimental section consider breaking down the baselines for each experimental task. Currently they are presented all at once at the beginning irrespective of the tasks and I find this to be too much information to process and too far from where it is useful.\n- The authors used the prompt “A photography of a ….”. This might not be grammatically correct. Photography is the art of creating photos or the process of creating photos. A better prompt would  be “A photo of a…” or “A photograph of a …”. I would be curious to know if this changes the results.\n- Tweedie’s formula should be cited.\n- From (10) to (13) the terms are swapped. Please consider maintaining consistent order.\n- While the fact that the work uses a pre-trained VAE is mentioned in the main paper, I think the fact that also the diffusion model is pre-trained is not explicitly mentioned before the appendix. Consider clarifying this aspect in the main paper.\n- The following sentences are unclear, consider rephrasing them: \n    - “We also prepare a situation that different class label is given as a text description. Here, we should carefully set proper text prompt for measurement to avoid ignoring the provided guidance”. \n    - “For the case that given text description is differnet from the original classes, we use “spaghetti\" and “macarons\", respectively.”. There is a typo (differnet) but beyond that it is unclear what this sentence means.\n    - “For data consistency methods, we utilize the measurement itself “. What is the measurement?\n    - “TReg leads to a unique solution corresponding to the given text description”. Maybe “TReg leads to similar solutions.”? Possibly one could say “semantically unique and aligned with the text”?\n    - “ This discrepancy is clearly observed in pixel-level uncertainty in Figure 3(c). “ I suppose this is the variance computed across the 10 reconstruction? Consider mentioning explicitly how uncertainty is computed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)