Regularization by Texts for Latent Diffusion Inverse Solvers

## Abstract

Abstract The recent advent of diffusion models has led to significant progress in solving inverse problems, leveraging these models as effective generative priors. Nonetheless, there remain challenges related to the ill-posed nature of such problems, often due to inherent ambiguities in measurements or intrinsic system symmetries. To address this, drawing inspiration from the human ability to resolve visual ambiguities through perceptual biases, here we introduce a novel latent diffusion inverse solver by regularization by texts (TReg). Specifically, TReg applies the textual description of the preconception of the solution during the reverse diffusion sampling, of which the description is dynamically reinforced through null-text optimization for adaptive negation. Our comprehensive experimental results demonstrate that TReg successfully mitigates ambiguity in the inverse problems, enhancing their effectiveness and accuracy.

###### Keywords:

## 1 Introduction

Consider a given forward measurement process:

$$ ${\bm{y}}={\mathcal{A}}({\bm{x}})+\bm{\epsilon}$ (1) $$

where ${\mathcal{A}}:{\mathbb{R}}^{m}\mapsto{\mathbb{R}}^{m}$ describes imaging systems, ${\bm{y}}\in{\mathbb{R}}^{n}$ and ${\bm{x}}\in{\mathbb{R}}^{m}$ represents the measurement and the true image, respectively, and $\bm{\epsilon}\in{\mathbb{R}}^{n}$ denotes the measurement noise.
Then, an inverse solver attempts to recover ${\bm{x}}$ from the measurement ${\bm{y}}$.

Unfortunately, the majority of the inverse problems are ill-posed, implying that many different visual inputs produce the same measurements as a significant amount of visual information in the original input is lost during the measurement process. For example, Fourier phase retrieval, a problem of reconstructing the phase information of a signal from only its Fourier intensity measurements, is a notorious example of an ill-posed inverse problem due to the intrinsic system symmetries such as shift, rotation, and flips.
So when inverting such systems, an input is mapped to multiple
symmetry-related outputs.

Classically, regularization techniques have been extensively studied to address the ambiguity in inverse problems.
For example, sparsity ($L_{1}$ norm), total variation (TV), and regularization by denoising (RED) have been widely used for regularization by exploiting the statistical properties of natural images  [, , ].
With the advent of the diffusion model, a new family of Diffusion Inverse Solvers (DIS) [, , , ] have been proposed with superior reconstruction performance. The main idea is to leverage the power of the diffusion model that has learned the score function of prior distribution to reduce the ill-posedness of the inverse problem.

While diffusion models have enabled substantial progress in inverse problems, it is important to note the persistent disparity between human perception and DIS.
For example, in Fourier phase retrieval, Diffusion Posterior Sampling (DPS) [, ] can recover the true signal much better than traditional approaches, but they are still not able to completely break the intrinsic symmetry as illustrated in Figure 1(a). This is because the diffusion prior is based on image statistics,
and image statistics alone might not be sufficient to break the symmetry and ensure a unique solution.
In contrast, the human visual system easily distinguishes whether the obtained solution is semantically meaningful or not. This ability likely stems from perceptual biases obtained through our experience with natural scenes, particularly in tasks like facial recognition.
In fact, perceptual bias plays a crucial role in how we perceive and understand the world around us, especially in situations where visual information is incomplete, ambiguous, or open to multiple interpretations.
However, no equivalent mechanism in DIS currently applies these preconceptions during reconstruction.

To bridge the gap between human perception and diffusion inverse solvers (DIS), here we introduce a new concept, Regularization by Text (TReg), by utilizing latent diffusion models. TReg specifically employs textual descriptions reflecting the preconceived notion of the desired outcome during the reverse sampling phase. However, a straightforward application of a text prompt reflecting these preconceptions is inadequate, as the intermediate images from reverse samples vary in noise levels. To effectively integrate the textual form of the preconception and guide the reverse sampling, we therefore propose an innovative adaptive negation method by null text optimization throughout the reverse diffusion sampling process. This approach dynamically adjusts the influence of the textual guide, ensuring it aligns with the evolving state of the reverse sampling.
For example, TReg with regularization prompt of "photography of face" can break the symmetry in Fourier phase retrieval and consistently obtains a unique true solution, as shown in Figure 1(a).
Moreover, our method is so general that it can be applied for any inverse problem in a zero-shot manner, by
reducing the uncertainty and improving accuracy, as shown in 512x512 images in Figure 1 (b-c).

## 2 Background

### 2.1 Latent diffusion model

Image diffusion models that operate on the pixel space are compute-heavy.
So the latent diffusion model (LDM) [] is a class of diffusion-based generative models [, , ], where the diffusion process is operated on low dimensional latent space instead of the pixel space.
Specifically, the LDMs are trained as a variational autoencoder by maximizing the evidence lower bound (ELBO) [, ].
Thus, the latent is represented as:

$$ $\displaystyle{\bm{z}}={\mathcal{E}}_{\bm{\phi}}({\bm{x}}):={\mathcal{E}}_{\bm{ \phi}}^{\bm{\mu}}({\bm{x}})+{\mathcal{E}}_{\bm{\phi}}^{\bm{\sigma}}({\bm{x}}) \odot\bm{\epsilon},\quad\bm{\epsilon}\sim{\mathcal{N}}(0,\bm{I}),$ (2) $$

where $\odot$ denotes the element-by-element multiplication, and ${\mathcal{E}}_{\bm{\phi}}^{\bm{\mu}},{\mathcal{E}}_{\bm{\phi}}^{\bm{\sigma}}$ are parts of the encoder that outputs the mean and the variance of the encoder distribution.
The resulting pixel domain representation can be obtained by

$$ $\displaystyle{\bm{x}}={\mathcal{D}}_{\bm{\varphi}}({\bm{z}})$ (3) $$

where ${\mathcal{D}}_{\bm{\varphi}}$ is the decoder.

Then, the forward diffusion starts from the latent representation for the clean image ${\bm{z}}_{0}={\mathcal{E}}({\bm{x}})\in{\mathbb{R}}^{d}$ and generates samples ${\bm{z}}_{t}=\sqrt{\bar{\alpha}_{t}}{\bm{z}}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}$ using VP-SDE formulation.
Accordingly, the neural backbone $\bm{\epsilon}_{\theta}(\cdot,t)$ is trained to estimate the noise $\bm{\epsilon}$ from ${\bm{z}}_{t}$ by solving the following optimization problem:

$$ $\min_{\theta}\mathbb{E}_{{\mathcal{E}}_{\bm{\phi}}({\bm{x}}),\bm{\epsilon}\sim {\mathcal{N}}(0,\bm{I}),t}\left[\|\bm{\epsilon}-\bm{\epsilon}_{\theta}({\bm{z} }_{t},t)\|^{2}_{2}\right],$ (4) $$

where the neural network $\bm{\epsilon}_{\theta}$ is commonly selected as time-conditional UNet []. Notably, the optimal solution $\bm{\epsilon}_{\theta}^{*}$ of (4) can be used to derive a score function as $\nabla_{{\bm{z}}_{t}}\log p({\bm{z}}_{t})=-\bm{\epsilon}_{\theta}^{*}({\bm{z}}
_{t},t)/\sqrt{1-\bar{\alpha}_{t}}$ [].
After training, we can conduct the reverse sampling from the posterior distribution $p_{\theta}({\bm{z}}_{t-1}|{\bm{z}}_{t},{\bm{z}}_{0})$ as

$$ $\displaystyle\hat{{\bm{z}}}_{0|t}=$ $\displaystyle\mathbb{E}\left[{\bm{z}}_{0}|{\bm{z}}_{t}\right]=({\bm{z}}_{t}- \sqrt{\bar{\alpha}_{t}}\bm{\epsilon}_{\theta}({\bm{z}}_{t},t))/\sqrt{\bar{ \alpha}_{t}}$ (5) $\displaystyle{\bm{z}}_{t-1}=$ $\displaystyle\sqrt{\bar{\alpha}_{t-1}}\hat{\bm{z}}_{0|t}+\sqrt{1-\bar{\alpha}_ {t-1}}\bm{\epsilon}_{\theta}({\bm{z}}_{t},t),$ (6) $$

under the assumption of deterministic DDIM sampling [].

### 2.2 Classifier Free Guidance

By inheriting the classifier guidance of generative models, [] proposed the classifier guidance for deterministic DDIM sampling by defining epsilon prediction that corresponds to $\nabla_{{\bm{z}}_{t}}\log p({\bm{z}}_{t}|{\bm{c}})$,

$$ $\hat{\bm{\epsilon}}_{\theta}({\bm{z}}_{t},{\bm{c}},t)=\bm{\epsilon}_{\theta}({ \bm{z}}_{t},t)-\omega\sqrt{1-\bar{\alpha}_{t}}\nabla_{{\bm{z}}_{t}}\log p({\bm {c}}|{\bm{z}}_{t})$ (7) $$

where $\omega$ is a scale (^1^11If $\omega=1.0$, $\hat{\bm{\epsilon}}_{\theta}({\bm{z}}_{t},{\bm{c}},t)=\sqrt{1-\bar{\alpha}_{t} }\nabla_{{\bm{z}}_{t}}\log p({\bm{z}}_{t}|{\bm{c}})$. Otherwise, we are approximating $p({\bm{z}}_{t}|{\bm{c}})\propto p({\bm{z}}_{t}|{\bm{c}})p({\bm{c}}|{\bm{z}}_{t })^{\omega}$) for the guidance and $\nabla_{{\bm{z}}_{t}}\log p({\bm{c}}|{\bm{z}}_{t})$ is computed by a trained classifier.
To reduce additional computation costs from classifier training, [] proposed the classifier-free-guidance by substituting $\nabla_{{\bm{z}}_{t}}\log p({\bm{c}}|{\bm{z}}_{t})=\nabla_{{\bm{z}}_{t}}\log p
({\bm{z}}_{t}|{\bm{c}})-\nabla_{{\bm{z}}_{t}}\log p({\bm{z}}_{t})$ so that the epsilon prediction is defined as

$$ $\hat{\bm{\epsilon}}_{\theta}^{\omega}({\bm{z}}_{t},{\bm{c}},t)=\bm{\epsilon}_{ \theta}({\bm{z}},\emptyset,t)+\omega(\bm{\epsilon}_{\theta}({\bm{z}},{\bm{c}}, t)-\bm{\epsilon}_{\theta}({\bm{z}},\emptyset,t)),$ (8) $$

where $\emptyset$ denotes null-text that indicates the class identifier for unconditional score prediction (i.e. $\bm{\epsilon}_{\theta}({\bm{z}}_{t},t)=\bm{\epsilon}_{\theta}({\bm{z}}_{t},
\emptyset,t)$) to achieve both unconditional and conditional score prediction with a single neural network. It is noteworthy that the null-text token is also a learnable parameter of LDM.

### 2.3 Diffusion Inverse Solvers

Recently diffusion models have been extensively used as powerful generative priors for inverse
problems [, , , , , ].
Earlier techniques in inverse imaging relied on an alternating projection method [, , ], enforcing hard measurement constraints between denoising steps in either pixel or measurement spaces. More advanced strategies have been proposed to approximate the gradient of the log posterior within diffusion models, broadening the scope to tackle nonlinear problems []. The field has seen further expansion with methods addressing blind inverse problem [], 3D [, ], and problems of unlimited resolution [].

Traditionally, these methods have utilized image-domain diffusion models, but a shift has been observed towards latent diffusion models such as latent DPS (LDPS) with a fixed point of autoencoder process [], LDPS with history update [] and Resample [].
Despite these innovations, the use of text embedding for regularization is often overlooked, which we opt to address.

## 3 Main Contribution: Regularization by Text

In this section,
we explain the details of TReg. Specifically, the reverse diffusion sampling by TReg consists of three distinct steps:
1) adaptive negation for reinforcing the textual constraint,
2) latent optimization to impose measurement and latent space consistency,
and 3) forward process to return the intermediate reconstruction
to correct noisy manifold.
It is noteworthy that TReg is a zero-shot algorithm for general inverse problems without additional training or fine-tuning.
The details are as follows.

### 3.1 Adaptive negation for textual constraint

While the text prompt for LDM can effectively narrow down the solution space, directly employing text prompts via CFG may not adequately capture the intended guidance due to the inherent error-prone nature of hand-crafted prompts.
In other words, the provided text prompts may not represent optimal conditioning.
To resolve this, one can leverage the prompt tuning [], but it can disrupt the user intention included in the text prompt during the tuning process.
Instead of finding an optimal text prompt that matches the user’s intention,
we found that the concept negation [] that comprises complement concepts of the given text prompt
is effective in reinforcing the original user’s concept.

In particular, during the reverse sampling,
we minimize the following cost function on the CLIP [] embedding space:

$$ ${\mathcal{L}}_{\emptyset}=\textrm{sim}\left({\mathcal{T}}_{img}(\hat{{\bm{x}}} _{0}({\bm{y}})),{\bm{c}}_{\emptyset}\right),$ (9) $$

where $c_{\emptyset}$ is the embedded null-text token via CLIP text encoder, ${\mathcal{T}}_{img}$ denotes CLIP image encoder, $\hat{{\bm{x}}}_{0}({\bm{y}})$ denotes denoised estimates on pixel space which is updated by the data consistency, and $\textrm{sim}(\cdot,\cdot)$ denotes inner product.
By minimizing the similarity between intermediate images $\hat{{\bm{x}}}_{0}({\bm{y}})$ and the null-text in the CLIP space, we can make the null-text have complementary concepts for given contexts which are accumulated to the intermediate images $\hat{{\bm{x}}}_{0}$.
Specifically, leveraging the null-text token facilitates the concept negation and the suppression of the complement concept is achievable by optimizing the null-text token.
This approach considers only the null-text token as a variable, and since back-propagation to other neural networks does not occur, the computational cost is negligible. This makes the proposed method extremely cost-effective for enhancing text guidance.
Figure 2 (b) demonstrates that the adaptive negation significantly improves the reconstruction quality.

### 3.2 Latent Optimization

For the latent optimization, consider the following loss, which is the maximum a posteriori (MAP) objective under the VAE prior in (2) by assuming that ${\mathcal{E}}^{\sigma}_{\bm{\phi}}({\bm{x}})$ is isotropic, i.e. ${\mathcal{E}}^{\sigma}_{\bm{\phi}}({\bm{x}})=\sigma_{\mathcal{E}}\bm{1}$ :

$$ $\displaystyle{\mathcal{L}}({\bm{x}},{\bm{z}})=\frac{\|{\bm{y}}-{\mathcal{A}}({ \mathcal{D}}_{\bm{\varphi}}({\bm{z}}))\|_{2}^{2}}{2\sigma^{2}}+\frac{\|{\bm{z} }-{\mathcal{E}}^{\bm{\mu}}_{\bm{\phi}}({\bm{x}})\|_{2}^{2}}{2\sigma_{\mathcal{ E}}^{2}}.$ (10) $$

We can now minimize (10) using the proximal optimization.
Specifically, for an initial latent ${\bm{z}}_{0}:=\hat{\bm{z}}_{0|t}$ from the Tweedie’s formula, the proximal approach
compute the solution that minimizes the following:

$$ $\displaystyle{\mathcal{L}}({\bm{x}},{\bm{z}})+{\gamma}\|{\bm{z}}-\hat{\bm{z}}_ {0|t}\|^{2}$ (11) $$

which we are interested in solving by variable splitting similar in spirit to the alternating direction method of multipliers []. Namely, using the decoder approximation and setting ${\bm{x}}={\mathcal{D}}_{\varphi}^{\bm{\mu}}({\bm{z}})$, the optimization problem with respect to ${\bm{x}}$ becomes

$$ $\displaystyle\min_{{\bm{x}}}\frac{\|{\bm{y}}-{\mathcal{A}}({\bm{x}})\|_{2}^{2} }{2\sigma^{2}}+\frac{\|{\bm{z}}-{\mathcal{E}}^{\bm{\mu}}_{\bm{\phi}}({\bm{x}}) \|_{2}^{2}}{2\sigma_{\mathcal{E}}^{2}}+\lambda\|{\bm{x}}-{\mathcal{D}}_{\bm{ \varphi}}({\bm{z}})+{\bm{\eta}}\|_{2}^{2}.$ (12) $$

Here, for simplicity, we set dual variable ${\bm{\eta}}$ as a zero vector and do not consider its update.
Then, from the initialization with ${\bm{z}}=\hat{\bm{z}}_{0|t}$, we have

$$ $\displaystyle\hat{{\bm{x}}}_{0}({\bm{y}})=\operatorname*{arg\,min}_{{\bm{x}}} \frac{\|{\bm{y}}-{\mathcal{A}}({\bm{x}})\|_{2}^{2}}{2\sigma^{2}}+\lambda\|{\bm {x}}-{\mathcal{D}}_{\bm{\varphi}}(\hat{\bm{z}}_{0|t})\|_{2}^{2},$ (13) $$

which can be solved with negligible computation cost such as conjugate gradient (CG).
Subsequently, using the encoder approximation and setting ${\bm{z}}={\mathcal{E}}^{\bm{\mu}}_{\bm{\phi}}({\bm{x}})$ with ${\bm{\eta}}=\bm{0}$, the optimization problem with respect to ${\bm{z}}$ reads

$$ $\displaystyle\hat{{\bm{z}}}_{0}^{ema}=\operatorname*{arg\,min}_{{\bm{z}}}\zeta \|{\bm{z}}-\hat{{\bm{z}}}_{0}({\bm{y}})\|_{2}^{2}+{\gamma}\|{\bm{z}}-\hat{{\bm {z}}}_{0|t}\|^{2}$ (14) $$

with the initialization of $\hat{{\bm{z}}}_{0}({\bm{y}}):={\mathcal{E}}_{\bm{\phi}}(\hat{{\bm{x}}}_{0}({
\bm{y}}))$,
leading to a closed-form solution

$$ $\displaystyle\hat{{\bm{z}}}_{0}^{ema}=\bar{\alpha}_{t-1}\hat{{\bm{z}}}_{0}({ \bm{y}})+(1-\bar{\alpha}_{t-1})\hat{{\bm{z}}}_{0|t}$ (15) $$

where $\zeta,\gamma$ are chosen to satisfy $\bar{\alpha}_{t-1}={\zeta}/{(\zeta+\gamma)}$.
Finally, for the optimized latent $\hat{{\bm{z}}}_{0}^{ema}$,
the update rule of the DDIM [] at timestep $t$ is given as follows

$$ $\displaystyle{\bm{z}}_{t-1}^{\prime}$ $\displaystyle=\sqrt{\bar{\alpha}_{t-1}}\hat{{\bm{z}}}_{0}^{ema}+\sqrt{1-\bar{ \alpha}_{t-1}}\tilde{\bm{\epsilon}}_{t}$ (16) $$

where $\tilde{\bm{\epsilon}}_{t}$ denotes the total noise given by

$$ $\displaystyle\tilde{\bm{\epsilon}}_{t}:=\frac{\sqrt{1-\bar{\alpha}_{t-1}-\eta^ {2}\tilde{\beta}_{t}^{2}}\hat{\bm{\epsilon}}_{\theta}+\eta\tilde{\beta}_{t}\bm {\epsilon}}{\sqrt{1-\bar{\alpha}_{t-1}}}$ (17) $$

In (16), $\eta\in[0,1]$ is a parameter controlling the stochasticity of the update rule: $\eta=0.0$ leads to fully deterministic sampling, whereas $\eta=1.0$ with $\tilde{\beta}_{t}=\sqrt{(1-\bar{\alpha}_{t-1})/(1-\bar{\alpha}_{t})}\sqrt{1-
\bar{\alpha}_{t}/\bar{\alpha}_{t-1}}$ recovers the ancestral sampling of DDPMs.
Empirically, we find that $\eta\tilde{\beta}_{t}=\sqrt{\bar{\alpha}_{t-1}}\sqrt{1-\bar{\alpha}_{t-1}}$ results in robust performance. In other words, the total noise is computed by $\tilde{\bm{\epsilon}}_{t}:=\sqrt{1-\bar{\alpha}_{t-1}}\hat{\bm{\epsilon}}_{
\theta}+\sqrt{\bar{\alpha}_{t-1}}\bm{\epsilon}$.
To sum up, the proposed algorithm is described as in Algorithm 1.

### 3.3 Latent DPS with updated null-text

In image inpainting and phase retrieval,
we found that alternating latent DPS steps with the aforementioned latent optimization steps improves the data consistency.
In this case, we set $\hat{\bm{z}}_{0|t}({\bm{c}}_{\emptyset})$ denotes the Tweedie’s formula computed with the null text embedding. Then, we can obtain the standard LDPS gradient equipped with $\hat{\bm{c}}_{\emptyset}$, the optimized null text embedding at previous timestep:

$$ $\displaystyle{\bm{z}}_{t-1}={\bm{z}}^{\prime}_{t-1}-\rho_{t}\nabla_{{\bm{z}}_{ t}}\|{\mathcal{A}}({\mathcal{D}}_{{\bm{\varphi}}}(\hat{\bm{z}}_{0|t}(\hat{\bm{ c}}_{\emptyset})))-{\bm{y}}\|$ (18) $$

where we set $\rho_{t}$ to be the step size that weights the likelihood, similar to [, ].
On the other hand, in image super-resolution and deburring, we neglect the DPS update term for computational
saving and just use Algorithm 1 as it is.
Nevertheless, our algorithm can effectively solve the problem and shows comparable performance with DPS update as shown in Figure 2.

Figure: Algorithm 1 Inverse problem solving with TReg

Figure: Figure 2: Ablation study on DPS update and adaptive negation. (a) Measurements are generated by SR (x16). Down-sampled images are displayed as insets. Class:"ice cream". Prompt "macaron". (b) Adaptive negation improves reconstruction.
Refer to caption: extracted/2311.15658v2/figure/ablation.png

## 4 Experiments

The goal of text regularization is to refine the solution space by resolving ambiguity. Therefore, we evaluate two key aspects of the proposed method: 1) the effectiveness of TReg in resolving ambiguity, and 2) the accuracy of the obtained solution, which encompasses alignment with the text and measurements. For more details on experimental settings, refer to the Appendix.

### 4.1 Experimental settings

Forward models.
We utilize extreme measurement conditions to demonstrate efficacy of our method. Specifically, we use the bicubic down-sampling with scale factor 16, Gaussian blurring with kernel size 61 and sigma 5.0. We define the linear forward operations and its transpose by following the previous studies [, , , ].
In the case of the box inpainting task, our intention is to employ a universal mask that encompasses the eyes and mouth of either animal or human. For this, we generate a box mask based on averaged images across all data points within each dataset.
For Fourier phase retrieval, we obtain the Fourier magnitude by applying the Fourier transform to the image. In the formula, the forward model is expressed as ${\bm{y}}\sim{\mathcal{N}}({\bm{y}}||FP{\bm{x}}_{0}|,\sigma_{0}^{2}\bm{I})$, where $F$ denotes the 2D Discrete Fourier Transform (DFT), $P$ represents oversampling, that is implemented by adding 256 zero-padding to each side, and $\mathbf{x}_{0}$ denotes the clean image.
For the measurement noise, the noise scale $\sigma_{0}^{2}$ is set to 0.01 for all tasks.

Baselines.
The accuracy of the solution requires careful experimental design.
As the first attempt to solve inverse problems using text-driven regularization, we establish baselines by combining 1) data consistency-preserving methods and 2) text-guided editing methods. Notably, these baselines involve sequential processing, while the proposed method stands out for its efficiency, eliminating the need for pre/post-hoc techniques.
For data consistency methods, we utilize the measurement itself and PSLD [] which is an LDM-based inverse problem solver. Regarding text-guided editing methods, we opt for state-of-the-art techniques, namely Delta Denoising Score (DDS) [] and Plug-and-Play diffusion (PnP) []. Since PnP involves an inversion process, we handle it as a two-stage approach.
Also, we use PSLD with text guidance as our baseline by applying CFG for computing the posterior mean.
For the inpainting task, we utilize Stable-Inpaint, a fine-tuned stable diffusion model for the inpainting task, and Repaint [] as our baseline.
Finally, in the Fourier phase retrieval task, we compared the results with and without the text prompt using our framework to emphasize the effect of text guidance in breaking symmetry. In other words, for the baseline, we provided null text for the text condition.

Figure: Figure 3: TReg reduces the ambiguity. Given text prompt for TReg is "bear". (a) Measurement is generated by Gaussian blur. (b) Pixel-wise standard deviation is displayed. Uncertainty value across white dotted lines on measurement is plotted. Gaussian smoothing with a kernel width of 10 pixels is applied for visualization. (c) Reconstructed solutions from multiple trials.
Refer to caption: extracted/2311.15658v2/figure/uncertainty.png

Dataset and text prompt.
For the effective quantitative evaluation of an obtained solution, it is crucial to select the measurement and text prompt that do not conflict with each other.
This ensures that the generated features through textual guidance coherently maintain the data consistency across the entire dataset. Furthermore, selecting an improper text prompt ${\bm{c}}$ (for instance, “dog" for the measurement generated by the baby image) can lead to undesired outcomes such as ignoring the provided guidance.
After exploring various datasets, we have determined that the Food-101 dataset [] is well-suited for our quantitative comparison due to its non-sparse and complex patterns. Also, given that all images in this dataset fall under the ’food’ category, it becomes more straightforward to select an appropriate ${\bm{c}}$ aligned with the original caption.
Specifically, we leverage 250 images from each of the “fried rice" and “ice cream" classes, accompanied by the corresponding text prompts “spaghetti" and “macarons".
Also, we design the null-text update schedule since the reconstructed images become saturated and fidelity is decreased if the null-text is optimized too frequently. Therefore, we apply updates to the null text only when data consistency updates are performed. For the optimization, we use AdamW [] optimizer with learning rate 1e-3 and update null-text for 10 iterations per time step.
For the qualitative comparison, we additionally use a validation set of ImageNet [].

Figure: Figure 4: Representative reconstructed images for super-resolution (x16) and Gaussian deblurring. SR: Original class is "ice cream" and given prompt is "macaron". Deblur: Original class is "fried rice" and given prompt is "spaghetti".
Refer to caption: extracted/2311.15658v2/figure/result_entire.png

For the inpainting, we select the AFHQ and FFHQ datasets for analysis to assess the efficiency of the proposed method across diverse domains.
For the Fourier phase retrieval, we utilize the FFHQ dataset, since a general text prompt such as "photography of face" effectively describe the entire dataset and is adequate for resolving ambiguity arising from rotation or flip.

### 4.2 Experimental Results

Ambiguity reduction.
To assess the capacity of TReg in reducing ill-posedness, we perform a comparative evaluation between the diffusion-based inverse solver with and without the TReg component. The elimination of TReg involves employing a null-text ($\varnothing$) as the text prompt while retaining adaptive negation, serving as an additional tool to suppress complementary concepts and enhance the generated image quality. We solve the inverse problem 10 times with different random seeds for a fixed super-resolution measurement.

The result, depicted in Figure 3, demonstrates that the TReg leads to a unique solution corresponding to the given prompt. Conversely, in the absence of TReg, reconstructed images exhibit multiple solutions, such as a car in the background or a bedroom. This discrepancy is clearly observed in its uncertainty map in Figure 3 which is defined as pixel-wise standard deviation values, revealing heightened uncertainty levels without TReg, particularly in regions corresponding to distinct objects or characters.
Although TReg implementation reduces uncertainty, some residual uncertainty persists, primarily attributed to in-class variations, notably in aspects like head positioning or mouth shape. However, it is noteworthy that these residual uncertainties do not violate the data consistency on the measurement space.
Additionally, in Fourier phase retrieval, text guidance helps to effectively reconstruct consistently clean images by breaking inherent system symmetries, whereas the same algorithm without text guidance fails to consistently produce correct solutions (Figure 1). For more results, please refer to the Appendix.

Accuracy of obtained solution.
To evaluate the quality of the obtained solution by TReg, we conduct a quantitative comparison with baseline methods for super-resolution and deblur tasks.
The evaluation is based on three metrics: LPIPS, the mean squared error between the provided measurement and the predicted measurement, namely y-MSE(^2^22In formula, we compute $\|{\bm{y}}-\mathcal{A}({\bm{x}})\|^{2}_{2}$.), and the CLIP similarity between the reconstructed image and the text prompt following the prior works in image editing [, ].
Each metric provides 1) quality of the reconstructed images, 2) data consistency, and 3) quality of text guidance, which allows us to investigate the advantage of TReg on inverse problems.
Note that LPIPS is computed with respect to real images with the class of target prompt.

**Table 1: Quantitative evaluation of SRx16 and Gaussian Deblurring task. Mean values are reported. Bold: the best score, underline: the second best.**
| SRx16 |  | Fired Rice $\rightarrow$ Spaghetti |  | Ice Cream $\rightarrow$ Macaron |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| # stages | method | LPIPS $\downarrow$ | CLIP-sim $\uparrow$ | y-MSE $\downarrow$ |  | LPIPS $\downarrow$ | CLIP-sim $\uparrow$ | y-MSE $\downarrow$ |
| 1 | Ours | 0.769 | 0.303 | 0.005 |  | 0.771 | 0.314 | 0.004 |
|  | PSLD | 0.756 | 0.265 | 0.020 |  | 0.743 | 0.293 | 0.020 |
| 2 | PnP | 0.826 | 0.251 | 0.005 |  | 0.808 | 0.239 | 0.005 |
|  | PSLD+DDS | 0.788 | 0.247 | 0.010 |  | 0.772 | 0.328 | 0.013 |
| 3 | PSLD+PnP | 0.801 | 0.291 | 0.014 |  | 0.784 | 0.306 | 0.008 |

The results in Table 1 indicate that the solution obtained exhibits high quality while preserving data consistency, as evidenced by lower LPIPS and y-MSE values compared to the baselines.
For some cases, PSLD achieves notably lower LPIPS but solutions are lost data consistency and exhibit large y-MSE values with significantly different reconstructions as shown in the fourth column of Figure 4.
In the case of CLIP similarity, TReg is comparable or superior compared to state-of-the-art image editing algorithms, highlighting the efficacy of text-guidance facilitated by TReg.
Direct application of the editing algorithm to the measurement, referred to as PnP, yields sub-optimal performance across all metrics. In the case of y-MSE, PnP achieves a lower value due to minimal alterations made to the measurement, as depicted in the last column of Figure 4.
Also, the performance of sequential approaches shows that reconstruction error is accumulated at each stage, resulting in substantial errors in the final outcome. In contrast, TReg effectively integrates the given text prompt while preserving robust data consistency, thereby validating its efficacy.

Figure 4 presents a qualitative comparison between the proposed method and the baseline methods. The quality of the reconstructed image aligns with the quantitative results. Interestingly, PnP and DDS show a notable similarity in the direction of editing, with DDS tending to generate relatively sharper images. However, both methods compromise consistency with the given measurement. In contrast, our method effectively preserves data consistency.
Meanwhile, we also compare the results of our method without TReg and PSLD with CFG guidance.
When TReg is removed from our method (third column in Figure 4), it provides a solution that aligns well with the given measurement but generates arbitrary structures. This result highlights the impact of TReg that reduces the ambiguity of inverse problem solving. PSLD with CFG guidance (fourth column in Figure 4) also fails to appropriately reflect given text prompt, as there is no consideration for text-guidance in solving inverse problems.

Figure: Figure 5: Solving inverse problem with multiple prompts. Measurements are generated from ImageNet, Food101, and Landsacpe dataset. (a) With TReg, we can solve the inverse problem with multiple prompts for a fixed measurement. (b) Deblur (Gauss) (c) Super-resolution. Best views are displayed.
Refer to caption: extracted/2311.15658v2/figure/diverse.png

Furthermore, we conduct a qualitative comparison for box inpainting task. illustrated in Figure 6 (a), TReg reconstructs image properly according to the given text prompt with higher fidelity compared to baseline methods.
From Figure 6 (b), we repeatedly observe that text regularization enables the discovery of solutions as intended. It allows the discovery of even rare solutions, such as a cat with green eyes.
Additionally, in Figure 6 (c), TReg demonstrates its capability to solve the inpainting problem by composing concepts. In other words, TReg can narrow down the solution space by utilizing not only a singular concept but also multiple concepts simultaneously.

Figure: Figure 6: Representative results for box inpainting task. (a) TReg is better at finding solution according to text with high-fidelity. (b) Text regularization helps to reconstruct as intended. (c) TReg is possible to reconstruct based on multiple concepts.
Refer to caption: extracted/2311.15658v2/figure/inpaint.png

### 4.3 Robustness of TReg with diverse texts and image domains

In previous sections, we demonstrate that TReg successfully mitigates the uncertainty of inverse problems and produces accurate reconstruction consistently.
We further investigate the robustness of TReg with diverse set of texts and image domain such as landscape.
Given that the primary role of TReg is to reduce the solution space according to $p({\bm{z}}|{\bm{c}})$, the proposed method is expected to reconstruct diverse images in response to the provided text prompt.
Figure 5 illustrates the results of solving inverse problems for super-resolution and deblur tasks with various text prompts. Regardless of the forward operation or data domain, TReg effectively reconstructs the solution. Importantly, different solutions maintain consistency with the given measurement. This distinctive feature sets our proposed approach apart from conventional stylization or personalization methods.
Furthermore, the result directly shows the existence of multiple solutions that satisfied given forward model, which is an evidence of the ill-posedness nature of the inverse problem.

## 5 Related work

Leveraging text prompts for inverse problem solving has been studied in several recent works. TextIR [] incorporates text embedding from CLIP during the training of an image restoration network to constrain the solution space in the case of extreme inverse problems.
DA-CLIP [] takes a similar approach of leveraging the embeddings and applies it to diffusion bridges.
P2L [] proposes to automatically tune text embedding on-the-fly while running a generic diffusion inverse problem solver. Notably, the latter two methods [, ] are focused on improving the overall performance rather than constraining the solution space. TextIR has similar objective to TReg, but requires task-specific training. In this regard, to the best of our knowledge, TReg is the first general diffusion-based inverse problem solver that does not require task-specific training, while being able to incorporate text conditions to effectively control the solution space in general inverse problems.

## 6 Conclusion

This study introduced a novel concept of latent diffusion inverse solver with regularization by texts, namely TReg. The proposed solver minimizes ambiguity in solving inverse problems, effectively reducing uncertainty and improving accuracy in visual reconstructions, bridging the gap between human perception and machine-based interpretation.
To achieve this, we derived LDM-based reverse sampling steps to minimize data consistency with text-driven regularization, consisting of adaptive negation. Specifically, to effectively integrate textual cues and guide reverse sampling, the paper introduced a null text optimization approach.
Experimental results and visualizations demonstrated that text-driven regularization effectively reduces the uncertainty of solving inverse problems, and further enables text-guided control of signal reconstruction.

## 7 Potential negative impacts

In this study, the proposed text regularization is heavily influenced by prior. Therefore, if there is bias in a model utilized as a prior, such as in the case of stable diffusion, the solution to the inverse problem may also contain bias. As a restoration model controllable through text, there is a potential for misuse, such as the generation of fake news. Countermeasures should be established to prevent incorrect results and ensure the integrity of the outcomes.

Supplementary material of "Regularization by Texts for Latent Diffusion Inverse Solvers"

Jeongsol Kim Geon Yeong Park Hyungjin Chung Jong Chul Ye

## 8 Implementation Details

In this section, we provide further details on implementation of TReg.
The code will be available to public on [https://github.com/TReg-inverse/Treg](https://github.com/TReg-inverse/Treg).

### 8.1 Stable-diffusion and CLIP

We leverage the pre-trained Latent Diffusion Model (LDM), including an auto-encoder and the U-net model, provided by diffusers. Due to the limitation of its modularity, we implement the sampling process for inverse problems by ourselves, rather than changing given pipelines. The Stable-diffusion v1.5 is utilized for every experiment in this work, and the ViT-L/14 backbone and its checkpoint is used for CLIP image encoder for adaptive negation.

### 8.2 Conjugate gradient method

In this work, we apply the CG method to find a solution to the following problem:

$$ $\min_{x}\frac{\|{\bm{y}}-{\mathcal{A}}{\bm{x}}\|^{2}_{2}}{2\sigma^{2}}+\lambda \|{\bm{x}}-{\mathcal{D}}_{\varphi}(\hat{\bm{z}}_{0|t})\|^{2}_{2}.$ (19) $$

As the objective function is a convex function, the solution ${\bm{x}}^{*}$ should satisfy

$$ $-{\mathcal{A}}^{\top}({\bm{y}}-{\mathcal{A}}{\bm{x}}^{*})+\lambda({\bm{x}}^{*} -{\mathcal{D}}_{\varphi}(\hat{\bm{z}}_{0|t}))=0,$ (20) $$

where the coefficients are absorbed to $\lambda$. Then, we can formulate it as a linear system as

$$ $(\lambda\bm{I}+{\mathcal{A}}^{\top}{\mathcal{A}}){\bm{x}}^{*}=\lambda{\mathcal {D}}_{\varphi}(\hat{\bm{z}}_{0|t})+{\mathcal{A}}^{\top}{\bm{y}}$ (21) $$

where $\mathbf{A}=\lambda\bm{I}+{\mathcal{A}}^{\top}{\mathcal{A}}$ and $\mathbf{b}=\lambda{\mathcal{D}}_{\varphi}(\hat{\bm{z}}_{0|t})+{\mathcal{A}}^{
\top}{\bm{y}}$.
Thus, we can solve (21) by CG method.
In this work, we use 5 iterations of CG update with $\lambda=1e-4$ for each time step if not explicitly stated otherwise.

### 8.3 CG update range

The data consistency update with CG algorithms is applied for a subset of sampling steps as described by $\Gamma$ in Algorithm 1 of the main paper.
Our empirical observation show that this partial data consistency update achieves better
trade-off between the image reconstruction consistency and the latent stability.
In the absence of the data consistency update, we employ the deterministic DDIM sampling step by setting $\eta\tilde{\beta}_{t}=0$ or apply DPS gradient to improve the reconstruction quality.
Overall, for inverse problems except the Fourier phase retrieval, we set the Network Function Evaluation (NFE) to 200 and design $\Gamma=\{t|t\text{ mod }3=0,t\leq 850\}$(^3^33The pre-trained Stable-diffusion uses $T=1000$. Thus, $t=850$ is equivalent to the sampling step with NFE = 170.), where mod denotes the modulo operation. For the Fourier phase retrieval, we set $\Gamma=\{t|t\text{ mod }10=0\}$.

### 8.4 TReg with DPS update

In the main manuscript, we only describe the pseudocode of TReg without DPS update for the readability.
However, the algorithm could readily conduct an additional DPS update as follow.
Under the definition of stochastic noise to be added at each timestep,

$$ $\displaystyle\tilde{\bm{\epsilon}}_{t}:=\frac{\sqrt{1-\bar{\alpha}_{t-1}-\eta^ {2}\tilde{\beta}_{t}^{2}}\hat{\bm{\epsilon}}_{\theta},+\eta\tilde{\beta}_{t} \bm{\epsilon}}{\sqrt{1-\bar{\alpha}_{t-1}}}$ (22) $$

the pseudocode could be described as in Algorithm 2.
We have used this additional DPS step for the box inpainting and Fourier phase retrieval task with step size $\rho_{t}=\sqrt{\bar{\alpha}_{t-1}}$. For box inpainting task, we set CFG scale $\omega_{2}$ for DPS gradient to 0, while we set CFG scale $\omega_{2}=\omega_{1}$ in Fourier phase retrieval task.

Figure: Algorithm 2 Inverse problem solving with TReg with DPS

### 8.5 CFG guidance scale

The classifier-free guidance (CFG) is defined as

$$ $\hat{\bm{\epsilon}}_{\theta}=\bm{\epsilon}_{\theta}({\bm{z}}_{t},\phi,t)+ \omega(\bm{\epsilon}_{\theta}({\bm{z}}_{t},\mathbf{c},t)-\bm{\epsilon}_{\theta }({\bm{z}}_{t},\phi,t))$ (23) $$

where $\omega$ is a scale for the guidance. In other words, $\omega$ could be interpreted as a magnitude for the negation. Thus, we leverage $\omega$ to control the extent of enhancement applied to a given text prompt. In fact, we can regulate the same feature by adjusting the null-text update strategy, including null-text update frequency and number of optimization iterations per time step. However, the CFG guidance scale is simpler and more intuitive than adjusting the null-text update strategy. Hence, we have used a proper CFG scale while fixing the null-text update schedule $\Gamma$.

For the main experiments with the Food101 dataset in the main paper, we use the default scale 7.5 for all results.
For the Fourier phase retrieval problem, we set the CFG scale $\omega_{1}=\omega_{2}=4.0$.
For other results, CFG scale among $\{3.0,4.0,5.0,7.5\}$ provide robust performance for various datasets including FFHQ, AFHQ, LHQ (landscape dataset), and ImageNet.

### 8.6 Optimization problem for Fourier phase retrieval

In contrast other inverse problems with linear operation, Fourier phase retrieval involves non-linear operation so we cannot leverage CG to solve the optimization problem in pixel space.
Thus, we use Adam optimizer with learning rate $1e-3$ and $\beta_{1}=0.9,\beta_{2}=0.999$ to obtain the solution of

$$ $\displaystyle\min_{\bm{x}}\frac{\|{\bm{y}}-{\mathcal{A}}({\bm{x}})\|^{2}_{2}}{ 2\sigma^{2}}+\lambda\|{\bm{x}}-\hat{\bm{x}}_{0}\|^{2}_{2}$ (24) $$

by setting $\lambda=0$.
This induces additional computational costs to solve the optimization problem. However, as analyzed in Section 9, these costs are comparable to those of other baseline algorithms, while consistently achieving the reconstruction of a unique solution.

## 9 Runtime Analysis

**Table S2: Runtime (min) of each algorithm for NFE=200.**
|  | PSLD | PSLD w/ AN | TReg |
| --- | --- | --- | --- |
| Linear | 1.47 | 1.47 | 0.26 |
| Non-linear | 1.12 | 1.12 | 1.09 |

We propose text regularization by defining a proximal optimization problem in the latent space. While we introduce a two-step optimization problem-solving approach, TReg does not require a large amount of computation time, as we leverage an accelerated optimization method, namely CG, and compute a closed-form solution for the subsequent problem. Consequently, TReg is effective compared to baseline algorithm, as shown in Table S2. We measure the runtime of linear inverse problem with delburring task, and measure the runtime of non-linear inverse problem with Fourier phase retrieval task.
Specifically, we did not use fixed point constraint of PSLD for the non-linear inverse problem as ${\mathcal{A}}^{\top}$ is not defined properly.
Furthermore, the optimization variable for adaptive negation is only the text embedding, which does not incur a significant additional cost. Therefore, the runtime for adaptive negation is negligible (the second column of Table S2).

Figure: Figure S7: Additional results on Fourier Phase Retrieval. Best views are displayed.
Refer to caption: extracted/2311.15658v2/figure/pr_additional.png

## 10 Additional Results

### 10.1 Fourier Phase Retrieval

In this section, we present additional examples for the Fourier phase retrieval problem. Specifically, we illustrate the variance of outcomes from the inverse problem solver without text regularization due to remaining symmetry, and compare them with the consistent outcomes from the TReg, which breaks the symmetry.
Figure S7 provides evidence for the capability of TReg to break the symmetry, contrasting with the LDPS algorithm without text regularization.

### 10.2 Super-resolution and Deblurring

In this section, we provide extended results for measurements to support the efficiency and compatibility of the proposed TReg with various domains. Specifically, we leverage the proposed method for super-resolution and delurring tasks on FFHQ. We examine the reconstruction with text prompts “baby face” and “adult face”. As shown in Figure S8, the proposed method successfully reconstructs images according to the given prompt, while the baseline method (sequential approach of PSLD and DDS) shows inferior reconstruction quality.

Figure: Figure S8: Reconstruction result for SR (x16) and Deblur (Gaussian) task on FFHQ dataset. Our method outperforms the existing methods (PSLD+DDS) in terms of the reconstruction quality.
Refer to caption: extracted/2311.15658v2/figure/append_age.png

## 11 Limitation

While TReg demonstrates its performance in breaking the symmetry in the Fourier Phase Retrieval problem, its effectiveness is contingent upon the text prompt. For instance, when the true image features a complex or colorful background, a simple text prompt such as "a photography of a face" may not suffice to reconstruct a unique solution without residual symmetry.

In our scenario, we assume that the user can access the category of the dataset (e.g., FFHQ for faces). However, in real-world scenarios, it may be challenging to derive more informative text prompts solely from measurements with severe degradation. In general, finding suitable text prompts remains an open problem in text regularization for solving inverse problems.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 8
presentation: 3
questions: please refer to weakness part.
soundness: 3
strengths: * this paper introduces an explicit regularization term during training via text-driven regularization (TReg)
* the proposed TReg effectively addresses the challenge of ambiguity in inverse problems
* this paper further introduces an adaptive negation to dynamically adjust the influence of textual guidance
* the paper is well written and easy to follow, extensive in main text and supplementary demonstrate the effectiveness of the proposed method for diffusion inverse solvers
summary: This paper introduces a novel method for solving inverse problems using diffusion models, with a focus on reducing ambiguity in the solutions.
The authors propose a latent diffusion inverse solver that incorporates regularization by texts (TReg), which applies textual descriptions of the expected solution during the reverse sampling phase. 
During training, this text guidance is dynamically adjusted through null-text optimization for adaptive negation.

Overall, I think this is a good paper.
weaknesses: 1. lack of overall results over whole dataset. all the experiments results are shown in visualizations or subset results of specific classes.
2. i notice in both table 1 and table 2, PSNR scores are higher without adaptive negation, do authors have any analysis or intuition about this results?
3. (this might not be a weakness, just naturally curious) in the experiments, the text regularization is only tested with class name, how about the results using some natural captions (such as generated with BLIP or GPT)?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: See above
soundness: 3
strengths: Overall the paper is well written, provides clear motivation, and the authors perform reasonably comprehensive experiments. The authors do a good job in their logical flow and provide pretty clear explanations of the proposed method and experimental setup. Figures look nice.

In terms of novelty, they authors introduce a reasonably novel approach to solving inverse problems by incorporating text-based regularization into latent diffusion models.  This addresses a significant limitation of existing solvers, mostly their struggle with ambiguity, and mitigates the need for task specific training. 

The proposed method is technically sound, with a clear mathematical formulation and well-defined optimization procedures.  The integration of LDPS further enhances the method's capabilities.
summary: This paper introduces TReg (Regularization by Text), a method for solving inverse problems in image processing using latent diffusion models by conditioning on text.  TReg leverages user provided textual descriptions to guide the image reconstruction process.

Prior diffusion-based inverse solvers often struggle with ambiguity, as multiple different images can produce the same degraded output. TReg addresses this by incorporating textual prompts as a form of regularization. The authors frame the problem as a text-conditioned latent optimization, where the goal is to find a solution that is both consistent with the measurements and aligned with the provided text description.

The authors introduce the following:
* A text based regularization term that encourages the reconstructed image's latent representation to be close to a text-conditioned denoised estimate. This helps guide the sampling trajectory towards solutions that are semantically aligned with the text.
* To further refine the textual alignment, this method uses a novel "null-text optimization" technique. This dynamically adjusts the influence of the textual guidance during the reverse diffusion sampling process, suppressing unintended signal components.


The authors also discuss limitations, particularly the challenge of finding suitable text prompts for some challenging inverse problems where the problem is extremely ill posed.
weaknesses: * `Line 071` In my view the connection to the human brain is quite weak, not sure why the authors put this in the paper...
* I would like to see additional, none cherry picked outputs. In my experience using diffusion models to recover images can often fail in very strange ways. I am specifically interested in image inpainting results, which nearly always result in boundary artifacts.
* I would like to see experiments in the presence of measurement noise, for example with JPEG compression artifacts after gaussian blurring.

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 2
questions: Main questions (answering these questions could affect the recommendation)

- Could the authors provide information about convergence of the alternate direction method? 
- Could the authors provide quantitive evaluation averaged across full datasets (as done in state of art works, e.g., P2L) and include relevant baselines, as mentioned above P2L should be included. Mokady would also be a good comparison for inpainting results. 

Other questions (this are more curiosity driven questions, it is unlucky the answers here will change the recommendation)
- In figure 2 authors show that thanks to the adaptive negation strategy the generated image is less noisy. It is unclear however why this would be the case. Could it be that without adaptive negation one could reach a similar results with more iterations?
- Due to (18) does it mean that the VAE needs to have the same dimensionality of CLIP embeddings?
- “The proposed solver without adaptive negation tends to achieve higher PSNR than with adaptive negation, since it obtains blurry images”. Could the Structural similarity index measure (SSIM) be a better metric in this case?
- For the phase retrieval experiments: the error in the LDPS reconstruction seem to go beyond the ambiguity of symmetries… Do the authors have any intuition about why the quality of LDPS reconstruction is so low compared to TReg?
- How sensitive is this frameworks to slight variation of the text? This would be an interesting discussion of the paper. The authors use “A photography of …” how much do results change if instead they used “A photo of xxx” or simply “xxx”?
soundness: 3
strengths: The paper tackles an interesting problems with many potential applications. 

While the individual ingredients are not completely new, even in the context of inverse problems (the use of prompts and the idea of solving the problem using alternate direction method have been explored by Chung et al in “PROMPT-TUNING LATENT DIFFUSION MODELS FOR INVERSE PROBLEMS”; the optimization of the null-text has been explored by Mokady et all in “Null-text Inversion for Editing Real Images using Guided Diffusion Models”) the overall framework and the way these ingredients are used is novel and seems to lead to compelling results.
summary: This work uses diffusion models with text guidance in order to solve inverse problems. Inverse problems (e.g., in-painting or deblurring) have been recently tackled using diffusion models, however, their ill posedness nature makes resolving some ambiguities still challenging. The authors propose to leverage a user bias via a text prompt in order to guide the diffusion process towards a solution that is aligned with the description provided. The proposed framework uses a latent diffusion model with classifier-free guidance as a core framework with two innovations compared to previous works: (1) a term that induces alignment between the latent being optimized and that conditioned through the text prompt, and (2) a dynamic null-text adaptation. (1) is the distance between the latent embeddings that are being optimized and those obtained with the text used as guidance (in a classifier free guidance algorithm). (2) The null-text (which usually is the representation provided without any guidance) is updated dynamically in order to be aligned with the CLIP embeddings of the text provided by the user, incentivizing in this way the embeddings to continue to learn new concepts by moving the null-text closer and closer to the text description. 

The authors show results on super resolution, deblurring and Fourier Phase Retrieval problems and compares with some baselines such as DDRM, PGDM, PSLD, and PnP.
weaknesses: The main weaknesses of this work are in the lack of quantitative evaluations on full datasets, lack of some important baselines and the clarity of the writing could be improved. See below for more detailed comments.

Evaluation: all tasks presented are obtained on specific classes rather than being averaged across full datasets as it is done in other the state of the art works (e.g., P2L). Specifically:
- Ambiguity reduction: it is unclear if the quantitative evaluation in Figure 3 is an average over a dataset or simply the results of the image shown (bear). It is also unclear from which source this image comes from. Maybe it is ok since this is an interesting experiment but not the core? In any case more clarity is needed.
- Accuracy of obtained solution: use true class as c. Results in  Table 1 and Table 2 show quantitative measurements (great) for two tasks (super resolution and de-blurring) but only on two classes “Ice-cream” and “fired rice” rather than averaging across full datasets.
- Accuracy of obtained solution: use different class as c. Results shown in Table 3 are for only two classes using two different C labels (fried rice to spaghetti, and Ice Cream to Macaroons). It is understandable to show selected classes in Figures but the quantitative evaluation should be reported on a larger datasets (with mean and standard deviations).
- Results on non-linear inverse problems. These results are only qualitative shown on 4 different images (Figure 6).

This work shares a lot of similarities with P2L (Chung et al.) so I was expecting P2L to be among the baselines. In the intro when talking about P2L the authors say “However, this method primarily focuses on data consistency and lacks robust alignment with textual prompts.” It would be great to support this statement with quantitative results.

For the alternate directions method there should be some additional information about convergence. Being an iterative method one would expect to see some analysis about it but I did not find any information. 

The work of Mokady et al also shares the idea of adaptive negation (but with different application and implementation) so for Inpainint experiments (which are suitable for Mokady) this would also be a useful baseline.

Writing: The paper is not alway easy to follow. The readability of the manuscript would benefit from some re-writing. For example 
- Background session could be summarized and moved to the appendix since these are all known concepts.
- The additional space gained should be used to provide more details about the proposed method. Here are some examples of things that could be clarified but there are more throughout the text:
    - Probably the most important: authors should mention that the proposed update is scheduled to be performed every so many iterations  end not always. This is somewhat hidden in the “range” argument of algorithm 1 but it becomes clear only reading the appendix. I strongly encourage the authors to bring this discussion into the main paper.
    - Consider moving the inpaiting results (with comparison with Mokady) in the main paper.
    - In Eq (13) and (14) there is the term A(x). Given that A is an “imaging system” but it is not clear the role it plays in practice when solving a specific problem. I would recommend the authors to provide some information about the role of the forward operator when discussing section 3.1.
    - “Here, ζ,γ are empirically chosen to satisfy¯αt−1 = ζ/(ζ+ γ).” Please provide an explanation why satisfying this equation is important.
    - …
- Since Figure 1 is even before the abstract the reader should be able to understand it without reading anything but the caption and looking at the figure itself. Consider increasing the details in the caption or moving the figure in a place where the reader has sufficient information to understand all of the examples.
- I find the abstract lacking important details. It should encourage a reader to continue the reading but in order to do so the more details  need to be presented. Here some potential improvements:
    - ``ambiguities in measurements or intrinsic system symmetries.’’ consider adding a concrete example of these problems.
    - define what adaptive negation is  or avoid adding a non-defined term.
    - ``Our comprehensive experimental results’’ consider telling the reader which are the most important quantitive results so that it encourages the reader to continue.
- In the experimental section consider breaking down the baselines for each experimental task. Currently they are presented all at once at the beginning irrespective of the tasks and I find this to be too much information to process and too far from where it is useful.
- The authors used the prompt “A photography of a ….”. This might not be grammatically correct. Photography is the art of creating photos or the process of creating photos. A better prompt would  be “A photo of a…” or “A photograph of a …”. I would be curious to know if this changes the results.
- Tweedie’s formula should be cited.
- From (10) to (13) the terms are swapped. Please consider maintaining consistent order.
- While the fact that the work uses a pre-trained VAE is mentioned in the main paper, I think the fact that also the diffusion model is pre-trained is not explicitly mentioned before the appendix. Consider clarifying this aspect in the main paper.
- The following sentences are unclear, consider rephrasing them: 
    - “We also prepare a situation that different class label is given as a text description. Here, we should carefully set proper text prompt for measurement to avoid ignoring the provided guidance”. 
    - “For the case that given text description is differnet from the original classes, we use “spaghetti" and “macarons", respectively.”. There is a typo (differnet) but beyond that it is unclear what this sentence means.
    - “For data consistency methods, we utilize the measurement itself “. What is the measurement?
    - “TReg leads to a unique solution corresponding to the given text description”. Maybe “TReg leads to similar solutions.”? Possibly one could say “semantically unique and aligned with the text”?
    - “ This discrepancy is clearly observed in pixel-level uncertainty in Figure 3(c). “ I suppose this is the variance computed across the 10 reconstruction? Consider mentioning explicitly how uncertainty is computed.

Meta-review: After rebuttal and multiple rounds of discussion, all three reviewers unanimously agreed to accept this submission，especially extra experiments over the entire dataset have been added to the rebuttal. Specifically, each reviewer's comments after rebuttal are summarized as follows.

For Reviewer CtoC, through additional experiments and detailed explanations, the authors successfully addressed all concerns raised by the reviewers, further demonstrating the effectiveness and advantages of TReg in solving inverse problems. Moreover, the authors identified promising future research directions, including the extension to natural language descriptions and the integration of more advanced text encoders.

For Reviewer vYBr, the authors effectively addressed all of the reviewers' concerns through additional experimental results and detailed explanations, thereby enhancing the completeness and academic rigor of the paper. While certain potential improvements were identified as directions for future work, the reviewers acknowledged the innovation and effectiveness of the TReg method and maintained their recommendation for the paper's acceptance.

For Reviewer 2SWh, the authors have comprehensively addressed the reviewers' main concerns through additional experiments and improvements in the manuscript's clarity, further demonstrating the applicability and innovation of TReg in tackling highly ambiguous and complex inverse problems. They have also outlined promising directions for future research, including extending the approach to more complex nonlinear problems, refining text-generated prompts, and conducting theoretical analyses of algorithmic convergence.

So,  I recommend Accept (poster).

Decision: Accept (Spotlight)