PromptFix: You Prompt and We Fix the Photo

Authors: yongsheng  yu
ziyun  zeng
hang  hua
jianlong  fu
jiebo  luo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 04:36:18

Abstract: Diffusion models equipped with language models demonstrate excellent controllability in image generation tasks, allowing image processing to adhere to human instructions. However, the lack of diverse instruction-following data hampers the development of models that effectively recognize and execute user-customized instructions, particularly in low-level tasks. Moreover, the stochastic nature of the diffusion process leads to deficiencies in image generation or editing tasks that require the detailed preservation of the generated images. To address these limitations, we propose PromptFix, a comprehensive framework that enables diffusion models to follow human instructions to perform a wide variety of image-processing tasks. First, we construct a large-scale instruction-following dataset that covers comprehensive image-processing tasks, including low-level tasks, image editing, and object creation. Next, we propose a high-frequency guidance sampling method to explicitly control the denoising process and preserve high-frequency details in unprocessed areas. Finally, we design an auxiliary prompting adapter, utilizing Vision-Language Models (VLMs) to enhance text prompts and improve the model's task generalization. Experimental results show that PromptFix outperforms previous methods in various image-processing tasks. Our proposed model also achieves comparable inference efficiency with these baseline models and exhibits superior zero-shot capabilities in blind restoration and combination tasks.

Main paper: 
 introduction: 
 In recent years, diffusion models [19, 57, 66] have achieved remarkable advancements in textto-image generation. Benefiting from large-scale training on image-text pairs [59] , these models can generate highly realistic and diverse images that align with text prompts. They have been successfully applied to various real-world applications, including visual design, photography, digital art, and the film industry. In addition, models trained with instruction-following data [7] have shown promising results in understanding human instruction and performing the corresponding imageprocessing tasks. Previous studies [21-23, 74, 73] have illustrated that with instruction-following data, we can simply fine-tune a text-to-image generation model to perform various vision tasks such as image editing [7, 21, 74] , object detection [22] , segmentation [23] , inpainting [23, 73] , and depth estimation [22, 9] . To follow the success of these methods, we train our model utilizing input-goal-instruction triplet data for low-level image-processing tasks.We first overcome the challenge of lacking the instruction-following data for low-level tasks. Specifically, we collect image pairs by generating degraded images from the source images and adopting data from existing datasets. Then, we employ GPT4 [51] to generate the diverse text instructions for each task. We obtain ∼ 1.01 million input-goal-instruction triplets in the collected dataset. This dataset covers various low-level tasks including image inpainting, object creation, image dehazing, colorization, super-resolution, low light enhancement [8] , snow removal, and watermark removal. We enrich the dataset through back-translation augmentation by swapping the inpainted and origi-Figure 1 : We propose PromptFix, a unified diffusion model capable of performing multiple imageprocessing tasks. It can understand user-customized editing instructions and perform the corresponding tasks with high quality. One of the key advantages of PromptFix is high-frequency information preservation, ensuring that image details are maintained throughout VAE decoding. PromptFix can handle various images with different aspect ratios. nal images within the triplet and inverting the semantic orientation of the prompt. This technique effortlessly converts datasets from object removal to object creation. We also provide comprehensive details in Section 3.With the dataset, we design a new diffusion-based model named PromptFix that can understand user-customized instructions and perform the corresponding low-level image-processing tasks. In PromptFix, we address several challenges that compromise the performance of the model. First, the use of stable-diffusion architecture [57] as a generative prior often faces the issue of spatial information loss, which is caused by VAE compression [20] . Unlike unconditional or text-to-image generation, maintaining spatial detail consistency in image processing poses significant challenges, particularly with high-frequency components like text, as shown in Figure 5 . To tackle this problem, we introduce High-frequency Guidance Sampling, in which we use a low-pass filter operator [50, 53] to calculate the fidelity constraint and integrate VAE skip-connect features during inference with a lightweight LoRA [27] fusion. Second, since the generative prior is not trained on low-level images, so relying solely on instructions may not always yield the desired outcome, especially when the image degradation is severe. To tackle this degradation adaptation problem, we introduce an auxiliary prompt module to provide models with more descriptive text prompts to enhance controllability for image generation. The auxiliary text prompt can be obtained by VLMs [43] . This approach introduces a semantic caption for a degraded image and the description of its defects, such as blurriness or insufficient lighting. The auxiliary prompt module is implemented by an additional attention layer in diffusion U-Net that adapts both instruction and auxiliary prompts as conditions and intermittently omits instructional prompts during training. We identify three key advantages of this approach: 1) enabling the model to process images with severe degradation, such as extremely low-resolution images, 2) adapting the model for blind restoration for different types of image degradation, and 3) providing additional pathways for a more precise semantic representation of the target image.Experimental results demonstrate that our model achieves superior performance in the instructionbased paradigm across three image editing tasks (colorization, watermark removal, object removal) and four image restoration tasks (dehazing, desnowing, super-resolution, and low-light enhancement) in terms of perceptual pixel similarity [75] and no-reference image quality [72] . In summary, our contributions are three-fold:• We propose a comprehensive dataset tailored for seven image processing tasks. The dataset contains ∼ 1.01 million diverse paired input-output images along with corresponding image editing instructions. • We propose a new all-in-one instruction-guided diffusion model -PromptFix for low-level image-processing tasks. Extensive experimental results show that PromptFix outperforms previous methods in a wide variety of image-processing tasks and exhibits superior zero-shot capabilities in blind restoration and combination tasks. • We introduce two approaches -high-frequency guidance sampling and auxiliary prompt module to diffusion models to effectively address the issues of high-frequency information loss and the failure in processing the severe image degradation for instruction-based diffusion models in low-level tasks.2 Related WorkQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The claims in the abstract and introduction align with the detailed findings and discussions presented in the paper, accurately representing its scope and contributions.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
instruction-guided image editing: 
 Instruction-guided image editing significantly improves the ease and precision of visual manipulations by adhering to human directions. In traditional image editing, models primarily focused on singular tasks such as style transfer or domain adaptation [24, 55] , leveraging various techniques to encode images into a manipulatable latent space, such as those used by StyleGAN [34] . Concurrently, the advent of text-to-image diffusion models [26, 57, 63, 65] has broadened the scope of image editing [7, 25] . Kim et al. [35] showed how to perform global changes, whereas Avrahami et al. [4] successfully performed local manipulations using user-provided masks for guidance. While most works that require only text (i.e., no masks) are limited to global editing [17, 36] . Bar-Tal et al. [6] proposed a text-based localized editing technique without using any mask, showing impressive results. For local image editing, precise manipulations are possible by inpainting designated areas using either user-provided or algorithmically predicted masks [16] , all while preserving the visual integrity of the adjacent areas. In contrast, instruction-based image editing operates through direct commands like "add fireworks to the sky," avoiding the need for detailed descriptions or regional masks. Recent approaches utilize synthetic input-goal-instruction triples [7] and incorporate human feedback [76] to execute editing instructions effectively. Despite the advances in using diffusion models for various instruction-guided image editing tasks, there is still a notable gap in research specifically addressing instruction-guided image restoration with these models. Our study aims to bridge this gap by collecting a comprehensive dataset of paired low-level instruction-driven image editing examples and proposing an all-in-one model for low-level tasks and editing.Recent advancements in the development of Large Language Models (LLMs) have led to the emergence of powerful models with extensive capabilities [15, 31, 39, 43, 77] . These LLMs, pretrained on large-scale internet-based datasets, are equipped with broad knowledge bases that enhance their zero-shot and in-context learning abilities [5, 29, 51] . Furthermore, there is a growing focus on using LLMs for multimodal tasks [3, 30, 42, 45] , incorporating methods like vision-language alignment and adapter fine-tuning. These techniques ensure that the visual data processed by visual encoders is semantically aligned with the textual input of LLMs [43] . This approach has spurred significant advancements in Text-to-Image generation, leading to the development of various LLMbased diffusion models for these tasks [28, 40, 41] . Despite these successes, there remains a relative scarcity of research focusing on using large Vision-Language Models (VLMs) for instructed image editing, particularly in detailed, low-level editing tasks.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
data curation: 
 Current off-the-shelf image datasets [7, 74, 76] with instructional annotations primarily facilitate image editing research, encompassing tasks such as color transfer, object replacement, object removal, background alteration, and style transfer. Nevertheless, their overlap with low-level applications is limited. Moreover, we find it challenging to achieve satisfactory results for existing models in image restoration. Our goal is to construct a comprehensive visual instruction-following dataset specifically for low-level tasks. We obtain ∼ 1.01 million training triplet instances.Paired Image Collection. We initially gather source images from various existing datasets. Subsequently, we produce degraded and inpainted images to create an extensive set of paired image data. We compile approximately two million raw data points across eight tasks: image inpainting, object creation, image dehazing, image colorization, super-resolution, low-light enhancement, snow removal, and watermark removal. For the test set, we randomly select 300 image pairs for each task. More details about the dataset composition are provided in the Appendix A.1.For each low-level task, we utilized GPT-4 to generate diverse training instruction prompts P instruction . These prompts include task-specific and general instructions. The task-specific prompts, exceeding 250 entries, clearly define the task objectives. For example, "Improve the visibility of the image by reducing haze" for dehazing. The general instructions include five ambiguous commands that we retained as "negative" prompts to promote adaptive tasks. The specific instruction prompts used for training are detailed in the appendix. For watermark removal, super-resolution, dehazing, snow removal, low-light enhancement, and colorization tasks, we also generate "auxiliary prompts" for each instance. These auxiliary prompts describe the quality issues for the input image and provide semantic captions. More details are discussed in Section 4.2. 
methodology: 
 Let I ∈ R H×W ×3 denote the degraded input image. Our PromptFix model aims to enhance I using the prompt P and the diffusion model H. θ ← θ -e -λt ∇ θ L(I, D θ (zt→0)) 9: end forDiffusion models transform data into noise through gradual Gaussian perturbations during a forward process and subsequently reconstruct samples from this noise in a backward process. In the forward phase, an original data point, denoted as z 0 , is incrementally altered towards a Gaussian noise distribution ϵ ∼ N (0, I), according to the equation:EQUATIONwhere α t and σ t are coefficients that manage the signal-to-noise ratio at each interpolation point z t . This process aims to maintain variance, adopting coefficient strategies as detailed in sources such as [33] . Modeled as a stochastic differential equation (SDE) in continuous time, the forward process can be expressed as dz = f (z, t)dt + g(t)dw t , where f (z, t) is a vector-valued drift coefficient, g(t) is the diffusion coefficient, and w t represents Brownian motion at time t.The backward diffusion process, made possible by notable characteristics of the SDE, is rearticulated via Fokker-Planck dynamics [66] to yield deterministic transitions with consistent probability densities, creating the probability flow ODE:EQUATIONThis equation outlines a transport mechanism that is learnable through maximum likelihood techniques, applying the perturbation kernel of diffused data samples ∇ z log p t (z|z 0 ), as demonstrated in [32, 66] . Next, we sample z t ∼ N (0, I) to initialize the probability flow ODE, and the estimates for the score function via ε(z t , t)/σ t . We employ the Euler method [64, 66] among numerical ODE solvers to obtain the solution trajectory:z 0 ≈ H N • H N -1 • • • • • H 1 (z T ), where H denotes the diffusion model and N represents the neural function evaluations (NFEs) for sampling.During the training phase, a simple diffusion loss [26] is utilized, whereby the neural network still employs forward inference to predict noise. The sample data estimate z 0 can be obtained at any step t by using the current noisy data and the predicted noise and is derived as:EQUATIONTo reduce computational costs, the aforementioned diffusion process initiates from isotropic Gaussian noise samples in the latent space [57] , rather than the pixel space. This space transformation is facilitated through VAE compression [20] . The VAE autoencoder comprises an encoder E(•) and a left inverse decoder D(•). For instance, an image x can be encoded into a latent code E(x), which can then be approximately reconstructed back into the pixel space as x ≈ D • E(x).Given that low-level image processing focuses on handling degraded images rather than real-world images, we adopt the integration of a VLM to estimate an auxiliary prompt for the low-level image I. This auxiliary prompt encompasses both semantic captions and defect descriptions to enhance the semantic clarity of the target image, thereby addressing the instructional gaps inherent in low-level image processing tasks.Based on text dialogues parameterized by ω within a VLM V(•; ω), we employ a frozen VLM, specifically the InternVL2 [15] model, which integrates visual and linguistic modalities as inputs. We facilitate this model to receive paired degraded image I and a textual query Q. To handle the visual input, the InternVL2 first employs a pre-trained encoding model to map each modality into a shared representation space. The visual encoding model ϕ embeds I into the textual space, resulting in ϕ(I), which is then combined with the tokenized language embedding τ (Q). These combined embeddings are fed into the large language model, producing the textual response R.EQUATIONThe visual encoding model of InternVL2 has not undergone extensive fine-tuning in the degradation domain. To acquire an explicit understanding from both semantic and low-level defect perspectives, we meticulously curate the queries Q semantic and Q defect to guide InternVL2, respectively. Specific query instances are provided in the Appendix. As illustrated in Figure 4 and described by Equation 5, we concatenate the responses related to semantics and degradation textually, forming P auxiliary , which serves as the auxiliary prompt. This acts as a supplement to the instruction prompt P instruction .EQUATIONThe conditioned text prompts guide the diffusion model by injecting the embedding into the crossattention layer [57] . After obtaining the auxiliary prompt, a straightforward approach involves concatenating it with the user-input instruction prompt before feeding the text embedding into the diffusion model. However, this concatenation can render the entire prompt excessively long, leading to forced truncation during tokenization. Therefore, after utilizing the pre-trained CLIP visual encoder ViT-L/14 [54] to extract linguistic features, we process the text embeddings of P instruction and P auxiliary separately. We introduce additional cross-attention layers identical to the original ones, as depicted in Figure 2 . The embeddings of P instruction and P auxiliary are fed into the Key and Value heads of consecutive attention networks, respectively, thereby achieving an augmented cross-attend adaptation.There is a fundamental requirement in image restoration and generation tasks: the processed image must maintain high accuracy in semantics. We observe that vanilla VAE reconstructions tend to lose image details such as textual rendering, which contains high-frequency information, as shown in Figure 5 . Therefore, we propose high-frequency guidance sampling to balance the quality and fidelity of generation.The denoising sampling is based on the EDM formulation [33] . To maintain spatial information, we utilize a modified VAE Decoder D θ to map from the latent space to the pixel space. We modify the VAE decoder by passing skip-connect features from the VAE encoder through additional LoRA convolutions [27] to merge the feature map. The LoRA networks are initialized randomly, with their trainable parameters denoted as θ. Since the parameters of the LoRA convolution are lightweight, merely multi-step backpropagation can maintain high-frequency consistency without requiring extensive fine-tuning.We propose a fidelity constraint to model the spatial discrepancy between the image and the ground truth. We implement two types of high-pass operators to extract high-frequency signals from the degraded image. For the Fourier filtering operator F(•), we convert the generated image from the spatial to the frequency domain using the Discrete Fourier Transform [50] . High-frequency components are then isolated via high-pass filtering and reconstituted into an image through the Inverse Fourier Transform [50] . Meanwhile, we apply the Sobel [53] edge detection operator S(•) as a complement. The fidelity constraint evaluates the divergence between the high-frequency components of the ground truth and the processed image, ensuring the preservation of spatial information throughout the sampling process. Additionally, to obtain the image at time step t, we utilize predicted noise ϵ from diffusion model H to compute the z t→0 estimation at any time step. The fidelity constraint is calculated as follows: , InstructDiff [23] , and MGIE [21] ) for image processing, as well as low-level generalist techniques (PromptIR [52] , AirNet [38] , and Diff-Plugin [46] ) for image restoration.L(I, D θ (z t→0 )) = ∥F(I) -F(D θ (z t→0 ))∥ 2 2 + ∥S(I) -S(D θ (z t→0 ))∥ 25 Experiments 
experimental setup: 
 Implementation details. We train PromptFix for 46 epochs on 32 NVIDIA V100 GPUs, employing a learning rate of 1 × 10 -4 with the Adam optimizer. The training input resolution is set to 512 × 512, matching the capabilities of our backbone models, InternVL2 [15] and Stable Diffusion 1.5 [57] .To facilitate classifier-free guidance [7] , we randomly drop the input image latent, instruction, and auxiliary prompt with a probability of 0.075 during training. The hyperparameter λ for the time-scale weight in Algorithm 1 is empirically set to 0.001. For more implementation details, please refer to the appendix.Baselines and metrics. We adopt instruction-based generalist models, such as InstructP2P [7] , MGIE [21] , and InstructDiffusion [23] , as our primary comparison. MGIE employs VLM-guided techniques for image editing, while InstructDiffusion addresses overlapping tasks with our training objectives, including watermark removal and inpainting. Additionally, we evaluate all-in-one image restoration methods like AirNet [38] and PromptIR [52] (which do not support instruction input), as well as image restoration expert models, fine-tuned for specific sub-tasks [46, 73] . We assess the similarity of the generated images to the ground truth using metrics such as PSNR, SSIM [69] , and LPIPS [75] . For no-reference image quality evaluation, we utilize the ManIQA [72] metric.Table 1 illustrates the comparative analysis of image restoration and editing techniques, evaluated via LPIPS and ManIQA metrics. The expert model -Diff-Plugin shows limited but notable performance in low-light enhancement (LPIPS/ManIQA: 0.227/0.453) and desnowing (0.133/0.508). Among generalist methods, AirNet demonstrates balanced capabilities in tasks like desnowing and dehazing, achieving LPIPS/ManIQA scores of 0.245/0.589 and 0.039/0.780, respectively. However, the instruction-driven diffusion methods reveal a more nuanced picture, with PromptFix emerging as particularly promising. It excels in colorization (LPIPS/ManIQA: 0.233/0.489), object removal (0.054/0.810), and watermark removal (0.071/0.811), consistently outperforming others. InstructP2P and InstructDiff also perform well in specific tasks, such as low-light enhancement and dehazing, but do not match the overall versatility of PromptFix. MGIE, though effective in certain domains, lacks the consistency seen in "PromptFix (Ours)." This highlights the robustness and superior performance Figure 3 illustrates the visual comparison across all selected baseline models. In colorization, our PromptFix produces the most visually accurate and vibrant results, closely resembling the ground truth. For the watermark removal task, it effectively restores the original image without introducing artifacts, outperforming MGIE [21] and other methods. In desnowing and low-light enhancement, PromptFix achieves clearer and more natural outputs, significantly reducing noise and enhancing visibility. Additionally, in super-resolution, PromptFix demonstrates remarkable clarity and accuracy, preserving fine details and surpassing all comparative methods. In dehazing, although PromptFix's performance is visually comparable to image restoration experts PromptIR [52] and AirNet [38] , PromptFix outperforms the recent stable-diffusion-based method Diff-plugin [46] , achieving a clean, sharp appearance, and closely matching the ground truth.Effectiveness of High-frequency Guidance Sampling. The High-frequency Guidance Sampling (HGS) method is introduced to balance fidelity and quality. To validate the effectiveness of HGS, we conduct qualitative experiments and quantitative experiments. As depicted in Figure 5 , in a low-light scenario, the model aims to enhance the visibility (quality) of the input image while preserving its original textual details (fidelity). For baselines leveraging stable-diffusion as a generative prior, the strong compression capability of VAE also brings the issue of spatial information loss, as demonstrated by InstructDiff [23] , MGIE [21] , and Diff-Plugin [46] in Figure 5 . This issue is independent of the model's ability to effectively follow instructions. As shown by the variant "Ours w/o HGS," our method significantly enhances low-light images compared to the three baselines, yet still fails to retain small-scale textual structures. By incorporating HGS, as seen in "Ours," the proposed framework delivers a high-fidelity solution that also meets the low-light enhancement instruction. The usage of F(•) and S(•) improves the quality of the image generated, demonstrated by the quantitative results shown in Table 4 .Ours Input Diff-Plugin MGIE InstructDiffOurs w/o HGS VLM-guided blind restoration. We utilize InternVL2 [15] to generate auxiliary prompts and leave the instruction prompt empty. This approach enables users to input an image without the need to provide instructions for its restoration. We evaluate the model's performance on such blind restoration tasks, including low-light enhancement, desnowing, and dehazing. As shown in Table 3 , our model achieves performance comparable to four baselines, showing minimal perceptual differences from the ground truth and superior zero-shot capabilities. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [NA] Justification: We are ready to release data and code, along with comprehensive instructions, enabling faithful reproduction of the main experimental results.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
conclusion: 
 We present PromptFix, a novel diffusion-based model along with a large-scale visual-instruction training dataset, designed to benefit instruction-guided low-level image processing. PromptFix effectively addresses challenges related to spatial information loss and degradation adaptation by high-frequency guidance sampling and a VLM-based auxiliary prompt module. These mechanisms improve the model's performance in the instruction-based image-processing paradigm. Extensive experiment results demonstrate PromptFix's advanced capabilities of generating accurate and high quality images. In addition to the improvement in terms of conventional metrics, we observe that PromptFix is also effective at processing multi-task processing and achieving blind restoration in low-light enhancement, desnowing, and dehazing. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA]Justification: The paper does not focus on potential societal impacts, either positive or negative, of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 PromptFix DatasetThe PromptFix dataset contains 1,013,320 triplet instances among 7 different tasks containing object removal, image dehazing, colorization, debluring, low light enhancement, snow removal and watermark removal. Each instance contains an input image, a processed image, an instruction, and an auxiliary prompt generated by InternVL2 [15] which is only available for tasks except object removal. The dataset is available at https://huggingface.co/datasets/yeates/ PromptfixData.Object removal: Mulan [67] provides multi-layer image decomposition annotations, inpainting occluded areas and isolating instances in a scene on separate RGBA layers. We combine the background layer and n foreground object layers to obtain the ground truth image and the background layer and n + 1 foreground object layers to obtain the input image, where n ∈ N. Different object layers will be placed onto the background image based on their foreground-background relationships.Image dehazing: We use a combination of synthetic datasets (RESIDE [37] , SRRS [13] ) and realworld datasets (Dense-Haze [1] , O-Haze [2] ) for training, comprising 100 pairs of real-world images and 102,230 pairs of indoor and outdoor synthetic images from ITS [37] , OTS [37] , SOTS outdoor [37] , and nyuhaze500 [37] .Colorization: We utilize a subset of Laion-5b [60] and Flickr comprising 317,225 images and generate grayscale images.Deblurring: We use a combination of GoPro [49] , HIDE [61] , LFDOF [58] , RealBlur [56] , TextOCR [62] , Wider-Face [71] and a subset of CelebA [48] , which collectively contain 70,600 pairs of images.Low light enhancement: We use a combination of LOL [70] , SID [11] , SMID [10] , SDSD [68] , and construct a set of low light synthetic data based on Pexels and Flickr by reducing brightness and adding noise, which collectively contains 212,618 pairs of images.We use a combination of SRRS [13] , CSD [14] , RVSD [12] , and Snow100K [47] , which collectively contains 99,177 pairs of images.Watermark removal: We use a combination of CLWD [44] and LOGO30K [18] , which collectively contain 124,805 pairs of images. Without relying on user input instructions, our PromptFix achieves blind restoration for low-level enhancement, dehazing, and desnowing. However, we observe that this approach occasionally results in out-of-conditioned image control, where the model performs text-to-image generation based on the auxiliary prompt rather than image processing. Although blind restoration is a feature of our VLMbased Auxiliary Prompt Module, for known degradations, we recommend providing user-custom instructions to specify the restoration.High-frequency Guidance Sampling significantly helps preserve the original image details, counteracting the spatial information loss caused by VAE compression. Nevertheless, we find that adopting HGS makes the restored image slightly resemble the degraded image. While PromptFix remains promising relative to baselines, as shown in Figure 5 , results without HGS appear brighter than those with HGS. As we have consistently claimed, employing HGS involves a trade-off between fidelity and quality. In scenarios where VAE reconstruction is sufficiently faithful and user needs prioritize quality, HGS may be omitted.Below are the more detailed implementation specifics:• Q semantic : "Describe this image and its style in a very detailed manner"• Q defect : "Introduce the drawback of the image"Parameters of the Sobel Operator: This involves applying two 3 × 3 kernels:G x = -1 0 1 -2 0 2 -1 0 1 and G y = -1 -2 -1 0 0 0 1 2 1These kernels calculate gradients in the horizontal and vertical directions, respectively. By convolving these kernels with the image, we obtain the gradient magnitudes at each pixel. The default parameters include a mid-range intensity threshold to discern significant edges.Time-scale Weight λ: This weight is determined based on the time step of the diffusion model. When the time step is larger, there is more noise, and the fidelity constraint should be decayed. As depicted in Figure 7 , the impact of the high-frequency loss exponentially decays with increasing timestep. Empirically, we have set λ to 0.001.The time-scale weight, which is determined by the time step of the diffusion model, decreases as the time step increases, due to the increase in noise. Consequently, the fidelity constraint should be decayed accordingly.HGS Algorithm Notion Clarification: Using T to represent the neural function evaluations (NFEs) for sampling, and S to denote a series of hyperparameters:• S churn controls the level of stochasticity.• S noise adjusts the noise standard deviation.• S tmin and S tmax define the range of noise levels for stochasticity.We conduct an efficiency analysis for PromptFix and other recent baseline models, including Diffplugin [46] and InstructDiff [23] , by profiling the floating-point operations (FLOPs) of their respective diffusion models. As demonstrated in the table below, our method exhibits comparable efficiency to these advanced models. This analysis reveals the capability of PromptFix to maintain high computational efficiency while delivering superior performance in various image-processing tasks.Diff-Plugin [46] 1.1 InstructDiff [23] 0.8 PromptFix (Ours) 0.8General PromptFix Results. We present additional image processing results in Figures 9-15, encompassing tasks including watermark removal, colorization, low-light enhancement, super-resolution, dehazing, desnowing, inpainting, and object removal via bounding boxes. These results are generated by our PromptFix model, utilizing an all-in-one pre-trained weight and guided by user-customized instructions.Mutli-task processing and blind restoration. In Figure 8 , we present the qualitative results of multitask processing as a complement to the quantitative assessments detailed in Table 2 . Additionally, we provide further visualizations of VLM-guided blind restoration outcomes as in Figure 16 . "Cleanse this photo of watermark.""Scrub logo off the image." "Immerse this image in a colorful palette.""Colorize the image to enhance its visual appeal.""Paint the image with color tones." The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:• You should answer [Yes] , [No] , or [NA] .• [NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.• Please provide a short (1-2 sentence) justification right after your answer (even for NA).The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.IMPORTANT, please:• Delete this instruction block, but keep the section heading "NeurIPS paper checklist",• Keep the checklist subsection headings, questions/answers and guidelines below.• Do not modify the questions and only use the provided macros for your answers.38th Conference on Neural Information Processing Systems (NeurIPS 2024).(6) Given that z t>0 represents a noisy latent variable, assigning equal weight to each timestep's latent is impractical. To mitigate the cumulative error induced by this practice, we introduce a time-scale weight e -λt . The overall sampling algorithm is detailed in Algorithm 1. 


Decision: After the rebuttal, the paper received the ratings of 4/5/6/6. Basically mostly all the concerns are addressed by the rebuttal provided by the authors. After examining the reviews and the rebuttal for the reviewer with a rating of 4, the AC believes the concerns should be addressed as well, as these concerns are the common ones shared by other reviewers. Generally, the reviewers recognize the strengths of the paper: a constructed dataset for image editing with prompts, PromptFix provides an interactive approach for image editing, and the experiments are compressive and convincing. Thus, the AC recommends acceptance of the paper. The authors are recommended to incorporate the suggestions from the reviewers for the camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper addresses low-level image processing tasks using a unified, Diffusion-based method. The key idea is to construct a dataset comprising pairs of editing instructions and targets for a variety of tasks and fine-tune a pre-trained text-to-image Diffusion Model on this dataset. Further innovations include augmenting editing instructions with text descriptions from a Vision Language Model (VLM) and a training loss that penalizes difference in high-frequency image components to facilitate detail preservation. The paper presents qualitative and quantitative results on a selected set of low-level image processing tasks, where the proposed method outperforms instruction-based Diffusion baselines and sometimes other task-specific models.
strength_weakness: strength_weakness: - Lack of novelty. Instruction-based image editing using Diffusion models dates back to InstructPix2Pix, which similarly utilizes text conditioning to unify arbitrary image processing tasks. Further, I don't think it is fair to sell the dataset as a main contribution of the paper. Instruction generation using GPT-4 is not new (Both InstructionPix2Pix and LLaVA did that). The pipeline for corrupting GT images also follows standard practice. The auxiliary prompts are new but are specific to the the proposed approach. The techniques for enhancing spatial alignment are also new and seem effective, but they alone cannot justify the broad claim of the paper.

- Regarding the two proposed techniques (auxiliary prompts and loss on high-frequency components), there is no ablation study showing they are absolutely needed for the method to succeed. Qualitative and quantitative experiments are needed to show that auxiliary prompts can help in the case where input images are corrupted. Similarly, ablation experiments are needed to show how precisely the F(.) and S(.) terms in Equation 6 affects output quality.

- Some model details are lacking. For example, I am not sure I fully understand how auxiliary prompts are incorporated into the Diffusion UNet. If the cross-attention layers are replicated, are they fine-turned during training? The paper leaves me with the impression that only the LoRA convolutions are learned.
questions: questions: - Are the experiments on low-light enhancement, dehazing, desnowing, etc. performed on simulated data? The artifacts look unrealistic to me. How does the model perform on real data? I am concerned about generalization on real data since the model is exclusively trained on simulated data.

- The paper claims in the abstract that the method "achieves comparable inference efficiency" with the baselines. Please provide concrete numbers to prove this claim.
limitations: limitations: Limitations are discussed in A.2. The model performance might be sensitive to user-provided instructions. Additionally, passing along high-frequency textures from the input image inevitably copies some artifacts to the output.
paper_summary: paper_summary: This paper employs prompts to perform low-level image restoration tasks using pretrained diffusion models. To facilitate this, a substantial paired dataset with image restoration instructions was collected. The proposed method relies on latent diffusion models, incorporating the input low-quality image as an additional input and using a VLM to generate auxiliary prompts, serving as another text condition for the network. During sampling, the paper introduces High-frequency Guidance Sampling, wherein the VAE decoder is optimized to better capture the high-frequency details of the input images.
strength_weakness: strength_weakness: 1. The dataset is collected by manually performing the degradation which has a distribution gap with the real-world low-quality images. How will the method perform for the real-world image degradation task, for example, motion blur captured by the phone camera?
2. Regarding the comparison, since the baselines are not trained on the collected dataset, it might be a bit unfair as they aren't aware of the low-level restoration tasks. It would be better to compare the methods under a similar setting.
questions: questions: 1. Regarding the additional cross-attention layers, do you simply add another cross-attention layer after all the existing cross-attention layers and do the two cross-attention sequentially or parallel? And do you clone the weights as well?
2. In Alg 1, it seems that the decoder is updated during the testing time for each input? Since the loss is based on the input (low-quality image), will it affect the output image quality?
3. I'm also curious about how robust the method is for different instructions. For example, if we use different text prompts as instructions, how will the image quality change?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper introduces PromptFix, a unified model designed to intelligently interpret and execute customized human instructions across a variety of low-level image tasks. To address the issue of spatial information loss in stable diffusion, PromptFix introduces a high-frequency guidance sampling strategy. Additionally, to tackle the degradation adaptation problem, PromptFix incorporates an auxiliary prompt module, providing models with more descriptive text prompts to enhance controllability in image generation.
strength_weakness: strength_weakness: 1. The authors claim that when user-input instructions are discarded, PromptFix occasionally performs text-to-image generation based on the auxiliary prompt rather than image processing tasks. It is recommended that authors provide examples of these failed cases and evaluate whether the generated images can preserve the layout and structure of the input images.
2. There is a concern that PromptFix might learn a mistaken shortcut by memorizing the degradation type of the input image. Therefore, authors are encouraged to demonstrate PromptFix's ability to correctly execute tasks, such as coloring a snowy image while preserving the snow.
3. Authors assert the superior generalization and zero-shot capabilities in blind restoration and combination tasks. Therefore, it is suggested that authors evaluate the model on out-of-distribution and real-world datasets to substantiate these claims.
questions: questions: The authors claim that the HGS strategy has the potential to introduce spatial information loss. However, the visual results in Figure 5 are insufficient to prove HGS's effectiveness in maintaining image fidelity. Quantitative results should be presented to demonstrate that the proposed HGS module is indispensable to PromptFix.
limitations: limitations: Authors claimed the limitations in the Appendix.
paper_summary: paper_summary: The paper introduces PromptFix, a novel framework that significantly enhances the capability of diffusion models in following human instructions for a wide range of image-processing tasks. The authors propose a comprehensive multi-modal dataset and further design a frequency-based diffusion model trained on this dataset. Experiments show competitive performance on various restoration tasks.
strength_weakness: strength_weakness: 1. While instructions are necessary for users, the types of degradation tasks (such as snow removal and low-light enhancement) are clearly defined. In other words, for images with the same type of degradation (such as foggy images), how to choose different instructions to achieve the best results remains to be clarified. Additionally, it should be considered whether users can use instructions other than the task-specific prompts provided by the authors.
2. The modules used are relatively common. The AuxiliaryPrompt, serving as an information cue derived from the image itself, has been utilized in the literature as referenced in [1,2]. And High-frequency Guidance is also a commonly used method [3,4], albeit in this paper, it is constrained in the form of a loss.

Ref:

[1] Lin J, Zhang Z, Wei Y, et al. Improving image restoration through removing degradations in textual representations. CVPR 2024.

[2] Chiu M T, Zhou Y, Zhang L, et al. Brush2Prompt: Contextual Prompt Generator for Object Inpainting. CVPR 2024.

[3] Miao Y, Deng J, Han J. WaveFace: Authentic Face Restoration with Efficient Frequency Recovery.  CVPR 2024.

[4] Zhao C, Cai W, Dong C, et al. Wavelet-based Fourier information interaction with frequency diffusion adjustment for underwater image restoration. CVPR 2024.
questions: questions: 1. It is important to consider whether the comparative methods have been retrained or fine-tuned on the same dataset. If not, the fairness of the comparison is questionable.
2. In line 205 of the paper, "low-pass operators" might be incorrect; it should probably be "high-pass operators".
3. The test set was randomly selected, and its domain is similar to that of the training set. It is necessary to supplement the comparison with untrained standard datasets.
4. The paper lacks ablation experiments for auxiliary prompts. Additionally, the High-frequency Guidance method lacks numerical metrics and only presents visual results.
5. In Table 1, for the unified restoration task, both PromptIR and AirNet, despite not incorporating instructions, demonstrate good performance, with half of the metrics surpassing those of PromptFix. This seemingly limits the perceived superiority of the proposed method's performance.
6. In line 228, The MGIE[19] was published in ICLR 2024. It has been incorrectly marked in the references.
limitations: limitations: As shown in the Weaknesses and Questions.

Comments: title: A friendly reminder
A friendly reminder
A friendly reminder
comment: Thank you! We’re pleased that our rebuttal has satisfactorily addressed your concerns. We greatly appreciate your suggestions for improving our paper and your positive feedback.
Thanks for the response. After reading the rebuttal and other reviews, I believe my concerns have been resolved and thus would like to increase my initial score to 5.
Thank you sincerely for your engaged and increased score!

Your constructive feedback has been instrumental in enhancing the clarity and contribution of our paper. The positive discussions we've had and the recognition of our efforts truly encapsulate the essence of the OpenReview process.

Your time and efforts are immensely appreciated.
Thanks for the responses. I believe my previous concerns have mostly been addressed and have changed my score.
Dear Reviewer,

The authors have posted an author response here. Could you go through the rebuttal and update your opinion and rating as soon as possible?

Your AC
Dear Reviewer,

The authors have posted an author response here. Could you go through the rebuttal and update your opinion and rating as soon as possible?

Your AC
As the phase of author-reviewer discussions draws to a close, we are so pleased to note your recognition of our efforts and the raised score.

We are grateful for the valuable suggestions you posed and appreciate the time and effort you devoted to the review process.
Thanks to the authors for their detailed responses. After considering the other reviews and the replies provided, I can confirm that the authors have addressed all my concerns. I raised my final rating to weak accept.
Dear Reviewer,

I would like to send a kind reminder. Has our response addressed your concerns? The reviewer discussion period is nearing its end, and we eagerly await your reply. Your suggestions and comments are invaluable to the community. Thank you!

Best, The authors
Dear Reviewer,

I would like to send a kind reminder. Has our response addressed your concerns? The reviewer discussion period is nearing its end, and we eagerly await your reply. Your suggestions and comments are invaluable to the community. Thank you!

Best, The authors
Dear Reviewer,

I would like to send a kind reminder. Has our response addressed your concerns? The reviewer discussion period is nearing its end, and we eagerly await your reply. Your suggestions and comments are invaluable to the community. Thank you!

Best, The authors
3. **Explanation of the HGS Design**

    The HGS is designed to preserve crucial structures from the original image and avoid the distortion of high-frequency details, such as the text shown in Figure 5. Traditional low-level processing methods address this effectively by employing skip-connection layers in the pixel space, ensuring these details remain intact.

    However, in Diffusion-based methods, skip-connections are only implemented in the latent space within the U-Net. Consequently, baseline methods like Diff-Plugin and MGIE, which focus solely on training the U-Net, can only retain details at a downsampled spatial dimension. When these methods perform upsampling using a frozen VAE decoder, the absence of skip-connections in the VAE leads to the distortions.

    To address this issue, updating the U-Net alone does not work; it is essential to update the decoder as well. Additionally, Stable Diffusion operates at the latent level ($z_t$), so directly accessing pixel-level noised map ($x_t$) is not feasible.

4. **Clarification of the Auxiliary Prompt**

    We would like to gently point out that the Auxiliary Prompt *is displayed in text form*, as illustrated by the blue and yellow prompts in Figure 4. Specifically, the text-based auxiliary prompt provides supplementary information through the U-Net's cross-attention layers, rather than the image encoder, as depicted in Figure 2.

    The proposed HGS enhances image quality in most cases, as demonstrated by the improvement in LPIPS from 0.2068 to 0.1600, as shown in the General Response II table. We observed that HGS does not consistently improve performance in all scenarios, as noted in the limitations section. We will make further improvements in this direction in future work.
  
    However, preserving the details of the original image remains an unresolved issue for all diffusion baselines. Our HGS method is effective in addressing this problem, as noted in your comments. We are among the first to make significant progress in tackling this issue. Therefore, we believe that our two proposed modules are both effective and distinct.

We sincerely hope this response addresses your concerns. Please let us know if you have any further questions. Thank you again!
We sincerely appreciate your feedback and acknowledgment of the validity of our experiments in General Responses II and IV, which demonstrate the effectiveness of our proposed method. We would also like to address the concerns raised in your comments:

1. **Regarding the Concern on General Response - I Data**

   We confirm that none of the data used originates from the training set. The data sources include online platforms, such as watermarked images from Adobe Stock and Shutterstock, black-and-white photography from Pexels, dark scenes from films, and personal mobile photography. This approach aims to validate the generalization capability of our method using real-world data, rather than relying on simulated data created by degrading specific images.

2. **On Providing Examples for General Response - III's Instructions**

   We show some examples of instructions from General Response III. Due to space constraints, we provide one example for each task:

   |  | $\mathbb{B}$ | $\mathbb{C}$ |
   |:---:| :---| :---|
   | *Watermark Removal* | Would you mind giving the image a fresh start by removing the watermark? | Could you perform a bit of digital alchemy and transform the image by removing that watermark? The picture deserves to be seen in its most pristine form, free from any distractions. Let the original artwork emerge unblemished, allowing every detail to shine through. Your skill in making such transformations would be greatly appreciated. |
   | *Colorization* | Could you breathe life into this image by adding vibrant colors that capture its essence?  | Would you be able to transform this image by imbuing it with a rich palette of colors? Imagine each stroke of color enhancing the depth and emotion within the scene, turning the monochrome into a vivid masterpiece. Your artistic touch could reveal hidden layers, adding warmth and character to every detail. The final creation will undoubtedly captivate the eye. |
   | *Dehazing* | Please lift the fog from this image, revealing its crisp and vibrant essence. | Imagine peeling back a veil to reveal the true clarity beneath—could you do the same for this image by removing the haze? The picture deserves to be seen in all its vivid glory, free from any clouding effects. Your skill in restoring sharpness would transform this image into something truly striking, with every detail standing out beautifully. |
   | *Snow Removal* | Please sweep away the snowy blanket covering this image. | Could you bring the image back to its original state by removing the snow that currently veils it? The scene underneath holds a story waiting to be told, free from the cold layer above. Allow the true colors and details to shine through, unmasked by the wintry cover. Your expertise in restoring this image to its pristine form would be invaluable. |
   | *Super Resolution* | Could you work your digital magic to sharpen this image and enhance its resolution? | Would you kindly apply your expertise to refine this image by eliminating the blur and boosting its resolution? The goal is to reveal the full clarity and sharpness hidden within, making every detail stand out. By enhancing its quality, you’ll allow the image to achieve its true potential, presenting it with the vividness and precision it merits. |
   | *Low-light Enhancement* | Could you boost the low light in this image to reveal more detail and clarity? | Could you work your magic on this image by subtly enhancing its low-light areas? The goal is to brighten up the dim portions without altering the overall mood, allowing the hidden details to emerge while maintaining the original tone. A careful balance in lighting adjustment will ensure the image remains true to its essence, while also improving visibility and depth. Your expertise in making this enhancement would be greatly appreciated. |
Thanks for the response. The explanation is generally reasonable. 

For General Response-II, the results show that HGS and Auxiliary Prompting are effective.

For General Response-IV, the fine-tuning of InstructDiff demonstrates the effectiveness of the proposed method.

But I still have some concerns and will keep my score.

For General Response-I, please confirm that test results on untrained real-world data have been provided. This will demonstrate the method's generalization performance.

For General Response-III, it is recommended to provide examples of the instructions designed in B & C settings, and explain how the GPT-4 generated instructions are guided by the prompts.

For Q2, HGS is a loss that does not require training, and each input image needs to fine-tune the decoder during the sampling process.
Is this design reasonable? Please explain why the decoder is updated instead of updating xt or the U-Net.

For Q7, AuxiliaryPrompt provides supplementary information about the image, which is also input through the image encoder, although it is not displayed in text form. 
As for High-frequency Guidance, some high-frequency information in real-world degraded images contains noise, making this guidance somewhat rough. 
Additionally, in the final steps of the diffusion model, it could have a negative impact. 
Therefore, these two contributions are somewhat common.