PromptFix: You Prompt and We Fix the Photo
Authors: Yongsheng Yu
Ziyun Zeng
Hang Hua
Jianlong Fu
Jiebo Luo
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion model
image processing
Primary area: diffusion_based_models

Abstract: Diffusion models equipped with language models demonstrate excellent controllability in image generation tasks, allowing image processing to adhere to human instructions. However, the lack of diverse instruction-following data hampers the development of models that effectively recognize and execute user-customized instructions, particularly in low-level tasks. Moreover, the stochastic nature of the diffusion process leads to deficiencies in image generation or editing tasks that require the detailed preservation of the generated images. To address these limitations, we propose PromptFix, a comprehensive framework that enables diffusion models to follow human instructions to perform a wide variety of image-processing tasks. First, we construct a large-scale instruction-following dataset that covers comprehensive image-processing tasks, including low-level tasks, image editing, and object creation. Next, we propose a high-frequency guidance sampling method to explicitly control the denoising process and preserve high-frequency details in unprocessed areas. Finally, we design an auxiliary prompting adapter, utilizing Vision-Language Models (VLMs) to enhance text prompts and improve the model's task generalization. Experimental results show that PromptFix outperforms previous methods in various image-processing tasks. Our proposed model also achieves comparable inference efficiency with these baseline models and exhibits superior zero-shot capabilities in blind restoration and combination tasks.

PromptFix: You Prompt and We Fix the Photo
Yongsheng Yu!
Ziyun Zeng'*
Hang Hua'
Jianlong Fu
Jiebo Luo!
University of Rochester, Miicoosoft Research
(yyu90,77eng24) @urrochhster..du, (hhua2.jluo) @ [hhuuz/jluo@@csncchestteera. rochestectt,, jian jianf microsof.ccooooommmmmmmmm
Abstract
Diffusion models equipped with language models demonstrate excellent controlla- bility in image generation tasks, allowing image processing adhere to human instructions. However, the lack of diverse cnetructionfoolowiiii data hampers the development of models that effectively recognize and execute user-customized instructions, particularly in low-level tasks. Moreover, the stochastic nature of the diffusion process leads to deficiencies image generation or editing tasks that require the detailed preservation of the generated images. To address these limita- tions, we propose PromptFix, a comprehensive framework that enables diffusion models to follow human instructions to perform wide variety of dataset that covers
tasks. First, we construct a large-scale instruction-followiin comprehensive image-prone processing tasks, including low-level tasks, image editing, and object creation. Next, we propose a sigh-freuuency guidance sampling method explicitly control the denoising process and preserve high-frequency details in un- processed areas. Finally, we design an auxiliary prompting adapter, utilizing Vision- Language Models (VLMs) enhance prompts and improve the model's task generalization. Experimental results show that PromptFix outperforms previous methods various sanage-processingg tasks. Our proposed model also achieves com- parable inference efficiency with these baseline models and exhibits superior zero- shot capabilities in blind restoration and combination tasks. The dataset and code are available at Ihttps:/ www www.yonnghenggu hotpeeennyuu.ooooeeeeemmeeeeee/ com/ PttnnPPrmptFi11 -age
Introduction
In recent years, diffusion models [19 57 66 have achieved remarkable advancements in text- to-image generation. Benefiting from large-scale training on image-text pairs [59], these models generate highly realistic and diverse images that align with text prompts. They have been successfully applied to various real-world applications, including visual design, photography. digital art, and the film industry. In addition, models trained with instruction-follovii promising results in understanding human instruction and performing the corresponding image- processing tasks. Previous studies 23 74 73 have illustrated that with dustruction-following data, we can simply fine-tune a text-to-image generation model to perform various vision tasks such image editing 77 21 74], object detection [22], segmentation 23], inpainting 23 73, and
data [7] have shown
depth estimation 22 9 To follow the success of these methods. we train our model utilizing
input-goal-ing instrrussstttnnnnno maput-goal-nnstruccion triplet data for low-level image-processing tasks.
We first overcome the challenge of lacking the instruction-following data for low-level tasks. Specifi- cally, collect image pairs by generating degraded images from the source images and adopting data from existing datasets. Then, we employ GPT4 [51] to generate the diverse text instructions for each task. We obtain 1.01 million
-ansttaannn triplets in the collected dataset. This dataset covers various low-level tasks including image inpainting, object creation, image dehazing, colorization, super-resolution, low light enhancement (8). snow removal, and watermark removal. We enrich the dataset through back-translation augmentation by swapping the inpainted and origi-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

bm.
Scrub logs mage
Enhance image
by rensving
Erose the iron the ragg
Improve image qua try by nemoving blur
this picture with colore.
Filter from this
Brightes the image ingrove visibiily.
Figure 1: We propose PromptFix, unified diffusion model capable of performing multiple image- processing tasks. It can understand user-customized editing instructions and perform the correspond- ing tasks with high quality. One of the key advantages of PromptFix high-frequency information preservation, ensuring that image details are maintained throughout VAE decoding. PromptFix can handle various images with different aspect ratios.
nal images within the triplet and inverting the semantic orientation of the prompt. This technique effortlessly converts datasets from object removal object creation. We also provide comprehensive details in Section 3
With the dataset, we design user-customized instructions and perform the corresponding low-level image-processing tasks. In PromptFix, we address several challenges that compromise the performance of the model. First, the use of stable-diffusion architecture [57] as a generative prior often faces the issue of spatial information loss, which is caused by VAE compression (0)]. Unlike unconditional or text-to-image generation, maintaining spatial detail consistency in image processing poses significant challenges, particularly with feg-frrey components like text, as shown in Figure [ To tackle this problem. we introduce High -igg-ffreuency Guidance Sampling, in which we use low- pass filter operator (50) (3 to calculate the fidelity constraint and integrate VAE skip-connect features during inference with lightweight LoRA 27 fusion. Second, since the generative prior is not trained on low-level images, so relying solely on instructions may not always yield the desired outcome, especially when the image degradation is severe. To tackle this degradation adaptation problem, introduce an auxiliary prompt
diffusion-bain based model named PromptFix that understand

---

module to provide models with more descriptive text prompts to enhance controllability for image generation. The auxiliary text prompt can be obtained by VLMs 49 This approach introduces semantic caption for a degraded image and the description of its defects, such as blurriness or insufficient lighting. The auxiliary prompi module is implemented by an additional attention layer in diffusion U-Net that adapts both instruction and auxiliary prompts as conditions and intermittently omits instructional prompts during training. We identify three key advantages of this approach: 1) enabling the model to process images with severe degradation, such as extremely low-resolution images, 2) adapting the model for blind restoration for different types of image degradation, and 3) providing additional pathways for more precise semantic representation of the target image.
Experimental results demonstrate that our model achieves superior performance in the instruction- based paradigm across three image editing tasks (colorization, watermark removal, object removal) and four image restoration tasks (dehazing, desnowing, super-resolution, and low-light enhancement) in terms of perceptual pixel similarity 75 and no-reference image quality [22]. In summary, our contributions are three-fold:
We propose comprchensive dataset tailored for seven image processing tasks. The dataset contains 1.01 million diverse paired input-output images along with corresponding image editing instructions.
We propose a new all-in-one instruction -ratuddtio-guidedd diffusion model - PromptFix for low-level proccssio tasks. Extensive experimental results show that PromptFix outperforms previous methods in a wide variety of image-proree processing tasks and exhibits superior zero-shot capabilities in blind restoration and combination tasks.
We introduce two approaches module to diffusion models effectively address the issues of nigh-frequency information loss and the failure in processing the severe image degradation for instruction- baase diffusion models in low-level tasks.
guidance sampling and auxiliary prompt
2 Related Work
2.1 Instruction-guided Image Editing
Instruction-guiide image editing significantly improves the ease and precision of visual manipulations by adhering to human directions. In traditional image editing, models primarily focused on singular tasks such as style transfer or domain adaptation 224 (55, leveraging various techniques to encode
images into a manipulatable latent space, such as those used by StyleGAN [34]. Concurrently, the advent of -xxaeto-mmage diffusion models 26 57 63 65 has broadened the scope of image editing 7 (25). Kim et al. (35] showed how to perform global changes, whereas Avrahami et al. [4] successfully performed local manipulations using user-providd
masks for guidance. While most works that require only text (i.e., no masks) are limited to global editing [17] 36) Bar-Tal et al.
6 proposed text-basee localized editing technique without using any mask, showing impressive results. For local image editing, precise manipulations are possible by inpainting designated areas using either user-provin ussviprovided or algorithmically predicted masks [16], all while preserving the visual integrity of the adjacent areas. In contrast, instruction-based image editing operates through direct commands like "add fireworks to the sky," avoiding the need for detailed descriptions or regional masks. Recent approaches utilize synthetic coptttgoallinttructioon triples (7] and incorporate human feedback (76) to execute editing instructions effectively. Despite the advances in using diffusion models for various instruction gugiddd image editing tasks, there is still a notable gap in research specifically addressing instruction -gguddd image restoration with these models. Our study aims to bridge this gap by collecting comprehensive dataset of paired low-level instruction-driven image editing examples and proposing an all-in-one model for low-level tasks and editing.
2.2 Large Language Models for Vision
Recent advancements in the development of Large Language Models (LLMs) have led to the emergence of powerful models with extensive capabilities [5 31
77 These LLMs, pre- trained on large-scale internet-based datasets, are equipped with broad knowledge bases that enhance their zero-shot and in-context learning abilities
29
51. Furthermore, there a growing focus incorporating methods like vision- vasion-language
on using LLMs for multimodal tasks [3] 30 42 45
3

---

User-Customizes Instructions
"Remave the sitting sefa"
"Apply exposure correction to this photo.
Auxiliary Prompts
"cimage captions, <image
VLM InternVL2
"<This photo buildig depth ceptume enewy
GLIP Encoder
dimensioo, artificel appearance may
Instruction Embedding
Prompt Embedding
VAE Encoder
Cross Allention
Cross Allention
VAE Decoder
U-Net
Output Images
Input Images
Nolsy latent
Figure The architecture of our proposed PromptFix.
alignment and adapter fine-tuning, These techniques ensure that the visual data processed by visual encoders is semantically aligned with the textual input of LLMs [43]. This approach has spurred significant advancements in Text-to- Text-to-Image generation, leading to the development of various LLM- based diffusion models for these tasks [28 40,44) 4I]. Despite these successes, there remains relative scarcity of research focusing using large Vision-Language Models (VLMs) for instructed image editing, particularly in detailed, low-level editing tasks.
Data Curation
Current off-the-shelf image datasets 7 74 76 with instructional annotations primarily facilitate image editing research, encompassing tasks such color transfer, object replacement, object removal, background alteration, and style transfer. Nevertheless, their overlap with low-level applications limited. Moreover, find it challenging to achieve satisfactory results for existing models in image restoration. Our goal is to construct a comprehensive visual intructtio-followwe dataset specifically for low-level tasks. We obtain 1.01 million training triplet instances.
Paired Image Collection. We initially gather source images from various existing datasets. Sub- sequently, we produce degraded and inpainted images create an extensive set of paired image data. We compile approximately two million raw data points across eight tasks: image inpainting, object creation, image dehazing, image colorization, super-resolution. low-light enhancement, snow removal, and watermark removal. For the test set, we randomly select 300 image pairs for each task. More details about the dataset composition provided in the Appendix A.I A.I
Instruction Prompts Generation. For each low-level task, we utilized GPT-4 to generate diverse training instruction prompts P Pinstrutioon These prompts include task speeiseecific and general instructions. The task-specific prompts, exceeding 250 entries, clearly define the task objectives. For example, "Improve the visibility of the image by reducing haze" for dehazing. The general instructions include five ambiguous commands that we retained "negative" prompts to promote adaptive tasks. The specific instruction prompts used for training are detailed in the appendix. For watermark removal, super-resolution, dehazing, snow removal, low-light enhancement, and colorization tasks, we also generate "auxiliary prompts" for each instance. These auxiliary prompts describe the quality issues for the input image and provide semantic captions. More details are discussed in Section 4.2
Methodology
Let X denote the degraded input image. Our PromptFix model aims to enhance / using the prompt P and the diffusion model H.

---

Algorithm 1 High-frequency Guidance Sampling.
Input: H. P. Hyper-parameter: X, (or far Mi Schurns Smoisn. Semin Stmax
1: sample ~ (0,00o71
min
Saturr v2-1)
if af (SsiiiniinSmmmm otherwise
2: forte [T. (T.....:1)) 1] do
sample c; (0, S Smoseo))
4: dt
+++ 21 + 00 0. Et, 8r + 0t + 7:00
D Increase noise temporarily and injoct new noise for state transition. D Return predicted noise from neural network and denoised latent.
5:
é(Z,, 1), 2-11 &- H(2, 1, I.P)
6:
2t-1 2, + (ac-1 ōi) (2r 2, 22-1)/6-
Execute Euler step moving forward from 0--1- Equation y
7:
ZC-4 (2t 0: (2e, 1))
8: A a
Vol (1, Do (Zc-su))
9:
end for
4.1 Diffusion Model
Diffusion models transform data into noise through gradual Gaussian perturbations during forward process and subsequently reconstruct samples from this noise in a backward process. In the forward phase, original data point, denoted as Zo, is incrementally altered towards Gaussian noise distribution e (0, I), according to the equation:
(Zo, e, () = 0z + oCe, Vt E [0,
where ae and T7 are coefficients that manage the signal-to-noise ratio at each interpolation point z This process aims to maintain variance, adopting coefficient strategies as detailed in sources such as stochastic differential equation (SDE) in continuous time, the forward process can be expressed as dz f(z, t)dt + g(t) where (z, t) is vector-valued drift coefficient, g(t)
33
Modeled
the diffusion coefficient, and wt represents Brownian motion at time
The backward diffusion process, made possible by notable characteristics of the SDE, is rearticu- lated via Lakker-Planchs dynamics (66] yield deterministic transitions with consistent probability densities, creating the probability flow ODE:
dz f(z. t) 2 v log dt.
(2)
This equation outlines a transport mechanism that is learnable through maximum likelihood tech- niques, applying the perturbation kernel of diffused data samples V, log Pt (z|2z)) as demonstrated in [32 [66). Next, we sample Z, ~ (0, I) to initialize the probability flow ODE, and the estimates for the score function via é(z,, t)/o,. We employ the Euler method 664 666 among numerical ODE solvers to obtain the solution trajectory: Zp Hy HN-1
1 (ZT), where H denotes the
diffusion model and N represents the neural function evaluations (NFEs) for sampling.
During the training phase, a simple diffusion loss (26 is utilized, whereby the neural network still employs forward inference to predict noise. The sample data estimate can be obtained at any step t by using the current noisy data and the predicted noise and is derived as:
Ze (zē(Ze,
Z
To reduce computational costs, the aforementioned diffusion process initiates from isotropic Gaussian noise samples in the latent space [57], rather than the pixel space. This space transformation is facilitated through VAE compression (20). The VAE autoencoder comprises an encoder E(-) and a left inverse decoder D(-). For instance, an image z can be encoded into latent code E(x), which can then be approximately reconstructed back into the pixel space as x = D o F(x)).
--M-b Auxiliary Prompt Module
Given that low-level image processing focuses on handling degraded images rather than real-world images, we adopt the integration of VLM to estimate an auxiliary prompt for the low-level image This auxiliary prompt encompasses both semantic captions and defect descriptions to enhance the semantic clarity of the target image, thereby addressing the instructional gaps inherent in low-level image processing tasks.
5

---

Based on text dialogues parameterized by w within a VLM V(:w), we employ frozen VLM, specifically the InterVLL2 [S] model, which integrates visual and linguistic modalities as inputs. We facilitate this model to receive paired degraded image T and a textual query Q. To handle the visual input, the Intern V1.2 first employs a pre-trained encoding model to map each modality into shared representation space. The visual encoding model Φ embeds / into the textual space, resulting in σ 7), which is then combined with the tokenized language embedding (2). These combined embeddings are fed into the large language model, producing the textual response R.
R(I, Q) = V(o(I),
The visual encoding model of InternVL2 has not undergone extensive fine-tuning in the degradation domain. To acquire an explicit understanding from both semantic and low-level defect perspectives, we meticulously curate the queries Qsemantic and Qaeied to guide Intern VL2, respectively. Specific query instances are provided in the Appendix. As illustrated Figure f and described by Equation [ we concatenate the responses related to semanties and degradation textually, forming P Poxxiliary. which serves as the auxiliary prompt. This acts as a supplement to the instruction prompt Pinstruction-
P Pauxiliary [R(T, @semantic), ), R(T, 2detsa)
(S)
The conditioned text prompts guide the diffusion model by injecting the embedding into the cross- attention layer [57]. After obtaining the auxiliary prompt, a straightforward approach involves concatenating with the user-input instruction prompt before feeding the text embedding into the diffusion model. However, this concatenation can render the entire prompt excessively long, leading to forced truncation during tokenization. Therefore, after utilizing the pre-trained CLIP visual encoder ViT-L/144 (54] to extract linguistic features, we process the text embeddings of Pimstuction and P Pauxiliary separately. We introduce additional cross-attention layers identical to the original ones, as depicted in Figure 2 The embeddings of P Pinstmuction and P Pauxiliary are fed into the Key and Value heads of consecutive attention networks, respectively, thereby achieving an augmented cross-attend adaptation.
4.3 High-frequency Guidance Sampling
There is fundamental requirement in image restoration and generation tasks: the processed image must maintain high accuracy in semanties. We observe that vanilla VAE reconstructions tend to lose image details such as textual rendering, which contains high-freque feehfrrqueney information, as shown in 5 Therefore, we propose high-frequency guidance sampling to balance the quality and fidelity of generation.
The denoising sampling is based the EDM formulation [33
To maintain spatial information, we utilize a modified VAE Decoder Ds to map from the latent space to the pixel space. We modify the VAE decoder by passing skip-connect features from the VAE encoder through additional LoRA convolutions (77 to merge the feature map. The LoRA networks are initialized randomly, with their trainable parameters denoted as 8. Since the parameters of the LoRA convolution are lightweight, merely multi-step backpropagation can maintain high-frequency consistency without requiring extensive fine-tuning.
We propose a fidelity constraint to model the spatial discrepancy between the image and the ground truth. We implement two types of high-pass operators to extract frequecyy signals from the degraded image. For the Fourier filtering operator F(.), we convert the generated image from the spatial the frequency domain using the Discrete Fourier Transform [50]. High-frequency components are then isolated via high-pass filtering and reconstituted into an image through the Inverse Fourier Transform [50]. Meanwhile, we apply the Sobel [53] edge detection operator S(-) as a complement. The fidelity constraint evaluates the divergence between the high- freeueereeeee components of the ground truth and the processed image, ensuring the preservation of spatial information throughout the sampling process. Additionally, to obtain the image at time step ł, we utilize predicted noise e from diffusion model H. to compute the Zt-50 estimation at any time step. The fidelity constraint is calculated as follows:
C(I, Do (0) = [F(
|[((T) S(Do(Z)-0))1
(6)
Given that Re50 represents noisy latent variable, assigning equal weight to each timestep's 's latent impractical. To mitigate the cumulative error induced by this practice, we introduce time-scale scale weight e At. The overall sampling algorithm is detailed in Algorithm

---

Colorization
"Draft this image into the world of calors."
Watermark Removal
"Peel away the watermarked layer of the photo."
Input Image
InstructP2P
InstructDiff
MGIE
PromptFix (Ours)
Ground Truth
Desnow
"Vanish snow from the photo."
Low-light Enhancement
"Image: Increase exposure levels."
Super Resolution
"Picture requires resolution improvement."
Dehazy
"Cleanse the photo of its hazy appearance.
Input Image
PromptIR
AirNet
Diff-Plugin
PromptFix (Ours)
Ground
Figure 3: Qualitative comparison between PromptFix and other instruct-driven diffusion methods (InstructP2P 77. InstructDiff [23]. and MGIE (1 for image processing, as well as low-level
generalist techniques (PrompuR 52 AirNet 38 and Diff-Plugin [46]) for image restoration.
5
Experiments
5.1 Experimental Setup
Implementation details. We train PromptFix for 46 epochs on 32 NVIDIA V100 GPUs, employing
a learning rate x 10-4 with the Adam optimizer. The training input resolution set to 512 512, matching the capabilities of our backbone models, Intern VI.2 [15] and Stable Diffusion 1.5 [57 To facilitate classifier-free frreee guidance [7]. we randomly drop the input image latent, instruction, and auxiliary prompt with probability of 0.075 during training. The hyperparameter À for the time-scale weight in Algorithm T empirically set 0.001, For more implementation details, please refer to the appendix.
Baselines and metrics. We adopt instruction-basee generalist models, such as InstructP2P 7 MGIE [11, and InstructDiffusion (23,, as our primary comparison. MGIE employs VLM -LMggudded techniques for image editing, while InstructDiffusion addresses overlapping tasks with our training objectives, including watermark removal and inpainting. Additionally, we evaluate all-in-one image restoration methods like AirNet (38 and PromptIR [52] (which do not support instruction input), well image restoration expert models, fine-tuned for specific tub-tasks [46] 73). We assess the

---

Table Quantitative comparison is conducted across seven low-level datasets with 512 512 imput resolution. Expert models refer to approaches, such as Diff-plugin [46,, which use non generalizable training pipelines and maintain separate pre-trained weights for each of the four restoration tasks. Image Restoration Generalist Methods denote models that integrate multiple low-level tasks into single framework. Instruct-driven Diffusion Methods represent diffusion generative baselines that follow human language instructions. f indicates higher is better and indicates lower is better. The best and second best results are in bold and underlined, respectively.
LPIPS, / ManlQAt
LPIPS ManIQAt
Object Removal
Watermark
Low-light Enhance.
Method
Colorization
Removal
Desnow
Dehazy
Super Res.
Expert Models
Diff-Plugin
45227/0.43
0.133/0.508
0.033/0.758
0.097/0.555
Inst-Inpaint
0,227/0.593
Image Restoration Generalist Methods
PRompilR
0.330/0.55 0.332/0.541 0,245/0.589
0,235/0.553
0.037/0.764
0.105/0.442
AirNet 38
0.039/0.780
0.107/0.450
Instruction-driven Diffusion Methods
InstrucIP2P
0.394/0.424
79
0.341/0.378
0.581/0.460
0.365/0.560
0.216/0.625
0.234/0.528
InstruciDiff 23
0.433/0.256
0.081
0,247/0.675
0.368/0.300
0,255/0.530
0.124/0.711
0.233/0.623
MGIE 21
0.425/0.393
0.463/0.550
0.491/0.33 356
0.483/0.417
0.249//. 704
0.397/0. 385
PromptFii (ours)
0.233/0.489
0.054/0.88 810
0.127/0.750
0.135/0.423
0.103/0.535
0.088/0.722
0.143/0.642
"<The image shows street cleaner sweeping during snowfall < Potential defects include crtificialllooiing and reduced clarity >
"<The image shows hazy mountain landscape with rainbow.>
<Potential defects include reduced visibilityy and clarity >
Input Image
Our Result
Input Image
Our Result
<<The image shows a dimly lit, empty with table.
*<The image
dimlly lit, empty room with couch <Potential defects Include poor lighting and details.>*
include low lighting obscuring details.>*
Input Image
Our Result
Input Image
Result
Figure 4: Qualitative analysis of VLM-guided blind restoration for desnowing, dehazing, and low- light enhancement. The results obtained from PromptFix without explicit task instructions, relying solely on the input image. The auxiliary prompt, automatically generated by VLM during inference, includes semantic captions and defect descriptions, indicated by <blue> and <yellow> tags, respectively.
similarity of the generated images the ground truth using metrics such PSNR, SSIM 669. and LPIPS 75). For no-reference image quality evaluation, we utilize the ManIQA 772 metric.
5.2 Quantitative and Qualitative Results
Table T Tillustrates the comparative analysis of image restoration and editing techniques, evaluated via LPIPS and ManIQA metrics. The expert model Diff-Plugin shows limited but notable per- formance in low-light enhancement (LPIPS/ManIQA: 0,227/0.453) and desnowing (0. (0.133/0.508) Among generalist methods, AirNet demonstrates balanced capabilities in tasks like desnowing and dehazing, achieving LPIPS/ManIQA scores of 0.245/0.59. and 0.039/0.780. respectively. However, the instruction- driven diffusion methods reveal more nuanced picture, with PromptFix emerging as particularly promising. It excels in colorization (LPIPS/ManIQA. 0.233/0.489) object removal (0.054/0.81) 110), and watermark removal (0.071/0.87) 1), consistently outperforming others. InstructP2P and InstructDiff also perform well in specific tasks, such as low-light enhancement and dehazing, but do match the overall versatility PromptFix. MGIE, though effective certain domains, lacks the consistency in "PromptFix (Ours)." This highlights the robustness and superior performance

---

Table 2: Quantitative comparison of multi-task pro-
Table 3:
Quantitative comparison on three low- level tasks. PromptFix i denotes blind restoration without input instruction prompts. The compared baselines specify task objectives explicitly.
cessing on 200 sampled test images, each paired with a degraded version exhibiting three defects (e.g., desnowing, dehazing and super-resolution)
and the corresponding ground truth.
LPIPS / ManlQA
Method
PSNR1
SSIMI
LPIPS]
ManIQAt
Low Low-light Enhancement
Method
Desnow
Dehazy
PromptRR
52
19.30
0.7385
0.2282
0.4310
AirNel
19.15
0.7197
0.2359
0.4576
PromptI 52
0.331 / 0.539
0.236 0.533
0.038/0 70.664
AirNet [8
0.332 0.332/0.544 0.541
0.245 0.589
0.039 / 0.781
InstructP2P
14.19
0.5845
0.3046
0.3790
InstruciPPP 7
0,581 / 0.461
0.365 0.560
0.216/
InstructDiff 17.12
0.6387
0.2865
0.4365
InstructDit 23
0.369 0.309
0.256
0.531
0.124/0.712
PromptFix 22.05
0.7654
0.1519
0,4889
PrompdFix
0.161 / 0.413
0.115/ /.5311
0.148/00 00.75
of PromptFix across diverse image processing tasks and indicates the potential of PromptFix to set new benchmarks in the field, driven by advanced instruction- diffusion methodologies.
Figure 3 jillustrates the visual comparison across all selected baseline models. In colorization, our PromptFix produces the most visually accurate and vibrant results, closely resembling the ground truth. For the watermark removal task, it effectively restores the original image without introducing artifacts, outperforming MGIE [21] and other methods. In desnowing and low-light enhancement, PromptFix achieves clearer and more natural outputs, significantly reducing noise and enhancing visibility, Additionally, in super-resolution, PromptFix demonstrates remarkable clarity and accuracy, preserving fine details and surpassing all comparative methods. In dehazing, although PromptFix's 'ss performance is visually comparable to image restoration experts PromptIR [52] and AirNet (38), PromptFix outperforms the recent diblee-iifusion-- based method Diff-plugin 46). achieving a clean, sharp appearance, and closely matching the ground truth.
5.3 Ablation Study
Effectiveness of High-frequency Guidance Sampling. The High-freut -reh-freuuenyy Guidance Sampling (HGS) method is introduced to balance fidunityy and quality. To validate the effectiveness of HGS, we conduct qualitative experiments and quantitative experiments. As depicted in Figure a low-light scenario, the model aims to enhance the visibility (quality) of the input image while preserving its original textual details (fdelity). For baselines leveraging stable-diffusion as a generative prior, the strong compression capability of VAE also brings the issue of spatial information loss, demonstrated by InstructDiff (23), MGIE (21, and Diff-Plugin [46) in F This issue is independent of the model's ' ability to effectively follow instructions. As shown by the variant "Ours w/o HGS." our method significantly enhances low-light images compared to the three baselines, yet still fails to retain small-scale textual structures. By incorporating HGS, seen in "Ours," " the proposed framework delivers a high-fidelity solution that also meets the low-light enhancement instruction. The usage of F()) and S(.) improves the quality of the image generated, demonstrated by the quantitative results shown in T
MALS
WEEES
Comply
Camppyy
Comply
Comply
requlat
cal
Fegunnte
Tog mngnint ts
requlate
Br
HO Sost
ied's kan
NO
NO tox
Input
InstructDiff
MGIE
Diff- biff-Plugin
Ours w/o HGS
Ours
Figure 5: Preservation of low-level image details using the proposed High-frequency Guidance
Sampling (HGS) method, compared to previous VAE-based baselines 21 diffusion architecture.
23 46
utilizing stable-
VLM-guided blind restoration. We utilize VL2 [15] to generate auxiliary prompts and leave the instruction prompt empty. This approach enables users to input an image without the need to provide instructions for its restoration. We evaluate the model's performance on such blind restoration tasks, including low-light enhancement, desnowing, and dehazing. As shown in Table 3 our model achieves performance comparable to four baselines, showing minimal perceptual differences from the ground truth and superior shro capabilities.
9

---

Table 4: Quantitative study HGS and Auxiliary Table Ablation Study on Different Types of Prompting. F-) and S(.) represent the type of
Instruction Prompt. A: instructions used during
high-frreueenyy operators used in HGS to guide sampling.
training: lB: out-of-training human instructions
with fewer than 20 words; C: out-of-training hu-
man instructions with 40-70 words.
Auxiliary Prompting
HGS (F())
HGS (S())
LPIPS
ManIQAt
Instruction Prompt Type
Method
LPIPS
ManIQAf
0.2068
0.6487
0,1707
0.6300
0.1795
0.6195
A
0.1600
0.6274
0,1990
0.5856
PromptFix
BB
0.1639
0.6258
0.1600
0.6274
C
0.1823
0.5958
Multi-task processing. Although PromptFix is not explicitly trained to handle multiple low-level tasks simultaneously within the same image, it demonstrates the capability for multi-task process- ing. We construct the validation dataset with 200 images, and each image contains restoration tasks such as colorization, watermark removal, low-light enhancement, desnowing, dehazing, and super-resolution. We benchmarked PromptFix against AirNet and PromptlR, both generalist image restoration methods, as well as InstructP2P and InstructDiff, which are instruction--drven diffusion methods. As shown in Table 2 PromptFix outperforms these baselines, achieving superior image quality, structural similarity, and minimal perceptual differences from the ground truth, as evidenced by competitive PSNR, SSIM, and LPIPS scores, along with higher ManIQA score indicating visu- ally pleasing and high-quality results. Conversely, while methods like InstructP2P and InstructDiff perform well in specific metries, they do not match the overall balanced performance of PromptFix. These results indicate the robustness and versatility of PromptFix.
Different types of instruction prompts. We verify PromptFix's generalization to various human instructions in Table 5 by conducting ablation comparisons with three types of prompts: instructions used during training and out-of-training human instructions with fewer than 20 words and with 40-70 words. PromptFix's performance slightly declines with out-of-training instructions, but the change is negligible. It indicates that PromptFix is robust for instructions under 20 words, which generally sufficient for low-level processing tasks. We observe a performance drop with longer instructions, possibly due the long-tail effect of instruction length in the training data. Although low-level processing tasks usually don't t require long instructions, addressing this issue by augmenting the dataset with longer instructions could be a direction for future work.
Conclusion
We present PromptFix, novel diffusion-basee based model along with large-scale visual-instruction training dataset, designed to benefit iustruction-guide low-level image processing. PromptFix effectively addresses challenges related spatial information loss and degradation adaptation by guidance sampling and a VLM- based auxiliary prompt module. These mechanisms improve the model's performance in the instruction- experiment results demonstrate PromptFix's advanced capabilities of generating accurate and high quality images. In addition to the improvement in terms of conventional metrics, we observe that PromptFix is also effective at processing multi-task processing and achieving blind restoration in low-light enhancement, desnowing, and dehazing.
image-processing paradigm. Extensive
References
[1] Ancuti, C.O., Ancuti, C., Shert, M., Timofte, R.: Dense-haze: A benchmark for image dehazing with dense-haze and haze-free images. In: ICIP (2019)
[2] Ancuti, C.O., Ancuti, C., Timofte, R., De Vleeschouwer, C.: O-haze: a dehazing benchmark with real hazy and haze-free outdoor images. In: CVPRW (2018)
[3] Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitmick, C.L., Parikh, D.: Vqa: Visual question answering. In: ICCV (2015)
[4] Avrahami, O., Lischinski, D., Fried, O.: Blended diffusion for text-driven editing of natural images. In: CVPR (2022)
10

---

[5] Awadalla, A., Gao, I,, Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre. S.. Sagawa. S., al.: Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 (2023)
[6] Bar-Tal, O., Ofri-Amar, D., Fridman, R., Kasten, Y., Dekel, T.: Text2live: Text-driven layered image and video editing. In: ECCV (2022)
[7] Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: CVPR (2023)
[8] Cai, Y., Bian, H., Lin, J., Wang. H., Timofte, R., Zhang, Y.: Retinexformer: One-stage retinex- based transformer for low-light image enhancement. In: ICCV (2023)
[9] Cai, Y., Xiao, Z., Liang, Y.. Qin, M., Zhang, Y., Yang, X.. Liu, Y., Yuille, A:: Hdr-gs: Efficient high dynamic range novel view synthesis 1000x speed via gaussian splatting. In: NeurIPS (2024)
[10] Chen, C., Chen, Q. Do, M.N., Koltun, V. Seeing motion in the dark. In: ICCV (2019)
[11] Chen, C., Chen, Q., Xu, J., Koltun, V:: Learning to in the dark. In: CVPR (2018)
[12] Chen, H., Ren, J., Gu, J., Wu, H., Lu, X.. Cai, H., Zhu, L.: Snow removal in video: A new dataset and a novel method. In: ICCV (2023)
[13] Chen, W.T., Fang, H.Y., Ding, J.J., Tsai, C.C., Kuo, S.Y.: Jstasr: Joint size and transparency- aware snow removal algorithm based modified partial convolution and veiling effect removal. In: ECCV (2020)
[14] Chen, W.T., Fang, H.Y., Hsieh, C.L., Tsai, C.C., Chen, I., Ding, J.J., Kuo, S.Y., et al.: All snow removed: Single image desnowing algorithm using hierarchical dual-tree complex wavelet representation and contradict channel loss. In: ICCV (2021)
[15] Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Muyan, Z., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: CVPR (2024)
[16] Couairon, G., Verbeek, J., Schwenk, H. Cord, M.: Diffedii:: Diffusion- semantic image editing with mask guidance. In: ICLR (2023)
[17] Crowson, K., Biderman, S., Kornis, D., Stander, D., Hallahan, E., Castricato, I.., Raff, E.: Vggan-clip: Open domain image generation and editing with natural language guidance. In: ECCV (2022)
[18] Cun, X., Pun, C.: Split then refine: Stacked attention attention-guided resunets for blind single image visible watermark removal. In: AAAI (2021)
[19] Dhariwal, P., Nichol, A.Q.: Diffusion models beat gans on image synthesis. In: NeurIPS (2021)
[20] Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. In: CVPR (2021)
[21] Fu, T.J., Hu, W. Du, X., Wang. W.Y., Yang, Y.. Gan, Z.: Guiding instruction-buss image editing via multimodal large language models. In: ICLR (2024)
[22] Gan, Y., Park, S., Schubert, A., Philippakis, A., Alaa, A.M.: Instructev: Instruction-tuned text-to-image diffusion models vision generalists. In: ICLR (2024)
[23] Geng, Z., Yang. B., Hang, T., Li, C., Gu, S., Zhang, T., Bao, J., Zhang, Z., Hu, H., Chen, D., et al.: Instructdiffusion: A generalist modeling interface for vision tasks. In: CVPR (2024)
[24] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. Communications of the ACM (2020)
[25] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to- prompt image editing with cross attention control. ICLR (2023)
11

---

[26] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. NeurIPS (2020)
[27] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation large language models. In: ICLR (2022)
[28] Hu, X., Wang, R., Fang, Y., Fu, B., Cheng, P., Yu, G.: Ella: Equip diffusion models with Ilm for enhanced semantic alignment. arXiv preprint arXiv:400300133 (2024)
[29] Hu, Y.. Hua, H., Yang, Z., Shi, W., Smith, N.A., Luo, J.: Promptcap: Prompt-guided image captioning for vqa with gpt-3. In: ICCV (2023)
[30] Hua, H., Shi, J., Kafle, K., Jenni, S., Zhang, D., Collomosse, J., Cohen, S., Luo, J.: Finematch: Aspect-based fine-graines image and text mismatch detection and correction. In: ECCV (2024)
[31] Hua, H., Tang, Y., Xu, C., Luo, J.: V2xum-Ilm: Cross-modal video summarization with temporal prompt instruction tuning. arXiv preprint Stt1:22004.12355 (2024)
[32] Hyvärinen, A., Dayan, P.: Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research (2005)
[33] Karras, T., Aittala, M., Aila, T., Laine, S.: Elucidating the design space of diffusion- generative models. NeurIPS (2022)
[34] Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: CVPR (2019)
[35] Kim, G., Kwon, T., Ye, J.C.: Diffusionclip: Text-guided diffusion models for robust image manipulation. In: CVPR (2022)
[36] Kwon, G., Ye, J.C.: Clipstyler: Image style transfer with single text condition. In: CVPR (2022)
[37] Li, B., Ren, W., Fu, D., Tao, D., Feng, D., Zeng, W., Wang, Z.: Benchmarking single-image dehazing and beyond. TIP (2018)
[38] Li, B., Liu, X., Hu, P., Wu, Z., Lv, J., Peng, X.: All-in-one image restoration for unknown corruption. In: CVPR (2022)
[39] Li, C., Xu, H., Tian, J., Wang, W., Yan, M., Bi, B., Ye, J., Chen, H., Xu, G., Cao. Z., et al.: mplug: Effective and efficient vision-language learning by cross-modal skip-connections. EMNLP (2022)
[40] Li, Y., Zhang, Y., Wang, C., Zhong, Z., Chen, Y., Chu, R., Liu, S., Jia, J.: Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814 (2024)
[41] Lian, L., Li, B., Yala, A., Darrell, T.: LIm-grounded diffiusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. TMLR (2023)
[42] Lin, J., Hua, H., Chen, M., Li, Y., Hsiao, J., Ho, C.,, Luo, J.: Videoxum: Cross-modal visual and textural summarization of videos. TMM (2023)
[43] Liu, H., Li, C., Wu, Q., Lee, YJJ.: Visual instruction tuning. NeurIPS (2023)
[44] Liu, Y., Zhu, Z., Bai, X.: Wdnet: Vatermark-decoopssition network for visible watermark removal. In: WACV (2021)
[45] Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al.: Mmbench: Is your multi-modal model all-around player? arXiv preprint urXiv:2307.0622
(2023)
[46] Liu, Y., Liu, F., Ke, Z., Zhao, N., Lau, R.W.: Diff-plugin: Revitalizing details for diffusion-basce low-level tasks. In: CVPR (2024)
[47] Liu, Y.F. Jaw, D.W., Huang, S.C., Hwang, J.N.: Desnownet: Context-aware deep network for snow removal. TIP (2018)
12

---

[48] Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: ICCV (2015)
[49] Nah, S., Hyun Kim, T., Mu Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: CVPR (2017)
[50] Nussbaumer, H.J., Nussbaumer, HJJ.: The fast Fourier transform. Springer (1982)
[51] OpenAl: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
[52] Potlapalli, V., Zamir, S.W., Khan, S.H., Khan, F.S.: Promptir: Prompting for all-in-one image restoration, In: NeurIPS (2023)
[53] Pratt, W.K., Jr., J.E.A.: Digital image processing, Electronic Imaging (2007)
[54] Radford, A., Kim, J.W. Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell,
A., Mishkin, P., Clark, J., al.: Learning transferable visual models from natural language supervision. In: ICML (2021)
[55] Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., Lee, H.: Generative adversarial text image synthesis. In: ICML (2016)
[56] Rim, J., Lee, H., Won, J., Cho, S.: Real-world blur dataset for learning and benchmarking deblurring algorithms. In: ECCV (2020)
[57] Rombach, R., Blattmann, A., Lorenz, D.,, Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR (2022)
[58] Ruan, L., Chen, B., Li, J., Lam, M.L.: Aifnet: All-in-focus image restoration network using a light lield-based dataset. TCI (2021)
[59] Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T. Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S., Crowson, K., Schmidt, I., Kaczmarezyk, R., Jitsev, J.: LAION-5B: an open large-scale dataset for training next generation image-text models. In: NeurIPS (2022)
[60] Schuhmann, C.. Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al.: Laion-5h: An open large-scale dataset for training next generation image-text models. NeurIPS (2022)
[61] Shen, Z.. Wang, W., Lu, X., Shen, J., Ling, H., Xu, T., Shao, L.: Human-aware motion deblurring. In: ICCV (2019)
[62] Singh, A., Pang. G., Toh, M., Huang, J. Galuba, W., Hassner, T.: Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. In: CVPR (2021)
[63] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: ICML (2015)
[64] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. In: ICLR (2021)
[65] Song, Y., Ermon, S.: Generative modeling by estimating gradients of the data distribution. NeurIPS (2019)
[66] Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. In: ICLR (2021)
[67] Tudosiu, P.D., Yang, Y., Zhang, S., Chen, F., McDonagh, S., Lampouras, G., lacobacci, I., Parisot, S.: Mulan: A multi layer annotated dataset for controllable text-to-image generation. In: CVPR (2024)
[68] Wang, R., Xu, X., Fu, C.W., Lu, J., Yu, B., Jia, J.: Sceing dynamic scene in the dark: A high-quality video dataset with mechatronic alignment. In: ICCV (2021)
[69] Wang, 7.,, Bovik, A.C., Sheikh, H.R., Simoncelli, F.P.: Image quality assessment: from error visibility to structural similarity. TIP (2004)
13

---

[70] Wei, C., Wang. W., Yang, W., Liu, J.: Deep retinex decomposition for low-light enhancement. In: BMVC (2018)
[71] Yang, S., Luo, P., Loy, C.C., Tang, X.: Wider face: face detection benchmark. In: CVPR (2016)
[72] Yang, S., Wu, T., Shi, S., Lao, S., Gong, Y., Cao, M., Wang, J., Yang. Y.: MANIQA: multi-
dimension attention network for no-reference image quality assessment. In: CVPRW (2022)
[73] Yildirim, A.B., Baday, V., Erdem, E., Erdem, A., Dundar, A.: Inst-inpaint: Instructing to remove objects with diffusion models. arXiv preprint 4cX1v:230..03246 (2023)
[74] Zhang, K., Mo, L., Chen, W., Sun, H., Su, Y.: Magicbrush: manually annotated dataset for cnarructionguidddd image editing. In: NeurIPS (2023)
[75] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as perceptual metric. In: CVPR (2018)
[76] Zhang, S., Yang, X., Feng, Y., Qin, C.,, Chen, C.C., Yu, N., Chen, Z., Wang, H., Savarese, S., Ermon, S., et al.: Hive: Harnessing human feedback for instructional visual editing. In: CVPR (2024)
[77] Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: Enhancing vision-language understanding with advanced large language models, In: ICLR (2024)
14

---

A Appendix
A.1 PromptFix Dataset
The PromptFix dataset contains 1,013,320 triplet instances among 7 different tasks containing object removal, image dehazing, colorization, debluring, low light enhancement, snow removal and watermark removal. Each instance contains input image, processed image, instruction, and an auxiliary prompt generated by Intern (T5] which is only available for tasks except object removal. The dataset is available at
P CromptfixDes aDatt
Mulan 67 provides multi- image decomposition annotations, inpainting occluded areas and isolating instances a scene on separate RGBA layers. We combine the background layer and foreground object layers to obtain the ground truth image and the background layer and n + 1 foreground object layers to obtain the input image, where N. Different object layers will be placed onto the background image based their foreground-backgroumm relationships.
Object removal:
Image dehazing: We use combination of synthetic datasets (RESIDE 337, SRRS [3] and real- world datasets (Dense-Haze []] O-Haze (2]) for training, comprising 100 pairs of real-world images
and 102,230 pairs of indoor and outdoor synthetic images from ITS OTS 337. SOTS outdoor 37 and nyuhaze500 37..
Colorization: We utilize a subset of Laion-5b generate grayscale images.
60 and Flickr comprising 317,225 images and
Deblurring: We combination of GoPro
49
HIDE 61] LFDOF (58). RealBlur [56). TextOCR
62).. Wider-Face 771 and a subset of CelebA which collectively contain 70,600 pairs of images.
Low light enhancement: We use combination of LOL (70]. SID [1I. SMID [10], SDSD [68]. and construct a set of low light synthetic data based on Pexels and Flickr by reducing brightness and adding noise, which collectively contains 212,618 pairs of images.
Snow removal: We a combination of SRRS [T3, CSD 14 RVSD 12 and Snow 100K
which collectively contains 99,177 pairs of images.
Watermark removal: We use combination of CLWD and LOGO30K 18 contain 124,805 pairs of images.
which collectively
Low-light Enhancement
Colorization
Dehazy
LAION
Dense Haze
LOL
Flickr
SCSD
SID
Deblurring
Object Removal
Posels
HIDE
MULAN
Ficir
LFDOF
Snow Removal
RealBtur
Watermark Removal
CSD
TexDOR
RVSD
CUWD
Wider
SRRS
LOBCSOK
CelebA
Snow100K
Dataset
curated dalasets
Dalaset
off the shell datasels
Figure 6: Data composition.
15

---

1.0
0
010001
0.8
Scale Weight wit
0.6
0.4
0.2
0.0
200
Time Step
Figure 7: Exponential Decay Weight Functions.
A.2 Limitations
Without relying on user input instructions, our PromptFix achieves blind restoration for low-level enhancement, dehazing, and desnowing. However, observe that this approach occasionally results in out-of--onditionee image control, where the model performs text-to-image generation based on the auxiliary prompt rather than image processing. Although blind restoration is feature of our VLM- based Auxiliary Prompt Module, for known degradations, we recommend providing user-custom instructions specify the restoration.
High-frequency Guidance Sampling significantly helps preserve the original image details, counter- acting the spatial information loss caused by VAE compression. Nevertheless, we find that adopting HGS makes the restored image slightly resemble the degraded image. While PromptFix remains promising relative baselines, as shown in Figure 5] results without HGS appear brighter than those with HGS. As we have consistently claimed, employing HGS involves a trade-off between fidelity and quality. In scenarios where VAE reconstruction is sufficiently faithful and user needs prioritize quality, HGS may be omitted.
A.3 More Implementation Details
Below are the more detailed implementation specifies:
Quemantic: "Describe this image and its style a very detailed manner"
Qaetsat "Introduce the drawback of the image"
Parameters the Sobel Operator: This involves applying two 3 x 3 kernels:
--1 2
1
-
-2
-17
G
and G
0
0
1
These kernels calculate gradients in the horizontal and vertical directions, respectively. By convolving these kernels with the image, we obtain the gradient magnitudes at each pixel. The default parameters include mid-range intensity threshold to discern significant edges.
Time-scale Weight A: This weight is determined based the time step of the diffusion model. When the time step is larger, there is more noise, and the fidelity constraint should be decayed. As depicted in Figure 7 the impact of the high-frequency loss exponentially decays with increasing timestep. Empirically, we have set to 0.001.
The time-scale weight, which is determined by the time step of the diffusion model, decreases as the time step increases, due to the increase in noise. Consequently, the fidelity constraint should be decayed accordingly.
Weight of Fidelity Constraint:
16

---

HGS Algorithm Notion Clarification: Using T to represent the neural function evaluations (NFEs) for sampling, and S to denote series of hyperparameters:
Sehurn controls the level of stochasticity.
nooie adjusts the noise standard deviation.
Sumin and Simax define the range of noise levels for stochasticity.
A.4 Efficiency analysis
We conduct an efficiency analysis for PromptFix and other recent baseline models, including Diff- plugin [46] and InstructDiff [23]. by profiling the floating-poin operations (FLOPs) of their respective diffusion models. As demonstrated in the table below, method exhibits comparable efficiency to these advanced models. This analysis reveals capability of PromptFix to maintain high computational efficiency while delivering superior performance in various image-processing tasks.
TFLOPs
Diff-Plugin [46]
1.1
InstructDiff 23
0.8
PromptFix (Ours)
0.8
A.5
More Results
General PromptFix Results. We present additional image processing results in Figures 15 encom- passing tasks including watermark removal, colorization, low-light enhancement, super-resolution, dehazing, desnowing, inpainting, and object removal via bounding boxes. These results generated by our PromptFix model, utilizing an all-in-one pre-trained weight and guided by user-customized instructions.
Mutli-task processing and blind restoration, In Figure we present the qualitative results of multi- task processing as complement to the quantitative assessments detailed in Table Additionally, provide further visualizations VL.M-gided blind restoration outcomes as in Figure 16
Input Image
Ours
"Remove the haze and snow from the image, and super resolution enhancement for this photograph."
"Fill this picture with colors and remove the watermark, and initiate clarity enhancement for the image." "
E douban
Figure 8: Visual results for Mutli-task processing.
17

---

"Eliminate watermark on the image."
Input Image
ATAR
Ours
"Image: Wipe out watermark."
ears
Input Image
Ours
"Cleanse this photo of watermark."
Inputmmage
Ours
"Scrub logo off the image." "
Input Image
Ours
Figure 9: More results for watermark removal.
18

---

"Apply chromatic touch to this photograph."
Input Image
Ours
"Immerse this image in colorful palette."
Input Image
Ours
"Colorize the image to enhance its visual appeal."
Image
Ours
"Paint the image with color tones."
NO MORE CORRUPTIOS CORPURATE MONEY
Input Image
NO MORE
Ours
CORRUPTIO CORPORATE MONEY Our
Figure 10: More results for image colorization.
19

---

"Execute illumination boost on the picture."
Input Image
Ours
"Activate detail enhancement in dark areas for this photograph."
Input Image
Ours
"Picture: Enhance for better night visibility."
Inputmmaee
Ours
"Enhance overall lighting of the image."
Input Image
Ours
Figure 11: More results for low-light enhancement.
20

---

"Remove the haze from this image."
Input Image
Ours
"Picture: Clarify the obscured details by removing haze." "
Input Image
Ours
"Make the photograph clearer by reducing the haze."
Input Image
Ours
"Reduce the haziness to sharpen picture." "
Input Image
Ours
Figure 12: More results for image dehazing.
21

---

"Remove snow from the image."
Input Image
Ours
"Picture: Delete all the snow." "
Input Image
Ours
"Eradicate snow mark the picture."
Input Image
Ours
"Eliminate snow on the image."
Input Image
Ours
Figure 13: More results for desnowing.
22

---

Input Image
Ours
"Remove the man which is to the right of the tall brick building." "
"Remove the stuffed teddy bear which is on the purple cloth wood couch." "
"Remove the gray off computer monitor which is on the brown desk."
"Remove the metal empty pan which is to the right of the brown large wood round empty bowl." "
Figure 14: More results for image inpainting.
23

---

Input Image
Ours
"Remove the object the red box." "
"Remove the object in the red box."
"Remove the object in the red box."
"Remove the object in the red box."
Figure 15: More results for object removal by bounding box.
24

---

Input Image
Our Result
Input Image
Our Result
Figure 16: Visual results for VLM-guided blind restoration.
25

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The claims in the abstract and introduction align with the detailed findings and discussions presented in the paper, accurately representing scope and contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
26

---

Justification:
Please see appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The paper includes all necessary assumptions for each theoretical result and provides thorough and accurate proofs, ensuring clarity and correctness.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided not)?
Question:
Answer: [Yes]
The paper provides detailed descriptions of the experimental setup, methodolo- gies, and parameters, ensuring that the main results can be reproduced accurately.
Justification:
Guidelines:
27

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification: We are ready to release data and code, along with comprehensive instructions, enabling faithful reproduction of the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public// guides/ Cede///ooeudaiiiicc ss wwm.olsssooooooooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new opunesource benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Swweeeeee//ooeSudedbem
sswonnooloolooooo liycclyy// for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
28

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: paper provides comprehensive details data splits, hyperparameters, selection methods, and the type of optimizer used, ensuring a thorough understanding of the experimental setup and results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not provide information on the statistical significance of the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the eтror bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper details the type of compute workers, providing all necessary information for reproduction.
29

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpss/tttt///.........ooooo//o http:///wuuppssscooooooo/ccccccc
Answer: [Yes]
The research adheres to the NeurIPS Code of Ethics, ensuring ethical standards are met in all aspects, including data use, transparency, and the potential impact of the findings.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper does not focus on potential societal impacts, either positive or negative, of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
30

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
The paper does not mention any specific safeguards for the responsible release of data or models that might be susceptible misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
Justification: They are cited appropriately the paper, with licenses and terms of use clearly mentioned and respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ppapersw thcode. com / data asets has curated licenses for some datasets. Their licensing guide help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide new visual instruction datasets for low-level tasks, and the docu- mentation is comprehensive and available alongside the datasets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
31

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification:
We do not conduct crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjeets
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The study does not involve human subjects, so IRB approval equivalent review is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
32


Reviews:
[{"review_id": "x0FL50gtvH", "reviewer": "Reviewer_gUoB", "summary": "This paper addresses low-level image processing tasks using a unified, Diffusion-based method. The key idea is to construct a dataset comprising pairs of editing instructions and targets for a variety of tasks and fine-tune a pre-trained text-to-image Diffusion Model on this dataset. Further innovations include augmenting editing instructions with text descriptions from a Vision Language Model (VLM) and a training loss that penalizes difference in high-frequency image components to facilitate detail preservation. The paper presents qualitative and quantitative results on a selected set of low-level image processing tasks, where the proposed method outperforms instruction-based Diffusion baselines and sometimes other task-specific models.", "questions": "- Are the experiments on low-light enhancement, dehazing, desnowing, etc. performed on simulated data? The artifacts look unrealistic to me. How does the model perform on real data? I am concerned about generalization on real data since the model is exclusively trained on simulated data.\n\n- The paper claims in the abstract that the method \"achieves comparable inference efficiency\" with the baselines. Please provide concrete numbers to prove this claim.", "limitations": "Limitations are discussed in A.2. The model performance might be sensitive to user-provided instructions. Additionally, passing along high-frequency textures from the input image inevitably copies some artifacts to the output.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The method addresses low-level image processing tasks in a unified framework. Specifically, the model is conditioned on text-based instructions that specify the tasks to solve. This formulation avoids training one model per task and allows knowledge (captured by model weights) to be shared among tasks.\n\n- Low-level image processing tasks often require precise alignment of input and output pixels, yet pre-trained Diffusion models exhibit distortion as a result of lossy VAE encoding. To this end, the proposed method borrows ideas from UNet and introduces skip connections to pass along image textures that might have lost due to encoding. It further introduces a loss term to facilitate the transfer of high-frequency details. This is a sensible design that might be applicable to problem domains with similar requirements on input-output alignment.\n\n- Strong experiment results. As the model naturally benefits from strong Diffusion priors, it performs well on tasks such as dehazing and low-light enhancement where the collection of paired training data is challenging.", "weaknesses": "- Lack of novelty. Instruction-based image editing using Diffusion models dates back to InstructPix2Pix, which similarly utilizes text conditioning to unify arbitrary image processing tasks. Further, I don't think it is fair to sell the dataset as a main contribution of the paper. Instruction generation using GPT-4 is not new (Both InstructionPix2Pix and LLaVA did that). The pipeline for corrupting GT images also follows standard practice. The auxiliary prompts are new but are specific to the the proposed approach. The techniques for enhancing spatial alignment are also new and seem effective, but they alone cannot justify the broad claim of the paper.\n\n- Regarding the two proposed techniques (auxiliary prompts and loss on high-frequency components), there is no ablation study showing they are absolutely needed for the method to succeed. Qualitative and quantitative experiments are needed to show that auxiliary prompts can help in the case where input images are corrupted. Similarly, ablation experiments are needed to show how precisely the F(.) and S(.) terms in Equation 6 affects output quality.\n\n- Some model details are lacking. For example, I am not sure I fully understand how auxiliary prompts are incorporated into the Diffusion UNet. If the cross-attention layers are replicated, are they fine-turned during training? The paper leaves me with the impression that only the LoRA convolutions are learned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive suggestions. Your endorsement of our method and experiments gives us significant encouragement.\n\n> **Q1.1**: Lack of novelty. Instruction-based image editing using Diffusion models dates back to InstructPix2Pix, which similarly utilizes text conditioning to unify arbitrary image processing tasks. \n\nWe recognize that the InstructPix2Pix paradigm can effectively unify various image-processing tasks. However, our exploration of low-level image processing revealed several limitations:\n\n1. **Preservation of Image Details:** InstructPix2Pix primarily focuses on image editing, often neglecting the fidelity of the original image's structure and high-frequency content. For low-level image processing tasks, preserving these details is essential. For example, when colorizing a grayscale image, losing the original high-frequency details due to VAE compression is unacceptable.\n2. **Handling Severe Degradations:** InstructPix2Pix uses only the image and instruction as inputs, resulting in arbitrary and unrealistic outcomes for severely degraded images. An auxiliary prompt, like descriptive text, is required for better guidance.\n\nTo address these issues, we propose two new mechanisms: the HGS and the Auxiliary Prompt Module. The experimental results (`strong experimental results` in your comments) demonstrate the effectiveness of our method. We believe our work provides a valuable contribution to the community.\n\n> **Q1.2**: Further, I don't think it is fair to sell the dataset as a main contribution of the paper. Instruction generation using GPT-4 is not new (Both InstructionPix2Pix and LLaVA did that). The pipeline for corrupting GT images also follows standard practice. \n\nWe clarify our dataset contribution from two perspectives:\n\n1. **The proposed dataset fills a gap**. It includes over 1 million paired images with instructions and auxiliary prompts, covering more than 7 types of low-level image processing tasks. **No such comprehensive dataset previously existed.**\n2. **The proposed dataset is constructed beyond GPT**. We use segmentation and inpainting models to create data for object removal and creation. The bounding box and point annotations are also valuable for subject-driven generation training beyond mere image processing and editing.\n\nWhile the dataset construction workflow may not be highly innovative, its primary focus is on filling a gap and benefiting the community. We firmly believe that the proposed dataset is a valuable contribution to our paper, especially given the substantial resources invested in its creation.\n\n\n> **Q1.3**: The auxiliary prompts are new but are specific to the proposed approach. The techniques for enhancing spatial alignment are also new and seem effective, but they alone cannot justify the broad claim of the paper.\n\nThank you for recognizing that our proposed auxiliary prompt and HGS are new. Our goal is to tackle the limitations of instruction-based image editing frameworks in low-level tasks by introducing two new mechanisms. Empirical results confirm the effectiveness of our proposed methods.\n\n> **Q2**: Regarding the two proposed techniques (auxiliary prompts and loss on high-frequency components), there is no ablation study showing they are absolutely needed for the method to succeed. Qualitative and quantitative experiments are needed to show that auxiliary prompts can help in the case where input images are corrupted. Similarly, ablation experiments are needed to show how precisely the F(.) and S(.) terms in Equation 6 affect output quality.\n\nWe conduct experiments following your advice, please refer to **General Response - II**.\n\n\n> **Q3**: Some model details are lacking. For example, I am not sure I fully understand how auxiliary prompts are incorporated into the Diffusion UNet. If the cross-attention layers are replicated, are they fine-turned during training?\n\nThe cross-attention layers for auxiliary prompting are additional, structured the same way, initialized with the original cross-attention weights, and need joint tuning during training. We will clarify this detail in our revision.\n\n\n> **Q4**: Are the experiments on low-light enhancement, dehazing, desnowing, etc. performed on simulated data? The artifacts look unrealistic to me. How does the model perform on real data? I am concerned about generalization on real data since the model is exclusively trained on simulated data.\n\nOur curated dataset includes realistic data. For the desnowing task, we used 2329 real data from datasets [1-2]; for the dehazing task, we used 5422 real data from datasets [2-5]; for the low-light enhancement task, we used 47139 real data from datasets [6-9], and these low-light image pairs are both taken by cameras with different ISO values.\n\nBesides, we provide real-world degraded image processing examples in the uploaded PDF to demonstrate the generalization ability of our model on real data. Please refer to it.\n\n[1] Desnownet: Context-aware deep network for snow removal. TIP (2018)\n\n[2] Jstasr: Joint size and transparencyaware snow removal algorithm based on modified partial convolution and veiling effect removal. ECCV (2020)\n\n[3] Benchmarking single-image dehazing and beyond. TIP (2018)\n\n[4] Dense-haze: A benchmark for image dehazing with dense-haze and haze-free images. ICIP (2019)\n\n[5] O-haze: a dehazing benchmark with real hazy and haze-free outdoor images. CVPRW (2018)\n\n[6] Deep retinex decomposition for low-light enhancement. BMVC (2018)\n\n[7] Learning to see in the dark. CVPR (2018)\n\n[8] Seeing motion in the dark. ICCV (2019)\n\n[9] Seeing dynamic scene in the dark: A high-quality video dataset with mechatronic alignment. ICCV (2021)\n\n\n> **Q5**: The paper claims in the abstract that the method \"achieves comparable inference efficiency\" with the baselines. Please provide concrete numbers to prove this claim.\n\nWe provide a comparison of different models' FLOPs in Appendix A.4 to support this claim. Please refer to this section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "phlhndX6n7", "reviewer": "Reviewer_uR5Y", "summary": "This paper employs prompts to perform low-level image restoration tasks using pretrained diffusion models. To facilitate this, a substantial paired dataset with image restoration instructions was collected. The proposed method relies on latent diffusion models, incorporating the input low-quality image as an additional input and using a VLM to generate auxiliary prompts, serving as another text condition for the network. During sampling, the paper introduces High-frequency Guidance Sampling, wherein the VAE decoder is optimized to better capture the high-frequency details of the input images.", "questions": "1. Regarding the additional cross-attention layers, do you simply add another cross-attention layer after all the existing cross-attention layers and do the two cross-attention sequentially or parallel? And do you clone the weights as well?\n2. In Alg 1, it seems that the decoder is updated during the testing time for each input? Since the loss is based on the input (low-quality image), will it affect the output image quality?\n3. I'm also curious about how robust the method is for different instructions. For example, if we use different text prompts as instructions, how will the image quality change?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper studies an interesting problem, utilizing instructions to do low-level image restoration tasks.\n2. The qualitative results look promising on various image restoration tasks.", "weaknesses": "1. The dataset is collected by manually performing the degradation which has a distribution gap with the real-world low-quality images. How will the method perform for the real-world image degradation task, for example, motion blur captured by the phone camera?\n2. Regarding the comparison, since the baselines are not trained on the collected dataset, it might be a bit unfair as they aren't aware of the low-level restoration tasks. It would be better to compare the methods under a similar setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time, thorough comments, and nice suggestions. We are pleased to clarify your questions step-by-step.\n\n> **Q1**: The dataset is collected by manually performing the degradation which has a distribution gap with the real-world low-quality images. How will the method perform for the real-world image degradation task, for example, motion blur captured by the phone camera?\n\n1. Our data isn't entirely human-crafted degradation. For instance, in low-light enhancement, we used 47,139 real images, with paired images taken by cameras at different ISO values.\n2. In the updated PDF's bottom-right image, we provide how PromptFix handles natural photos with motion blur. This photo was taken with a camera.\n\n\n\n> **Q2**: Regarding the comparison, since the baselines are not trained on the collected dataset, it might be a bit unfair as they aren't aware of the low-level restoration tasks. It would be better to compare the methods under a similar setting.\n\nGood suggestion. We discuss this in the **General Response - IV**, please refer to it.\n\n\n> **Q3**: Regarding the additional cross-attention layers, do you simply add another cross-attention layer after all the existing cross-attention layers and do the two cross-attention sequentially or parallel? And do you clone the weights as well?\n\nWe add cross-attention layers sequentially after the existing ones. These new layers for auxiliary prompts are initialized with the original cross-attention weights and tuned during training. We will clarify this in the revision.\n\n\n> **Q4**: In Alg 1, it seems that the decoder is updated during the testing time for each input? Since the loss is based on the input (low-quality image), will it affect the output image quality?\n\n- Yes, it will be updated for each input. \n- Our HGS may not always enhance performance. In some cases, HGS makes the restored image slightly resemble the degraded image. We have discussed this limitation in the second paragraph of Appendix A.2.\n\n\n> **Q5**: I'm also curious about how robust the method is for different instructions. For example, if we use different text prompts as instructions, how will the image quality change?\n\nThanks for your advice. We discuss this in the **General Response - III**, please refer to it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BZ7QN2XT5p", "reviewer": "Reviewer_sDoY", "summary": "This paper introduces PromptFix, a unified model designed to intelligently interpret and execute customized human instructions across a variety of low-level image tasks. To address the issue of spatial information loss in stable diffusion, PromptFix introduces a high-frequency guidance sampling strategy. Additionally, to tackle the degradation adaptation problem, PromptFix incorporates an auxiliary prompt module, providing models with more descriptive text prompts to enhance controllability in image generation.", "questions": "The authors claim that the HGS strategy has the potential to introduce spatial information loss. However, the visual results in Figure 5 are insufficient to prove HGS's effectiveness in maintaining image fidelity. Quantitative results should be presented to demonstrate that the proposed HGS module is indispensable to PromptFix.", "limitations": "Authors claimed the limitations in the Appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors construct a comprehensive dataset tailored for low-level image processing tasks. \n2. The proposed PromptFix presents a user interactive image processing method, exhibiting superior zero-shot capabilities in blind restoration and hybrid degradation tasks. \n3. Both the visual and quantitative results demonstrate the effectiveness of PromptFix.\n4. The paper is well-written, with the motivation, method, and experiments clearly explained.", "weaknesses": "1. The authors claim that when user-input instructions are discarded, PromptFix occasionally performs text-to-image generation based on the auxiliary prompt rather than image processing tasks. It is recommended that authors provide examples of these failed cases and evaluate whether the generated images can preserve the layout and structure of the input images.\n2. There is a concern that PromptFix might learn a mistaken shortcut by memorizing the degradation type of the input image. Therefore, authors are encouraged to demonstrate PromptFix's ability to correctly execute tasks, such as coloring a snowy image while preserving the snow.\n3. Authors assert the superior generalization and zero-shot capabilities in blind restoration and combination tasks. Therefore, it is suggested that authors evaluate the model on out-of-distribution and real-world datasets to substantiate these claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time, thorough comments, and valuable suggestions. We are pleased that you acknowledged our clearly explained idea, the well-written paper, and our convincing experiments.\n\n> **Q1**: The authors claim that when user-input instructions are discarded, PromptFix occasionally performs text-to-image generation based on the auxiliary prompt rather than image-processing tasks. It is recommended that authors provide examples of these failed cases and evaluate whether the generated images can preserve the layout and structure of the input images.\n\n\nFor this very occasional case mentioned in the limitations section, the results do not preserve structure at all. They behave like regular text-to-image generation. We will include examples in the revision.\n\n\n> **Q2**: There is a concern that PromptFix might learn a mistaken shortcut by memorizing the degradation type of the input image. Therefore, authors are encouraged to demonstrate PromptFix's ability to correctly execute tasks, such as coloring a snowy image while preserving the snow.\n\nThank you for your advice. In the uploaded PDF, we present the case as per your suggestion to demonstrate that our method performs tasks correctly. We have placed the example in the top right corner, showing a black-and-white snowy road scene being colorized while retaining the snow.\n\n\n> **Q3**: Authors assert superior generalization and zero-shot capabilities in blind restoration and combination tasks. Therefore, it is suggested that authors evaluate the model on out-of-distribution and real-world datasets to substantiate these claims.\n\nGood suggestion. Figure 4 and several figures in the appendix show real-world processing. To further support this claim, we provide more qualitative results on real-world testing in the updated PDF. Please refer to it.\n\n\n> **Q4**: The authors claim that the HGS strategy has the potential to introduce spatial information loss. However, the visual results in Figure 5 are insufficient to prove HGS's effectiveness in maintaining image fidelity. Quantitative results should be presented to demonstrate that the proposed HGS module is indispensable to PromptFix.\n\nA quantitative ablation study is presented in the **General Response - II**. Please check them for details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UoJeXPGkjx", "reviewer": "Reviewer_6nKQ", "summary": "The paper introduces PromptFix, a novel framework that significantly enhances the capability of diffusion models in following human instructions for a wide range of image-processing tasks. The authors propose a comprehensive multi-modal dataset and further design a frequency-based diffusion model trained on this dataset. Experiments show competitive performance on various restoration tasks.", "questions": "1. It is important to consider whether the comparative methods have been retrained or fine-tuned on the same dataset. If not, the fairness of the comparison is questionable.\n2. In line 205 of the paper, \"low-pass operators\" might be incorrect; it should probably be \"high-pass operators\".\n3. The test set was randomly selected, and its domain is similar to that of the training set. It is necessary to supplement the comparison with untrained standard datasets.\n4. The paper lacks ablation experiments for auxiliary prompts. Additionally, the High-frequency Guidance method lacks numerical metrics and only presents visual results.\n5. In Table 1, for the unified restoration task, both PromptIR and AirNet, despite not incorporating instructions, demonstrate good performance, with half of the metrics surpassing those of PromptFix. This seemingly limits the perceived superiority of the proposed method's performance.\n6. In line 228, The MGIE[19] was published in ICLR 2024. It has been incorrectly marked in the references.", "limitations": "As shown in the Weaknesses and Questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The large-scale, instruction-following dataset that covers comprehensive image-processing tasks could be helpful in the field of low-level image processing.\n2. Integrating a high-frequency guidance sampling method and an auxiliary prompting adapter shows reasonable problem-solving capability.", "weaknesses": "1. While instructions are necessary for users, the types of degradation tasks (such as snow removal and low-light enhancement) are clearly defined. In other words, for images with the same type of degradation (such as foggy images), how to choose different instructions to achieve the best results remains to be clarified. Additionally, it should be considered whether users can use instructions other than the task-specific prompts provided by the authors.\n2. The modules used are relatively common. The AuxiliaryPrompt, serving as an information cue derived from the image itself, has been utilized in the literature as referenced in [1,2]. And High-frequency Guidance is also a commonly used method [3,4], albeit in this paper, it is constrained in the form of a loss.\n\nRef:\n\n[1] Lin J, Zhang Z, Wei Y, et al. Improving image restoration through removing degradations in textual representations. CVPR 2024.\n\n[2] Chiu M T, Zhou Y, Zhang L, et al. Brush2Prompt: Contextual Prompt Generator for Object Inpainting. CVPR 2024.\n\n[3] Miao Y, Deng J, Han J. WaveFace: Authentic Face Restoration with Efficient Frequency Recovery.  CVPR 2024.\n\n[4] Zhao C, Cai W, Dong C, et al. Wavelet-based Fourier information interaction with frequency diffusion adjustment for underwater image restoration. CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your invaluable feedback and the opportunity to address your queries regarding our approach.\n\n> **Q1**: While instructions are necessary for users, the types of degradation tasks (such as snow removal and low-light enhancement) are clearly defined. In other words, for images with the same type of degradation (such as foggy images), how to choose different instructions to achieve the best results remains to be clarified. Additionally, it should be considered whether users can use instructions other than the task-specific prompts provided by the authors.\n\nPromptFix is designed to understand and follow user-customized instructions for low-level image processing tasks. We discuss this in detail in **General Response III**. As illustrated in the table in General Response III, the impact of different instructions on numerical performance is minimal when the prompt length is less than 20, demonstrating that PromptFix has low sensitivity to varying instructions.\n\n\n> **Q2**: The modules used are relatively common. The AuxiliaryPrompt, serving as an information cue derived from the image itself, has been utilized in the literature as referenced in [1,2]. High-frequency Guidance is also a commonly used method [3,4], albeit in this paper, it is constrained in the form of a loss.\n\n- Integrating VLM to improve restoration results is a straightforward method. However, our paper goes beyond [1,2] by discovering and analyzing that the VLM-based auxiliary prompt module helps the diffusion model handle multi-degradation processing and blind restoration.\n- Unlike existing high-frequency modules [3,4], our HGS is training-free and adaptable to multiple low-level tasks, not limited to a specific restoration domain.\n\n\n> **Q3**: It is important to consider whether the comparative methods have been retrained or fine-tuned on the same dataset. If not, the fairness of the comparison is questionable.\n\nThank you for your advice. We discuss this in the **General Response - IV**, please refer to it.\n\n\n> **Q4**: In line 205 of the paper, \"low-pass operators\" might be incorrect; it should probably be \"high-pass operators\".\n\nThank you for pointing out. We will revise it.\n\n> **Q5**: The test set was randomly selected, and its domain is similar to that of the training set. It is necessary to supplement the comparison with untrained standard datasets.\n\nGood suggestion. We provide more real-world low-level processing results in the updated PDF. Please refer to it.\n\n> **Q6**: The paper lacks ablation experiments for auxiliary prompts. Additionally, the High-frequency Guidance method lacks numerical metrics and only presents visual results.\n\n\nWe provide the quantitative ablation experiments in **General Response - II**. Please refer to it.\n\n> **Q7**: In Table 1, for the unified restoration task, both PromptIR and AirNet, despite not incorporating instructions, demonstrate good performance, with half of the metrics surpassing those of PromptFix. This seemingly limits the perceived superiority of the proposed method's performance.\n\nIt is challenging for an all-in-one model to outperform all the task-expert models in all metrics. However, our model offers several advantages:\n\n1. **Natural Language Guidance:** PromptFix is guided by user-customized instructions, rather than instruction tags, enabling it to perform tasks like object removal, which PromptIR and AirNet cannot achieve.\n2. **Task Unification:** Our single model unifies various image processing tasks, such as colorization, object removal, and watermark removal.\n3. **Comprehensive Restoration:** PromptFix can perform blind restoration and handle multi-degradation processing.\n\n\n> **Q8**: In line 228, The MGIE[19] was published in ICLR 2024. It has been incorrectly marked in the references.\n\nThanks for pointing out. We will revise it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response to Reviewers and ACs

We thank the reviewers for their detailed and valuable comments. To better support our response, we have uploaded a rebuttal PDF (need to download it) containing the supporting materials. The figures within this PDF are labeled using Roman numerals, such as Figure A.

In this post:

- (1) We summarize positive feedback from the reviews.
- (2) We address four common issues raised in the reviews.

## (1) Positive feedbacks

- **Strong empirical performance of the proposed method** 
    - `[gUoB]`: "*Strong experiment results. ... it performs well on tasks such as dehazing and low-light enhancement*"
    - `[uR5Y]`: "*The qualitative results look promising on various image restoration tasks.*"
    - `[sDoY]`: "*Both the visual and quantitative results demonstrate the effectiveness of PromptFix.*"
    - `[6nKQ]`: "*Experiments show competitive performance on various restoration tasks.*"

- **The proposed dataset**
    - `[6nKQ]`: "*The large-scale, instruction-following dataset that covers comprehensive image-processing tasks could be helpful in the field of low-level image processing.*"
    - `[sDoY]`: "*The authors construct a comprehensive dataset tailored for low-level image processing tasks.*"

- **Presentation**
    - `[sDoY]`: "*The paper is well-written, with the motivation, method, and experiments clearly explained.*"


## (2) Addressing common issues

### **I**: Real-world Testing

Reviewers`[gUoB]`, `[uR5Y]`, `[sDoY]`, and`[6nKQ]` noted that more real-world data testing is expected. We have included results in Figure 4 and multiple cases in the appendix to demonstrate real-world image processing performance. To further illustrate our model's robustness, we have added more qualitative results using real-world low-level images in the uploaded PDF. Please refer to it.

### **II**: Quantitative Study on HGS and Auxiliary Prompting

Reviewers `[gUoB]`, `[sDoY]`, and `[6nKQ]` suggest using numeric metrics to evaluate the importance of our HGS and Auxiliary Prompt Module for the proposed PromptFix. We conducted quantitative experiments, as shown in the table below.

| HGS |   | Auxiliary Prompting | LPIPS↓  | ManIQA↑  |
| ---: | :--- | :---: | --- | --- |
| $\mathcal{F}(\cdot)$ |  $\mathcal{S}(\cdot)$ |  |  |  |
|  |  | $\checkmark$ | 0.2068 | 0.6487 |
|  | $\checkmark$ | $\checkmark$ | 0.1707 | 0.6300 |
| $\checkmark$ |  | $\checkmark$ | 0.1795 | 0.6195 |
| $\checkmark$ | $\checkmark$ |  | 0.1990 | 0.5856 |
| $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.1600 | 0.6274 |

In the table, $\mathcal{F}(\cdot)$ and $\mathcal{S}(\cdot)$ represent the type of high-frequency operators used in HGS to guide sampling. These quantitative results indicate the effectiveness of the proposed HGS and auxiliary prompting and their essential role in PromptFix.
In addition to the results above, Tables 2 and 3 in the paper also demonstrate the superiority of the Auxiliary Prompt Module in blind restoration and multi-task processing.


### **III**: Ablation Study on Different Types of Instruction Prompt

Reviewers `[uR5Y]` and `[6nKQ]` suggest we assess the model's generalization to various human instructions. To verify this, we conduct ablation comparisons with three types of prompts:
- $\mathbb{A}$: instructions used during training;
- $\mathbb{B}$: out-of-training human instructions with fewer than 20 words;
- $\mathbb{C}$: out-of-training human instructions with 40-70 words.

Instructions $\mathbb{B}$ and $\mathbb{C}$ are generated by GPT-4. The experimental results are presented in the following table:

| Instruction Type | LPIPS↓ | ManIQA↑ |
| --- | --- | --- |
| $\mathbb{A}$ | 0.1600 | 0.6274 |
| $\mathbb{B}$ | 0.1639 | 0.6258 |
| $\mathbb{C}$ | 0.1823 | 0.5958 |

The model's performance slightly declines with out-of-training instructions, but the change is negligible. This indicates that our model is robust for instructions under 20 words, which is generally sufficient for low-level processing tasks.
We observe a performance drop with longer instructions, possibly due to the long-tail effect of instruction lengths in the training data. Although low-level processing tasks usually don't require long instructions, addressing this issue by augmenting the dataset with longer instructions could be a direction for future work.


### **IV**: Comparison with the Baseline Finetuned on Our Dataset

Reviewers `[uR5Y]` and `[6nKQ]` note that some baseline models may not trained with our proposed dataset. To ensure the effectiveness of PromptFix and improve the fairness of our comparison, we initialize the pre-trained checkpoint of instructdiff and fine-tune it on our dataset for about 150,000 iterations using a learning rate of 5e-6 on 16 80G GPUs. We refer to this as InstructDiff\*. The table below shows the detailed quantitative results:

| Method | LPIPS↓ | ManIQA↑ |
| --- | --- | --- |
| InstructDiff | 0.2815 | 0.5560 |
| InstructDiff\* | 0.2149 | 0.6086 |
| Ours | 0.1600 | 0.6274 |


> The results of the above three quantitative results consolidate all low-level processing tasks. We will update these experimental analyses and include real-world test visualizations in the revision.

Meta-review: After the rebuttal, the paper received the ratings of 4/5/6/6. Basically mostly all the concerns are addressed by the rebuttal provided by the authors. After examining the reviews and the rebuttal for the reviewer with a rating of 4, the AC believes the concerns should be addressed as well, as these concerns are the common ones shared by other reviewers. Generally, the reviewers recognize the strengths of the paper: a constructed dataset for image editing with prompts, PromptFix provides an interactive approach for image editing, and the experiments are compressive and convincing. Thus, the AC recommends acceptance of the paper. The authors are recommended to incorporate the suggestions from the reviewers for the camera-ready version.
Decision: Accept (poster)