InstructRestore: Region-Customized Image Restoration with Human Instructions
Authors: Shuaizheng Liu
Jianqi Ma
Lingchen Sun
Xiangtao Kong
Lei Zhang
Venue: NeurIPS 2025 poster ()
Keywords: real-world general restoration
real-world super-resolution
bokeh tuning
human instruction
region-customized
local enhancement
consistent local control
contineous control
Primary area: applications

Abstract: Despite the significant progress in diffusion prior-based image restoration for real-world scenarios, most existing methods apply uniform processing to the entire image, lacking the capability to perform region-customized image restoration according to user preferences. In this work, we propose a new framework, namely InstructRestore, to perform region-adjustable image restoration following human instructions. To achieve this, we first develop a data generation engine to produce training triplets, each consisting of a high-quality image, the target region description, and the corresponding region mask. With this engine and careful data screening, we construct a comprehensive dataset comprising 536,945 triplets to support the training and evaluation of this task. We then examine how to integrate the low-quality image features under the ControlNet architecture to adjust the degree of image details enhancement. Consequently, we develop a ControlNet-like model to identify the target region and allocate different integration scales to the target and surrounding regions, enabling region-customized image restoration that aligns with user instructions. Experimental results demonstrate that our proposed InstructRestore approach enables effective human-instructed image restoration, including restoration with controllable bokeh blur effects and region-specific restoration with continuous intensity control. Our work advances the investigation of interactive image restoration and enhancement techniques.  Data, code, and models are publicly available at https://github.com/shuaizhengliu/InstructRestore.git.

InstructRestore: Region-Customized Image Restoration with Human Instructions
Shuaizheng Li Jianqi Ma', Lingchen Sun 1., Xiangtao Kong" :2, Lei Zhang 12,
'The Hong Kong Polytechnic University POPPO Research Institute chunizzengglu221@@maa.. com, (jianqi ma, ling-chen suen.sun, xiangtao. kong) @connect. polyn polyu. cslzhang@comp polyu. polyu.edu.bl edu.
Abstract
Despite the significant progress in diffusion prior-based image restoration for real- world scenarios, most existing methods apply uniform processing to the entire image, lacking the capability to perform region-customized image restoration ac- cording to user preferences. In this work, we propose new framework, namely InstructRestore, to perform region-adjustable image restoration following human instructions. To achieve this, we first develop data generation engine pro- duce training triplets, each consisting of a high-quality image, the target region description, and the corresponding region mask. With this engine and careful data screening, we construct a comprehensive dataset comprising 536,945 triplets to sup- port the training and evaluation of this task. We then examine how to integrate the low-quality image features under the ControlNct architecture to adjust the degree of image details enhancement. Consequently, to identify the target region and allocate different integration scales to the target and surrounding regions, enabling gegton-csttoizzed image restoration that aligns with user instructions. Experimental results demonstrate that our proposed Instruc- iRestore approach enables effective human-instructed image restoration, including restoration with controllable bokeh blur effects and region-specific restoration with continuous intensity control. Our work advances the investigation of interactive im- age restoration and enhancement techniques, Data, code, and models are publicly available at hhtps:: hoap:://iitan http://gtthuu.c.....oooeeee http://hhh...ceeeeeaaaaaeann//// http:/iiwwnnntttttetsttoooo git
develop ControlNet-like model
Introduction
Image restoration (IR) is a fundamental problem in computer vision to recover high-quality images from degraded inputs. Early works have achieved significant progress individual IR tasks based
on specific simulated degradation assumptions, including denoising [59 60) deblurring [88 36), and
super-resolution [9] [4 While demonstrating strong performance within their target domains, these
approaches exhibit inherent limitations when generalizing to real-world scenarios characterized by unknown and composite degradations. This has motivated the emerging paradigm of real-world image restoration, which aims to handle complex degradation in practical imaging scenarios, particularly for challenging cases like real-world super-resolution [5 4 To address this challenge, recent advances have developed sophisticated degradation models to better approximate real-world conditions [57 39). Building upon these advanced degradation models, and with the advent of pretrained text-to-image (T2I) generation models such as Stable Diffusion (SD) [33], which can more effectively model the complex distribution of natural images, researchers have started explore the of powerful SD
priors to produce realistic IR outcomes 38
50 46 45 35
Corresponding author. This work is supported by the PolyU-OPPO Joint Innovative Research Center.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

LO
GT
SeoER
SUPPR
InsmeeRaatone
Instruct
"make deg clear with 0.8 keep ether parts with (scale)'
scale=1
#cole:0.4 4
"moke the tree front of building clear (scale) and keep other parts clear with 1"
Scale=1
ScoleF0.3
Our proposed InstructionRestore framework enables region-customized restoration following human instruction. As shown in (a), current methods 46 52) tend to restore the bokeh blurry region incorrectly, while our approach allows for adjustable control over the degree of blur based on user instructions. In (b), existing methods fail to achieve region-specifie enhancement intensities, while our approach can simultaneously suppress the over-enhancement areas of building and improve the visual quality in areas of leaves.
Figure 1:
By generating details semantically consistent with the underlying content of the image. SD- based
methods achieve significantly better perceptual quality than previous approaches [23 26
6
34
29 40
However, image restoration is an ill-posed problem that admits multiple plausible solutions. Existing methods are limited to a single restoration outcome applied uniformly across the image, lacking the ability accommodate varied user preferences across different image regions. For example, in the photography of targeted objects (e.g.. portrait), the background is intentionally blurred for aesthetic focus. During restoration, users typically want preserve even adjust bokeh effects, yet existing generative prior-based IR methods may produce unnecessary textures that disrupt the intended bokeh effects on the background regions, as shown in Fig. (.). In addition, user preferences for content fidelity and perceptual quality vary across image semantic regions. For irregular texture regions (e.g., trees), it's challenging accurately recover pixel-wise details due signal aliasing in the degradation process (22). In these cases, strictly enforcing fidelity often leads to swoothe results. Therefore, users generally prioritize perceptual quality for irregular texture regions, favoring more aggressive detail generation. Conversely, for structural regions (e.g., architecture) or flat areas (e.g., skies), large pixel-wise differences are more perceptually sensitive and easily detected as artifacts [47]. Therefore, user preference may shift towards content fidelity to preserve accuracy, as illustrated in Fig. [b). Unfortunately, existing methods cannot achieve such customized restoration of different regions.
39 22
To address the limitation mentioned above, we propose InstructRestore, novel framework that enables users to realize region-specific restoration through natural language instruction for real-world scenarios, including bokeh adjustment and region- aware tuning of content fidelity and perceptual quality. Our InstructRestore approach can precisely adjust restoration effects in target semantic regions while keeping other areas unaffected, showing the ability of instruction following. To begin with this novel task, we need dataset for training and evaluation, which should offer descriptions of target regions to construct human instruction, along with corresponding region masks. To the best of our knowledge, there is not a publicly available dataset that provides such triplets of high-quality images, referring descriptions, and the corresponding region masks. The most relevant datasets to task can be the referential segmentation datasets such RefCOCO [53). However, its image quality and resolution are insufficient to support IR tasks. To bridge this gap, we develop a data generation engine. Utilizing Semantic-Sam [8] and Osprey (55) models, we obtain masks and initial deseriptions from a set of selected high-quality images. We then use large language models (LLMs), more specifically Qwen [88). to iteratively parse and refine these descriptions, formatting them to meet the instructional requirements of IR tasks. Finally, we build a dataset of 536,945 triplets, covering diverse scenes such plants, buildings, animals, elc.
Building upon this dataset, we train the InstructRestore model for region-customized IR with user instructions. To ensure that the model can accurately identify the human-specified region and properly

---

enhance the designated area, we propose integrating the conditional features of low- low-quality input images into -ontrolNet-like architecture. Instructions are used as text prompts to the control-branch of ControlNet [61. Trained on our curated dataset, the control-branch could simultaneously generate region masks and conditional features. By applying distinct integration scales to the conditional features of user-customized regions and their surroundings, our InstructRestore model achieves locally controlled restoration that aligns with user intentions.
Our key contributions are summarized as follows. (1) First, we introduce the task of region-customized image restoration with human instruction, which represents an important class of practical IR tasks. (2) Second, we develop a data generation engine and construct a large-scale dataset with 536, 945 triplets to support this task. (3) Finally, we present InstructRestore, the first model that understands user instructions for costomnzsttmizzed restoration for real-world complex degradation. Our experiments demonstrate the capability and effectiveness of our InstructRestore model, showcasing its great potential for interactive and user-instructed image restoration.
2 Related Work
Diffusion-based Restoration in Real World. Recent diffusion models have significantly advanced the task of IR in real world, addressing mixed degradations such as noise, blur, JPEG compression, and resolution reduction. StableSR (38] and DiftBIR (25) treat the low-quality (LQ) input as condition to guide reverse diffusion process. PASD 50) and SecSR [46] introduce the semantic prompts
like short captions or tags to enrich the result with finer semantic details. SUPIR 52 scales up datasets along with long descriptions to boost perceptual quality with SDXL [30] pre-trained model. DreamClear [2] and FluxIR [8] introduce Diffusion Transformer (DiT)-based models designed for enhanced performance in image restoration. To tackle the inefficiency of iterative sampling, one- step diffusion methods [45] 35 [6)] have emerged. ensuring quality with faster inference. Despite their advancements, existing methods perform restoration uniformly, failing to accommodate user preferences for region-specific refinements.
Instruction-guider Editing and Restoration. Natural language instructions enable intuitive human- cnerructionguuddee
AI collaboration by translating high-level intent into pixel-level operations.
image editing methods like InstructPix2Pix 4 and MagicBrush (58) have demonstrated remarkable
capabilities in spatially aware manipulations. Subsequent works like MGIE [0] and SmartEdit [13
further advance instruction comprehension through multimodal LLMs. Others [19 focus region-specifie control, ensuring editing explicitly defined areas by user instructions. However, these breakthroughs remain confined to semantic-level manipulation rather than physically grounded restoration. To address this, recent efforts have incorporated user instructions into restoration frameworks. InstructIR (7) and PromptFix (54 leverage task-specifie instructions enable single model to handle multiple restoration tasks, including denoising. deblur, rain removal, etc. SPIRE [31] incorporates semantic descriptions to handle in-the-wild restoration scenarios. However, these methods primarily use instructions for task differentiation or global parameter tuning, lacking the ability to perform region-specific refinements. Our work introduces the first iestruction-uuicee restoration framework for real-world scenarios that enables region-specific refinements through natural language commands, addressing the critical limitation of global-only operations in prior arts.
3 Dataset Construction
InstructRestore aims to adjust restoration effects on user-specified regions following human instruc- tions. To achieve this goal, the model needs to understand the semantic information of the target regions for performing localized restoration. A critical requirement for training such a model the availability of large-scale dataset, which simultaneously offers high-quality images, descriptions of target regions, and corresponding region masks. In this paper, we develop a data generation engine to build such a comprehensive dataset, named Tri-IR, to facilitate the research of InstructRestore tasks. The data generation process is detailed in Fig.
3.1 Dataset Construction Pipeline
High-quality ground-truth image collection. High-resolution and high-quality ground-truth (GT) images are critical for training IR models. Therefore, we collect high-quality images from LSDIR 20).
3

---

descriptions
"seak_
"Vorigtion"
Reformal desicriptions extract sunyect
maac pane": "Etrrgaar, "SAtROTT
"maaait"r Meslin Male".
"naskin (NeeNrr 7ab
"massa":; Tercor masdding foot
"oaas2""; Neeoor mandiao the fooe
Qwen
"nasks" (Neearraaenlyg N1.. Sor7
"nasko" (Neearrasooly
"masseth % Aoge of
("Arge of goant yeur))
Desoription merging
resd
naact
Thask _ase"" "Ancrppiio"
HQ
"aazit": slacaby.
Semantic-SANN
Mask marging
*aact?": to lagy fooror
Generate inttial
Fiuzl
Final Region Descrigtions
Figure 2: Illustration of the annotation pipeline. For selected high-quality images, Semantic-SAM [8] generates initial masks, followed by Osprey [55] for region-level descriptions. Qwen [48] reformats descriptions into noun phrases and extracts semantic subjects. Identical semantics are merged to produce final masks and region captions.
EntitySeg 32 train set and EBB! 14 bokeh train set with shorter side larger than 512 pixels and MUSIQ
17 score larger than 60.
Annotation pipeline. To obtain high-quality images, we design an automatic annotation pipeline to extract the semantic region masks and their corresponding descriptions with a combination of state- of-the-art models. In the mask extraction phase, we first utilize state-of-the-art segmentation model, egg. Semantic-SAA SAM [8]. generate coarse coarse-grained semantic segmentation masks for the semantic region of the images. For images from EntitySeg [32), we directly reuse their pre-annotated masks. Once the masks are obtained, we pair each image with its mask and feed them into mulli-modal large language model, Osprey [55], to generate region-level descriptions. These descriptions serve as part of instructions to specify the regions to be processed restored. At this stage, though we obtain preliminary masks and descriptions, they are still far from perfect as our training data due to [18] occasionally produces multiple mask pieces for one semantic meaning, leading region ambiguity and harmful for the region customization learning; (2) the descriptions are not always noun phrase format due to the response arbitrariness, making them unsuitable for embedding into instructions.
two key issues below: (1) Semantic-SANe
To address these issues, we first utilize Qwen-7B [3). large language model (LLM), perform
the following tasks through prompt tuning: (1) parsing the subject from the descriptions and (2) reformatting them into noun phrases. Due to the randomness in LLM's outputs. we iteratively perform the refinement process. Specifically, we identify error cases and re-execute the above process by larger LLM, Qwen-72B 3. This cycle is repeated times to ensure high-quality outputs. More details can be found in the appendices. Finally, based on the parsed subject, we merge the masks and their corresponding descriptions for regions with identical semantics.
3.2 Dataset Statistics
As shown in Fig. 2 our Tri-IR dataset, provides triplets of high-quality GT images, region masks, and descriptive captions. To underscore the rel- evance and utility of our dataset, we compare it
Table 1: Statistics of our dataset and related datasets.
Amnotation Amount
Min Reselution
Max Resolution
Datasets
MUSIO
RctCkef
59,523
320x4880
360x
67.06
ReCCOCo 53
196,771
640x637
with the most relevant referential segmentation datasets including RefClef [6), RefCOCO [53], ReICOCO+ [53] and RefCOCOg 27 in Ta-
RefCOCO+
53
196.737
167x160
640x633
69.73
RefODCOg
27
208.660
157x160
640x
69.73
536,945
540x340
4464x2241
71.87
ble which also provide masks and captions for semantic regions. The comparison focuses on the number of annotations, the range of image resolutions, and MUSIQ-based quality scores.
As can seen from Table existing datasets, while widely used for segmentation, exhibit critical limitations for IR tasks. Their images are capped at resolution less than 650 pixels and their MUSIQ scores fall significantly below ours. In contrast, our dataset not only provides 536,945 annotated regions (surpassing other datasets in scale) but also delivers higher-resolution images with superior

---

Instraction:
make oregion captioro cleor with <$ş* keep other ports clear with <E2?
Resipe
Mask Decoder
make "region caption: clear"
M
make the dog sand beach clear"
A
Orea Mineeenee
os + M])
Mi
LQ
Training Flow
2.
Inference Flow
Trainable
stugiom caption
Frosen
Result
the dog on sond beach
Figure 3: Framework InsturctRestore. The framework uses red and green arrows to denote training and inference processes respectively. During testing, user instructions parsed to generate target- region semantic masks, with differentiated coefficient modulation applied to conditional features inside/outside mask regions, enabling instruction-guided region-specific restoration effects.
perceptual quality, meeting the need for IR tasks. Our dataset enables both precise semantic control and photorealistic restoration. To further illustrate the semantic diversity and applicability of our dataset, plot word cloud reflecting the relative frequency of different semantic content in the appendices, which demonstrates that our dataset covers wide range of semantic regions that are commonly targeted in restoration tasks, such as plants, buildings, animals, elc.
InstructRestore Model Design
Our network is designed achieve region-specific restoration effects based user instructions, where each instruction contains both spatial region specifications and restoration strength. The core challenge lies in how accurately localize specified regions and implement continuous controllable local restoration effects with given strength, while keeping the restoration of remaining regions unchanged. straightforward approach might be to construct training data pairs corresponding to different local restoration strengths. However, this approach is too labor-intensive. We develop more elegant solution address the above challenges without requiring different strength pairs. Specifically, existing SD-based IR models often employ a ControlNet architecture with the low- quality (LQ) image conditional signal. In this architecture, the pre-trained SD backbone generates text-guided features while the ControlNet branch extracts LR-derived features. The fusion between these two pathways determines the final restoration output. We observe that scaling the ControlNet features by a coefficient during inference provides flexible control of the data fidelity and semantic enhancement. Intuitively, a smaller coefficient allows the SD backbone to dominate, resulting in richer generated details and enhanced perceptual quality, while larger coefficient makes the output closer to the degraded image, increasing fidelity but reducing generated details. Building on this insight, develop InstructRestore, which employs ControlNet-like architecture during training using only standard restoration data. The network learns to perform restoration while predicting region masks from instructions. During inference, InstructRestore generates region masks and scales ControlNet feature by different coefficients inside and outside the masks based on user instructions achieve region-specific restoration.
4.1 Training Framework
Architecture design. As shown in Fig. 3 our InstructRestore model consists of pre-trained SD backbone, ControlNet adaptor, and a lightweight mask decoder. The SD model is frozen during the entire training stage. The region captions en extracted from the user instructions ey act as text prompts for the SD model, providing semantic guidance generate semantic details. ControlNet duplicates the encoder and middle blocks of the SD UNet trainable copies. It receives features
5

---

extracted from the LQ image and user instructions C₁ as input, then extracts hierarchical conditional features from the input and injects them into the UNet decoder blocks at multiple scales.
To accurately localize the target regions in user instructions, we design a mask decoder to predict a spatial mask M. Since ControlNet is initialized from the pre-trained SD UNet, it has been revealed [1] that the cross-attention features between textual and visual embeddings exhibit strong responses to text-described semantic regions. We then extract cross cooss-tttetionn features {At)== between textual embeddings and visual features at each scale of the ControlNet as input to the Mask decoder. which is designed with pyramidal structure to effectively process multi-scale features. The features of each scale A are first passed through two blocks, each consisting of convolutional layer (Conv), group normalization (GN), and ReLU activation. The processed features are then upsampled and concatenated with the rross-attention features A1+1 at the next scale. The combined features are processed by another Conv-GN -e-ReLU block and passed to the subsequent scale.
Training process. Our constructed dataset consist of triplets [/RO, M, ear), where IHQ denotes the high-quality GT image, . is the binary mask specifying the target region, and eM is the textual cap- tion of the masked area. To generate training samples, we first apply the Real-ESRGAN degradation pipeline to Iи to obtain the LQ input Lo- Subsequently, we construct specific instructions and region caption eR based on CM. tailored different restoration purposes. For region-specifie restora- tion, Cr follows the template "make {cm} J clear". while on is the same as CM- For bokeh-aware restoration, the template becomes "make {cM) } clear and keep other parts bokeh blur", while on follows the template "(caz) } in front of bokeh background".
During training, IHQ is first encoded into the latent space by the pre-trained VAE encoder, yielding 20. The diffusion process progressively corrupts 2g with Gaussian noise over randomly sampled timesteps I, resulting in noisy latent states 2 z0 + vVV Ote, where ~ .N I) and a follow a cosine noise schedule. We utilize = and region caption CR as inputs to pre-trained SD backbone. The ControlNet takes 4L. 2, and ej as input to produce conditional features (Frondd f. 11=1" which are added to the frozen SD UNet decoder with scaling factor = 1 during training. The training flow is highlighted in red in Fig. 3
The mask decoder takes the cross-attention features {Ail/ from ControlNet as input to generate target region masks M, supervised by the GT masks M with Cross-Entropy loss. The InstructRestore network, denoted by 6ø, is conditioned on noisy latent 2s, LQ images I1. instruction cr, and region caption en. The training objective L combines the diffusion loss and mask supervision Lmnsk
L = E,,, [|le co(2z, I, IQ: cr, CR)] XL AM,
where Cmask (M, M) = CrossEntropy M) and À balances the two terms.
4.2
Region-customized Inference
After training, our framework enables users to specify target regions and restoration intensities through structured instructions during inference, as shown in the green flow in Fig. 3 The user instructions follow task-specific templates. For general restoration, we set the template as "make fregion caption) clear with [sı], and make other parts clear with (sz]": for bokeh-aware restoration, the template is "make fregion caption) clear with [sı], and keep other parts bokeh blur with [sz]". The fregion caption) specifies the textual caption of region of interest (e.g., "the dog on the sand beach"), and 81, 82 E R define the enhancement scales towards fidelity for the target and other regions, respectively. Here, larger values of 81 and 82 result higher fidelity (closer to the degraded input), while smaller values allow more semantic enhancement. The instruction parsing process extracts three key components: the region caption for SD backbone text conditioning, the main instruction body for ControlNet text encoding, and the fidelity scales $1, 82 for mask modulation. For region-customizze region-csttomrr restoration, the region caption is directly used as SD text condition, while for bokeh-aware restoration, it is modified to "(region caption) in front of bokeh background" to stimulate the pre- pre-trained SD backbone to generate bokeh blur effect features. Similarly, the main instruction body differs between tasks: "make (region caption) clear" for general restoration and "make fregion caption) clear and keep other parts bokeh blur" " for bokeh-aware restoration.
The trained ControlNet branch processes the degraded image and the parsed instruction to generate a mask M E [0, 1] indicating the target region. It dynamically resized to match the spatial dimensions of each U U-Net upsampling decoder layer, producing masks at multiple scales {M) At each layer

---

1, a modulation map is computed as:
=1 -M + 52 (1)
(2)
where 8 and $2 control the fidelity scales for the target and background regions, respectively. This modulation map determines how much ControlNet features contribute to the final output: higher values preserve more original content, while lower values allow more semantic enhancement. The modulated ControlNet features Frond are fused with the base SD features Fidd via element-wise multiplication:
PT = F/ Mi O Frond
By applying this modulation progressively across all decoder layers, our framework ensures precise alignment with user intent, i.e., enhancing target regions with intensity 81 while maintaining natural fi- delity in non-target areas with intensity $2. Our architecture seamlessly transitions between differently enhanced regions, producing photorealistic restoration results that follow user instructions.
Experiments
5.1 Experiment Settings
Training details. Our method is built on SD2.1 [3]. Training data is generated by the data generation engine described in Section 3 The LQ images are obtained by the Real-ESRGAN [39] degradation pipeline. The LQ images and instructions serve as inputs to the model, while the GT images and region masks provide supervision. Our model is first trained on the general degradation dataset for 120K iterations, guided by the instruction template "make the region caption } clear ". The training continues by combining the bokeh dataset with the general degradation dataset for 14k iterations. During this stage, the sampling probability is set to 25% for the general degradation dataset and 75% for the bokeh dataset, which paired with the instruction template "make the { region caption clear and keep other parts bokeh blur". The training is conducted on two A A100 GPUs with a batch size of 64 and initial learning rate of 5e A is adopted as the optimizer for network training.
Comparison methods. As the first instruction-based region-customized IR approach, InstructRestore mainly benchmarks against: (1) GAN-based Real-ESRGAN 39]; (2) Diffusion-based methods like
StableSR 38), DifBBIR 25). PASD [50], SeeSR [46), SUPIR 52 and OSEDiff 4M
5.2 Results on Localized Enhancement
We first show InstructRestore's results with user instructions. Then demonstrate its precise restoration of specified regions. Finally, we compare it with existing methods.
Test dataset. We curate 100 real-world images from multiple sources, including RealSR (5) DRealSR [43, and the RAIM challenge (11, to construct our Instruct Specifically, we select and crop images with clear semantic region ensure meaningful evaluation. The foreground masks are generated using the pipeline described in Section 3 ensuring accurate and consistent ROI extraction. The user instructions used in the experiment are the format of "make { target region caption } clear with { fidelity level / and keep other parts clear with { fidelity level 2 }."
Evaluation metrics. To comprehensively assess the performance of our method, we adopt both reference- based and no-reference metrics, evaluating both target regions and the entire image. The reference-baset metrics include PSNR, SSIM [42] (on the Y channel in YCbCr space) and LPIPS 62). The no-reference metrics include MANIQA [4), MUSIQ [17] and CLIPIQA (37. For region- specific evaluation, we compute PSNR and SSIM exclusively within human- speeiffed regions by using the provided GT mask. For other metrics, we zero out pixels outside the target region based on GT mask for computation. This ensures the evaluation focusing the target regions while being compatible with standard implementations of these metrics.
Localized enhancement with insr-inst We first showcase our method's ability to perform localized enhancement with user-instructions. By specifying the target region and enhancement strength, our method allows users to explicitly control the balance between data fidelity and generative details. As illustrated in Fig. 4 by applying different fidelity scale instructions to the flower region, we successfully adjust the level of details in the flower region while keeping other regions (e.g., leaves and soil) largely unchanged. To our best knowledge, our method is the first one to allow iser-instructed local enhancement. To quantitatively validate the instruction foolooinion-folloiing capability
7

---

Table 2:
Quantitative evaluation the instruction following capability of InstructRestore. Experi- ments are conducted on the Instruct with instruction of "make { region caption of target area clear with { fidelity scale and keep other parts clear with 1."
Fidelity Scale
Target Area
Remaining Area
PSNR+
SSIM
LPIPS
MUSIQ
MANIQAT
PSNK1
SSIM;
0.5
29.71
0.7522
0.1610
0.6801
67.86
0,6108
31.27
0.8949
0.7
30.37
08818
0.1439
0.6931
68.23
0.6161
31.55
0.9047
1.1
30.73
0.8649
0.1253
0.6639
66.92
0.3934
31,56
0.9108
re
re
re
re
re
Centre
Centre
Centre
Centre
Centre
LQ
Real ESRGAN
SubNeSR
DEiDIR
PASD
re
re
re
re
Centre
Centre
Centre
Centre
Centre
GT
SoUSR
OBEDIT
SUFRR
Cuus
Figure 5: Visual comparison of different methods. We set the instruction as "make the bush in front of sign clear with 0.5 and keep other parts clear with 0.9" to keep the fidelity of sign and prioritize detail enhancement of bushes.
of our method, we conduct experiments by varying the enhancement fidelity scales exclusively within the target region while keeping the fidelity scale of the surrounding areas unchanged. For surrounding regions, we calculate reference- based metrics to assess their stability. The quantitative results are shown in Table 2 We see that when the fidelity scale is small, the non-refcrence metrics for the target region are significantly higher, indicating that the method tends to generate more details. As the fidelity scale increases, the reference- based metrics (e.g., PSNR and SSIM) improve, while the non-reference metrics gradually de- crease. This demonstrates that the method ef- fectively follows the instructions, transitioning from detail-oriented generation faithful reconstruction. Furthermore, the PSNR of the target region varies by 1.02 db, while the surrounding regions vary only by 0.29 db. This stark contrast confirms that the enhancement process is localized to the target region, leaving the surrounding areas largely unaffected. Due to space limitations, ablation studies on the mask decoder and feature modulation mechanism are provided in the appendices. We also conduct in- struction variation experiments in the appendix, testing with instructions that do not follow the standard templates. Although not trained on such variations, model demonstrates reason- able robustness in generating masks. Please re- fer to the appendices for details.
Instruction "moke yellow flower with loaees clear with fiielity
scale 5y) and keep parts 1
more input-
41
$1 -
$1
Figure Localized enhancement following in- struction on real-world test data. The details in flowers are enhanced gradually while the other re- gions keeping almost unchanged.
Comparison with other methods. We then compare InstructRestore with the competing methods. For images with heavier degradations, we prioritize stronger generative prior to synthesize more details; for regions with high-frequency and irregular textures (e.g., flowers, brushes), we favor generative enhancement to achieve realistic appearances; while for regions with regular structures (e.g., sign and buildings), a conservative enhancement level is selected avoid unnatural artifacts. As shown in Figure 5 our method can handle well distinct regions, namely the sign and the bushes,

---

Table 3: Quantitative comparison between our InstructRestore method and other methods
InStruct The best and second best results of each metric are highlighted in red and blue.
Method
PSNRT
SSIMT
CLIPIQAT
Target Arca
MUSIQU
MANIQAT
PSNK
SSIMT
LPPSS
Full Image
CLIPIQAT
MUSIQI
MANIQAT
RealESRGAN
$1.69
0.50665
0.7124
58.63
0.4991
27.69
0.7871
0.3185
0.7250
60.39
0.5030
StableSR
30.36
65.75
0.5915
0.7072
0.3001
0.7072
0.6691
DiMBIR
30.95
0.8804
0.6820
66.80
0.5971
26.64
0.6897
0,3434
0.7456
69.96
0.6609
PASD
31,80
0.5724
61.02
0.5323
0.7893
0.2590
62.92
SeeSR
30.90
0.8788
0.6758
67.73
0.5974
26.75
0.7324
0,2879
0.7246
71.49
0.6691
SUPIR
30.74
62.98
0.5655
26.29
0.6997
0.3235
OSEDia
30.21
0.8657
0.6417
66.75
0.5851
26.07
0.7340
0,2870
0.7342
71.88
0.6635
30.55
0.6887
68.17
0.6137
0.6999
0.3245
within the same scene. In comparison, methods such DiffBIR and SUPIR tend to over-enhance the sign, introducing unnecessary artifacts and distortions, while other methods fail to adequately reconstruct the bush, resulting in a smeared and over-smoothed appearance.
Our method allows adjusting the fidelity scale to meet the specific requirements of each region. For example, for the sign, which requires high fidelity, we set the fidelity scale to 0.9 for faithful restoration, For the bush, we prioritize detail enhancement with a fidelity scale of 0.5 to generate richer textures. To provide an example of quantitative evaluation, for the foreground at 0.8, while that for other regions to 1. The evaluation results are reported in Table 3 Since this setting prioritizes generative enhancement in target regions to achieve richer details, it shows better no-reference metrics but relatively lower scores in reference metrics that favor strict fidelity preservation. It is important to note that our InstructRestore enables users adaptively adjust restoration results based on their preferences. The metrics here only serve as an example to demonstrate that our approach can produce visually pleasing results following user instructions.
simply set the fidelity scale
5.3 Results on Images with Bokeh Effects
In this section, we perform experiments to demonstrate that our method can perform image restoration while preserving bokeh effects and controlling the bokeh intensity.
Test dataset. We construct a test dataset by selecting images from two sources; the EBB! dataset [4 and images with bokeh background carefully curated from Pixabay [T]. We select 70 images from them with distinct semantic foregrounds and bokeh background. Masks are generated for the foreground regions to precisely define the ROI, based on which the images are center-cropped to ensure a consistent resolution of 512 x 512. Subsequently, we apply Real-ESRGAN 39 degradations to generate LQ and GT image pairs for evaluation. To support instructed interaction, we leverage the masks and GT images to generate foreground descriptions using the pipeline illustrated in Section 3 The user instructions are formatted as "make (foreground description) clear with (enhancement fudelity level) and keep other parts bokeh blur with the {bokeh level)."
Evaluation Metrics. We compute reference- based metrics for full image and background re- gions. In addition, we employ D-DFFNet (15), a model for detecting blurred background, to generate the background mask and compute the Intersection-one -eeroonn-vve---eeee Union (loU) with GT of back- ground mask as measure of bokeh preservation performance.
Instruction: "make flower clear with 0.8 and keep other parts bakek blu with (fidelity scole s,y"
0.5
Instruction: "make flower clear with (ficelity scole s.) and keep other ports baken blun
Control of bokeh effect. Our method allows users to specify the desired intensity of bokeh effects and foreground enhancement level via instructions. As illustrated in Fig. 66 (a), our method successfully adjusts the background blur based on user instructions, simulating vary- ing depth-of of effects while maintaining the sharpness and details in the foreground. More importantly, the adjusted blur is not merely uniform increase in blur intensity. It faithfully replicates the circular light spots of realistic bokeh, mimicking the optical effects produced by high- quality digital single-lens reflex (DSLR) cameras. As mentioned in Section 4.2 we stimulate the
10
21
0.6
- 0.4
Figure 6: Control of bokeh effect and foreground enhancement. (a) Restoration with controlled bokch effect while restoring foreground. (b) Restoration with varying foreground enhancement levels while preserving background bokeh.

---

P
P
P
P
P
LQ
TANO
BOKESRIAAN
P
P
P
P
PARGSA
P
Figure 7: Visual Comparison of bokeh preservation results between the compared methods.
generation of authentic bokeh features by incorporating "bokeh blur background" text into the SD backhone. So smaller scale values result in less integration of LR input features, leading to increased blur effects, enabling simulation of different depth-of-field effects. In addition specifying the intensity of bokeh effects, users can further specify the enhancement strength for the foreground, achieving flexible control on the level of details in focal regions. As illustrated in Fig. (b), the fine details in the flower stamens become more pronounced as the instruction changes. Note that such a feature is not supported by existing restoration methods. We also provide quantitative results validating our controllable blur adjustment in the appendices.
Comparison with other methods. Since the foreground semantics in EBB! mainly include objects like cars and road signs requiring high fidelity, we set the foreground enhancement strength to 1.0. For simplicity and fairness, the bokeh fidelity scale is also set to default value of 1, representing the weakest depth-of-fied effect. The quantitative comparison results are presented in Tab. 4
Our method demonstrates significantly better performance fidelity-oriented metrics com- pared to competing methods, reflecting its abil- ity to accurately approximate the GT's bokeh characteristics. In contrast, existing methods fail to preserve bokch effects, leading to devia- tions from the GT. Visual comparison is shown in Fig. More comparisons are provided in the appendices. Competing methods tend to restore the background with sharp details, disrupting the bokeh effect, whereas our method preserves natural background blur while enhancing foreground details, ensuring both fidelity and artistic quality.
Table 4: Quantitative comparison on Bokeh testset
Method
TSNKT
Backgmund
SSIMF
Bolch IoU
PSNE
SSIMF
Image
[PPPST
ReaIESRCAN
30.36
0.7203
0,7060
0.3700
StableSR
30.24
0,8049
0.7405
22.55
0.6305
0.3965
DiABBR
30.46
0.8017
0.6289
22.20
0.5943
0.4415
PASD
31.87
018453
0.8334
24.27
0.3523
SeeSR
30.42
0.8149
0.7580
22.95
0.6652
0,3677
SLPIR
29.92
0.7847
0.7739
0.5745
014375
OSEDiff
29.89
0.8175
0.7990
22.58
0.6707
0.3609
Ours
31.46
0.8462
0.8482
24.69
0.7437
0,394
Conclusion
We presented InstructRestore, the first framework for region-customized image restoration guided by human instructions. To support this task, we designed a scalable data annotation engine and constructed a dedicated dataset comprising 536.945 triplets, each containing a high-quality image. the region mask and region caption. Building on this dataset, we developed an InstructRestore model that parsed human instructions to achieve region-specific restoration. Our framework allowed users to apply distinct enhancement intensities to different regions and adjust background bokeh effects. By enabling fine-grained control via user instructions, our work advanced research in interactive image restoration and enhancement techniques.
Limitations. While InstructRestore offers baseline for region-customized restoration guided by human instructions, it has several limitations. Currently, it lacks support for instance-level object specification, which requires instance-level masks and captions. Moreover, users recommended to follow predefined instruction format, though an off-the-shelf LLM can convert free-form inputs. Furthermore, although our method achieves competitive results, focuses more on localized cus- tomization, while it deserves further exploration of global quality optimization. Reducing the number of inference steps is also worth exploring. Addressing these limitations would boost the applicability and performance of user- instruntte image restoration in real world scenarios.
10

---

References
[1] Pixabay: 5.3 million+ stunning free images to use anywhere. https://ixabay..cmm.
[2] Yuang Ai, Xiaoqiang Zhou, Huaibo Huang, Xiaotian Han, Zhengyu Chen, Quanzeng You, and Hongxia Yang. Dreamclear: High-capacity real-world image restoration with privacy-safe dataset curation. Adv. Neural Inform. Process. Syst. 2024.
[3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan. Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang. Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[4] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In IEEE Conf. Compus. Vis. Pattern Recog., pages 18392-18402, 2023.
[5] Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, and Lei Zhang. Toward real-world single image super-resolution: A new benchmark and new model. In Int. Conf. Comput. Vis., pages 3086-3095, 2019.
[6] Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. Eur. Conf. Comput. Vis., pages 17-33, 2022.
[7] Marcos V Conde, Gregor Geigle, and Radu Timofte. Instructir: High-quality image restoration following human instructions. In Eur. Conf. Comput. Vis., pages 1-21, 2024.
[8] Junyuan Deng, Xinyi Wu, Yongxing Yang. Congehao Zhu, Song Wang, and Zhenyao Wu. Acquire and then adapt: Squeezing out text-to-image model for image restoration. In IEEE Conf. Comput. Vis. Pattern Recog., pages 23195-23206, 2025.
[9] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell., 38(2):295-3077 2015.
[10] Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. Guid- ing instruction-based image editing via multimodal large language models. arXiv preprint arXiv:2309.17102 2023.
[11] Qin Guo and Tianwei Lin. Focus on your instruction: Fine-grained and multi-instruction image editing by attention modulation. In IEEE Conf. Comput. Vis. Pattern Recog., pages 6986-6996. 2024.
[12] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-promp image editing with cross attention control. arXiv preprint urXiv:2208.0166 01666, 2022.
[13] Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong. Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction- based image editing with multimodal large language models. In IEEE Conf. Comput. Vis. Pattern Recog., pages 3362-8371 2024.
[14] Andrey Ignatov, Jagruti Patel, and Radu Timofte. Rendering natural camera bokeh effect with deep learning. In IEEE Conf. Comput. Vis. Pattern Recog. Worksh. pages 418-419, 2020.
[15] Yuxin Jin, Ming Qian, Jincheng Xiong, Nan Xue, and Gui-Song Xia. Depth and dof cues make a better defocus blur detector. In Int. Conf. Multimedia and Expo, pages 882-887, 2023.
[16] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects photographs of natural scenes. Conf. Empir. Methods Nat. Lang. Process., pages 787-798, 2014.
11

---

[17] Junjie Ke, Qifei Wang, Yilin Wang. Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Int. Conf. Comput. Vis., pages
2021.
[18] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, Lei Zhang, and Jianfeng Gao. Segment and recognize anything at any granularity. In Eur. Conf. Comput. Vis. pages 467-484, 2024.
[19] Shanglin Li, Bohan Zeng. Yutang Feng. Sicheng Gao, Xiuhui Liu, Jiaming Liu, Lin Li, Xu Tang. Yao Hu, Jannzuung Liu, et al. Zone: Zero-shot cnrructtoonguideen local editing. In IEEE Conf. 2024.
Comput. Vis. Pattern Recog., pages
[20] Yawei Li, Kai Zhang, Jingyun Liang, Jiezhang Cao, Ce Liu, Rui Gong, Yulun Zhang, Hao Tang. Yun Liu, Denis Demandolx, et al. Lsdir: A large scale dataset for image restoration. In IEEE Conf. Compur. Vis. Pattern Recog., pages 1775-1787, 2023.
[21] Jie Liang, Radu Timofte, Qiaosi Yi, Shuaizheng Liu, Lingchen Sun, Rongyuan Wu, Xindong Zhang, Hui Zeng, Lei Zhang, Yibin Huang, al. Ntire 2024 restore any image model (raim) in the wild challenge. In IEEE Conf. Compul. Vis. Pattern Recog., pages 6632-6640, 2024.
[22] Jie Liang, Hui Zeng, and Lei Zhang. Details or artifacts: A locally discriminative learning approach to realistic image super-resolution. In IEEE Conf. Comput. Vis. Pattern Recog, pages 5657-5666, 2022.
[23] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Int. Conf. Comput. Vis., pages 1833-1844, 2021.
[24] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 336-144, 2017.
[25] Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Bo Dai, Fanghua Yu, Yu Qiao, Wanli Ouyang, and Chao Dong. Diffbir: Toward blind image restoration with generative diffusion prior. In Eur. Conf. Comput. Vis., pages 430-448, 2024.
[26] Zhisheng Lu, Juncheng Li, Hong Liu, Chaoyan Huang, Linlin Zhang, and Tieyong Zeng. Transformer for single image super-resolution. In IEEE Conf. Comput. Vis. Pattern Recog.. pages 457-466, 2022.
[27] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In IEEE Conf. Comput. Vis. Pattern Recog., pages 11-20, 2016.
[28] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In IEEE Conf. Comput. Vis. Pattern Recog., pages 3883-3891, 2017.
[29] JoonKyu Park, Sanghyun Son, and Kyoung Mu Lee. Content-aware local gan for photo-realistic super-resolution. In Int. Conf. Comput. Vis., pages 10585-10594, 2023.
[30] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
[31] Chenyang Qi, Zhengzhong Tu, Keren Ye, Mauricio Delbracio, Peyman Milanfar, Qifeng Chen, and Hossein Talcbi. Spire: Semantic prompt-driven image restoration. Eur. Conf. Comput. Vis., pages 446 464, 2024.
[32] Lu Qi, Jason Kuen, Tiancheng Shen, Jiuxiang Gu, Weidong Guo, Jiaya Jia, Zhe Lin, and Ming-Hsuan Yang. High-quality entity segmentation. In Int. Conf. Comput. Vis., October 2023.
[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEFE Conf. Comput. Vis. Pattern Recog., pages (0684-10695, 2022.
12

---

[34] Lingchen Sun, Jie Liang, Shuaizheng Liu, Hongwei Yong, and Lei Zhang. Perceptionddistotion balanced super-resolution A multi-objective optimization perspective. IEEE Trans. Image Process., 2024.
[35] Lingchen Sun, Rongyuan Wu, Zhiyuan Ma, Shuaizheng Liu, Qiaosi Yi, and Lei Zhang. Pixel- level and semantic-level adjustable super-resolution: A dual-lora approach. arXiv preprint wrXiv22412.03017 2024.
[36] Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang, and Jiaya Jia. Scale-recurrent network for deep image deblurring. In IEEE Conf. Comput. Vis. Pattern Recog., pages 8174-8182, 2018.
[37] Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In AAAI, volume 37, pages 3555-2563, 2023.
[38] Jianyi Wang. Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. Int. J. Comput. Vis., pages 1-21, 2024.
[39] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Int. Conf. Comput. Vis., pages 1905-1914, 2021.
[40] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Eur. Conf. Comput. Vis. Worksh. pages 0-0, 2018
[41] Yufei Wang, Wenhan Yang, Xinyuan Chen, Yaohui Wang, Lanqing Guo, Lap-Pui Chau, Ziwei Liu, Yu Qiao, Alex C Kot, and Bihan Wen. Sinsr: diffusion- based image super-resolie super-resolution in a single step. In IEEE Conf. Comput. Vis. Pattern Recog., pages 25796-25805, 2024.
[42] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 13(4):600-6) 612, 2004.
[43] Pengxu Wei, Ziwei Xie, Hannan Lu, Zongyuan Zhan, Qixiang Ye, Wangmeng Zuo, and Liang Lin. Component divide-and-conquer for real-world image super-resolution. In Eur. Conf. Comput. Vis., pages 101-117, 2020.
[44] Rongyuan Wu, Lingchen Sun, Zhiyuan Ma, and Lei Zhang. One-step effective diffusion network for real-world image super-resolution. arXiv preprint arXiv:2406.0811 2024.
[45] Rongyuan Wu, Lingchen Sun, Zhiyuan Ma, and Lei Zhang. One-step effective diffusion network for real-world image super-resolution. Adv. Neural Inform. Process. Syst., 37:92529-92553, 2025.
[46] Rongyuan Wu, Tao Yang, Lingchen Sun, Zhengqiang Zhang, Shuai Li, and Lei Zhang. Seesr: Towards semantics-aware real-world image super-resolution. In IEEE Conf. Comput. Vis. Pattern Recog., pages 25456-25467, 2024.
[47] Liangiin Xie, Xintao Wang, Xiangyu Chen, Gen Li, Ying Shan, Jiantao Zhou, and Chao Dong. Desra: detect and delete the artifacts of gan-based real-world super-resolution models. arXiv preprint arXiv:2307.022 02550727777..7.... 2023.
[48] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.1511 15115, 2024.
[49] Sidi Yang. Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujiu Yang. Maniqa: Multi-dimension attention network for no-reference image quality assessment. In JEEE Conf. Comput. Vis. Pattern Recog., pages 1191-1200, 2022.
[50] Tao Yang, Rongyuan Wu, Peiran Ren, Xuansong Xie, and Lei Zhang. Pixel-aware stable diffusion for realistic image super-resolution and personalized stylization. Fur Conf. Comput. Vis., pages 74-91, 2024.
13

---

[51] Qiaosi Yi, Shuai Li, Rongyuan Wu, Lingchen Sun, Yuhui Wu, and Lei Zhang. Fine- structure preserved weal-world image super-resolution via transfer vae training. arxiv preprint arXiv:2507.20291, 2025.
[52] Fanghua Yu, Jinjin Gu, Zheyuan Li, Jinfan Hu, Xiangtao Kong, Xintao Wang, Jingwen He, Yu Qiao, and Chao Dong. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. In IEEE Conf. Comput. Vis. Pattern Recog,, pages 25669-25680 2024.
[53] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Eur. Conf. Comput. Vis., pages 69-85, 2016.
[54] Yongsheng Yu, Ziyun Zeng, Hang Hua, Jianlong Fu, and Jiebo Luo. Promptlix: You prompt and we fix the photo. arXiv preprint arXiv:2405. 2024.
[55] Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang. Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In IEEE Conf. Comput. Vis. Pattern Recog., pages 28202-2821 11, 2024.
[56] Zongsheng Yue, Kang Liao, and Chen Change Loy. Arbitrary-steps image super-resolution via diffusion inversion. In JEEE Conf. Comput. Vis. Pattern Recog., pages 23153-23163, 2025.
[57] Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte. Designing practical degradation model for deep blind image super-resolution. In Int. Conf. Comput. Vis., pages 4791-4800 2021.
[58] Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su, Magicbrush: A manually annotated dataset for instruction-guiin guided image editing. Adv. Neural Inform. Process. Syst., 36, 2024.
[59] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Trans. Image Process., 26(7):314233155, 2017.
[60] Kai Zhang, Wangmeng Zuo, and Lei Zhang. Ffdnet: Toward a fast and flexible solution for enn-based image denoising. IEEE Trans. Image Process. 27(9):6688 2018.
[61] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Int. Conf. Comput. Vis., pages 3836-3847, October 2023.
[62] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In IEEE Conf. Comput. Vis. Pattern Recog,, pages 586-595, 2018
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to content from line 73 to line 79
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to content from line 366
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
the paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Have described the detail of data and method
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: The data is too big to be zipped as single file. Open access for data and code will be offered after review. We offer details in main paper and appendices.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/ppp cc/public/guu getggiddddeeeoeebbnnnnnn SttSSSabmiiiioooiiiioiooooii for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have described the detail
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We detail the type of compute resources in main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:: http://pww....ee...ooo/o///////a
Answer: [Yes]
Justification: Yes, the research in the paper fully conforms to the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we analyze the potential social impact in appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, we clearly indicate the baseline methods and data used in the paper. Their licenses permit use with academic scope.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cww/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We introduce the detail of data and model, and will release all if accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowd sourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowd sourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Please refer to section 3 in main paper and corresponding content in appendix.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https://neure) //tpaarrppp.p////oeeeeer//////// for what should or should not be described.
21


Reviews:
[{"review_id": "AKTwLMvYgL", "reviewer": "Reviewer_qjkL", "summary": "This paper integrates image super-resolution with the bokeh task, enabling precise restoration of target regions and differentiated background processing based on user instructions. By integrating models like Semantic-SAM, Osprey, and Qwen, it builds an image–mask–text triplet. Using ControlNet, it parses user instructions to generate spatial masks and applies different enhancement scales, achieving fine-grained, end-to-end control over region and intensity.", "questions": "1. Different Bokeh scales should be fully matched with the instructions. Apart from Figure 6(a), more experimental results should be provided to demonstrate this point.\n\n2.  What is the purpose of Figure 6(b)? As the parameter S1 decreases (0.6h and 0.4), more details inconsistent with the real images are generated, suggesting that more imagined components are introduced. This seems to reflect an adjustment of the consistency and detail generation capability of the diffusion model-based image super-resolution, rather than enhancing the clarity of the foreground objects. \n\n3. The experimental comparison in Table 4 is unfair. Even though the two parameters of the model have been set, other comparison methods have not been trained on the Bokeh training set, making it an OOD task for them.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This work combines image super-resolution with bokeh, showing practical application value in scenarios requiring both clear foregrounds and blurred backgrounds.\n2. The constructed triplet dataset is valuable for region-related image editing tasks.\n3. The paper is well-written and easy to read.\n\nWeaknesses：\n1. The novelty is limited and largely based on combinatory innovation. The integration of natural language instructions with region editing in the image editing domain (e.g., IIR-Net [1]) has already been widely explored. Additionally, there are existing works based on natural language instructions in the image super-resolution field (e.g., SPIRE [2], InstructIR [3], PromptSR [4] ). Furthermore, the use of region masks in this paper has also been studied in the image editing field (e.g., DIFFEDIT [5]). Therefore, the paper does not seem to introduce a new contribution in terms of methodology.\n\n2. The title \"Instruction-Customized Image Restoration\" is inappropriate. The paper only addresses one task in image restoration (super-resolution) and one task in image editing (Bokeh). Firstly, Bokeh is not an image restoration task but rather belongs to image beautification or editing. Secondly, common image restoration tasks such as dehazing, denoising, low-light enhancement, and deblurring are not explored in this work. Therefore, the task setting in this paper is not appropriate.\n\n3. There is a lack of comparison with recent works, such as image super-resolution methods like InvSR [6], PiSASR [7], and others. Additionally, there is no comparison with the performance of Bokeh methods like drbokeh [8], bokehme [9], etc.\n\n4. In the image super-resolution task, the authors compare their method on the custom-built Instruct100Set rather than widely-used image super-resolution benchmark datasets, which diminishes the persuasiveness of the results. Additionally, the comparison methods lack the latest image super-resolution techniques mentioned earlier. Furthermore, Figures 10 and 11 in the supplementary materials show that the visual quality of the image super-resolution results does not surpass OSEdiff, and the proposed method lacks detail, appearing blurry and smeared.\n\n[1] Zhang, Zhongping, et al. \"Text-to-image editing by image information removal.\" Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2024.\n\n[2] Qi, Chenyang, et al. \"Spire: Semantic prompt-driven image restoration.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[3] Conde, Marcos V., Gregor Geigle, and Radu Timofte. \"Instructir: High-quality image restoration following human instructions.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[4] Chen, Zheng, et al. \"Image super-resolution with text prompt diffusion.\" arXiv preprint arXiv:2311.14282 (2023).\n\n[5] Couairon, Guillaume, et al. \"Diffedit: Diffusion-based semantic image editing with mask guidance.\" arXiv preprint arXiv:2210.11427 (2022).\n\n[6] Yue, Zongsheng, Kang Liao, and Chen Change Loy. \"Arbitrary-steps image super-resolution via diffusion inversion.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[7] Sun, Lingchen, et al. \"Pixel-level and semantic-level adjustable super-resolution: A dual-lora approach.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[8] Sheng, Yichen, et al. \"Dr. bokeh: differentiable occlusion-aware bokeh rendering.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[9] Peng, Juewen, et al. \"Bokehme: When neural rendering meets classical rendering.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.", "was_revised": true, "final_justification": "This paper is well-written and easy to read. While I raised some questions, such as limited novelty and unfair comparison, during the initial review stage, the authors have effectively addressed most of these concerns in their rebuttal and the subsequent discussion phase. Therefore, I decided to raise my score to a borderline acceptance.", "rebuttal": "We sincerely  thank the reviewer for the constructive comments and hope our point-to-point responses address the concerns raised.\n\n**[W1]: Methodology Novelty.**\n\nWe appreciate this reviewer for the comments and provided literature. However, we believe there are misunderstandings here. Our work is fundamentally different from both image editing and prior instruction-based restoration. Our method tackles the unexplored challenge of region-customized restoration, with unique task formulation and technical implementation. \n\n*1. Difference from Image Editing*\n\nImage editing modifies clean inputs to alter semantics (e.g., scene composition or object attributes), while InstructRestore processes degraded inputs while preserving semantics. The ill-posed nature of restoration implies multiple valid solutions for a degraded input, where users may prefer varying perceptual details across regions. Our goal is to recover content while enabling continuous, intent-driven regional control.\n\nThe image editing methods mentioned by this reviewer (e.g., DiffEdit, IIRNet) operate on clean-domain inputs to transform an image by using target text descriptions. Typical approaches first encode the clean image into the noise for diffusion models, then perform denoising using target text prompts. For example, DiffEdit employs DDIM encoding to obtain this noise, then substitutes masked-area noise with text-aligned versions during decoding to alter content. However, these methods cannot be used for restoration. First, the noise-conversion pipeline fails on degraded inputs, hurting output fidelity. Second, unlike editing, our target output shares identical semantic descriptions with the input.\n\nOther editing methods require supervised training with edited image pairs, such as IIRNet mentioned by this reviewer. IIRNet conditions on clean inputs but deliberately corrupts specified regions (via color removal/noise injection) to avoid identity mapping, forcing text-consistent generation only in masked areas. Such editing approaches fundamentally conflict with the restoration task. In contrast to the deliberate corruption strategy, we prioritize both strict semantic consistency and maximal information preservation from degraded inputs. \n\nIn summary, image editing fundamentally differs from our task. It requires clean inputs and produces semantically altered outputs by design. Even advanced regional editing techniques are incompatible with restoration objectives. Achieving human-aligned perceptual control for region-specific image restoration is the goal of our work.\n\n*2. Difference from Previous Instruction-Based Restoration*\n\nExisting instruction-based restoration methods (InstructIR, PromptSR, and SPIRE) can only achieve global uniform restoration. In contrast, our work enables region-customized restoration. We need to solve previously unaddressed challenges: (1) precisely localizing user-specified regions in degraded images through instruction parsing, (2) enabling continuous effect modulation within target areas, and (3) guaranteeing artifact-free transitions between modified and unmodified zones. All of these requirements are beyond the scope of existing global instruction-based approaches.\n\n*3. Methodological Innovations*\n\nTo address the challenges mentioned above, we first created a large-scale dataset specifically designed for this task, and developed a mask decoder that accurately localizes target regions using both user instructions and the degraded image. We found that a ControlNet-based adaptation enables precise restoration fidelity control by modulating feature integration in Stable Diffusion. Based on this, we selectively adjust feature fusion in target regions, preserving original fidelity elsewhere. Our results show seamless blending without visible edge artifacts.. \n\nIn summary, our method differs from prior work in three key ways: instruction usage, mask generation, and mask operation. Our novelty lies in the building of the first unified framework for controllable region-level image restoration, which cannot be achieved by existing methods.\n\n**[W2]: Title and Task.**\n\nWe thank the reviewer for the feedback regarding the title and task scope of our paper. We respectfully clarify the following points:\n\n*1. On the Use of \"Image Restoration\" in the Title*\n\nOur work follows the paradigm of recently published papers (e.g., SUPIR[1], SPIRE[2], DreamClear[3], FluxIR[4]) that address real-world image restoration under mixed degradations in the wild. Kindly note that all those works use \"image restoration\" in their titles, and they report experimental results only on real-world image super-resolution, which is one of the two tasks reported in our paper. \n\nLike these works, our degradation model combines noise, blur, JPEG compression, and downsampling to synthesize real-world distortions. We also evaluate on real-world datasets with complex, entangled and unknown degradations. The title \"Region-Customized Image Restoration\" aligns with this broader definition of the restoration task adopted by the existing works.\n\n[1] Yu F, et al. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. CVPR 2024\n\n[2] Qi C, et al. Spire: Semantic prompt-driven image restoration. ECCV 2024\n\n[3] Ai Y, et al. DreamClear: High-Capacity Real-World Image Restoration with Privacy-Safe Dataset Curation. NeuralPS 2024\n\n[4] Deng J, et al. Acquire and then Adapt: Squeezing out Text-to-Image Model for Image Restoration. CVPR 2025\n\n*2. On the Requirement of Bokeh Preservation in Our Task*\n\nThis reviewer may misunderstand our experiments on bokeh-preserved image restoration. Please kindly note that \"bokeh preservation\" is a requirement in our **region-customized restoration** task. Existing SD-based methods often fail to distinguish bokeh from degraded images, leading to over-restoration of blurred area.  Our approach allows users to specify background blur strength and foreground detail recovery through instructions. It is not an image editing or beautification problem, but a user-instructed image restoration problem.\n\n*3. Concerns on Broader Tasks*\n\nRegarding more restoration tasks such as dehazing, denoising, low-light enhancement, we believe that our data generation pipeline and instruction-based restoration framework can be extended to these tasks, which will have their own unique requirements for localized adjustment. In this work, we focus on the task of real-world image restoration [1-4]. We hope that our work can inspire more researchers to work in this underexplored but valuable direction.\n\n**[W3]: Comparison Baselines.**\n\n*1. Regarding real-world super-resolution methods (InvSR/PiSASR):* \n\nWe compare with InvSR/PiSASR in table below. Note that the metrics are computed in the target area. Sorry that we cannot show visual comparisons due to NeurIPS' policy.Actually, like other Real-ISR methods in the main paper, these methods cannot achieve region-level restoration guided by human instructions.The metrics in the table cannot reflect the real performance. We will add the two methods in revision.\n\n| Method | PSNR↑| SSIM↑ | CLIPIQA↑ | MUSIQ↑| MANIQA↑ |\n|-|-|-|-|-|-|\n|PisaSR|30.53|0.8783|0.6648|67.90|0.6028|\n|InvSR|29.89|0.8321|0.6491|67.58|0.6051|\n|Ours|30.55|0.8368|0.6887|68.17|0.6137|\n\n*2. Regarding bokeh rendering methods (drbokeh/bokehme):* \n\nPlease kindly note that bokeh rendering methods (e.g., DRBokeh, BokehMe) are architecturally incompatible with our task. They synthetically generate bokeh from clean images with depth maps, while we restore natural bokeh from degraded inputs without depth data. So these approaches could not serve as baselines for our task.\n\n**[W4]: Test-sets and Comparison.**\n\nKindly note that no existing benchmark suits our novel task. While current SD-based SR methods evaluate on 512×512 crops from RealSR/DRealSR, these lack sufficient semantic diversity for region-specific instructions. Therefore, we created Instruct100Set by strategically selecting semantically rich regions from RealSR, DRealSR and RAIM containing multiple distinct elements, enabling comprehensive evaluation of localized restoration under real-world degradations. Kindly note that Figures 10 and 11 is to show that only our method preserves background blur as intended, unlike others that over-sharpen these regions.\n\n**[Q1]: More Experiments for Bokeh Tuning.**\n\nFollowing your suggestion, we fixed the foreground fidelity at 0.8 while varying background bokeh fidelity (0.4,0.7,1), where lower values indicate stronger blur. We measured foreground PSNR(fg_psnr), background PSNR(bg_psnr), and background blur strength using the Brenner metric (lower values means stronger blur).\n\nAs shown in the below table, there are limited variations in foreground PSNR (0.5 dB) but significant changes in background PSNR (2.52 dB). Moreover, reducing the bokeh fidelity scale lowers Brenner values, confirming increased blur as intended.\n\n|Scale|fg_psnr|bg_psnr|Brenner|\n|-|-|-|-|\n|0.4|29.42|28.80|137.23|\n|0.7|29.87|30.42|150.27|\n|1.0|29.90|31.32|156.44|\n\n**[Q2]: The Purpose of Figure 6(b).**\n\nThis figure shows our method's dual capability: controlling background blur restoration while regulating foreground detail generation.\n\n**[Q3]: OOD Training Concern.**\n\nExisting datasets (LSDIR, Unsplash) contain bokeh images, so all methods have seen blurred backgrounds during training. However, severe degradation makes natural bokeh hard to distinguish. Guided by user instructions, our model preserves blur while restoring targets. We also retrained OSEDiff with our bokeh-enriched data. As shown in the table below, it achieves slight background metric improvements over original OSEDiff, but lags behind our method. The background bokeh distortions remain (images omitted due to NeurIPS' policy).\n\n|Method|PSNR↑|SSIM↑|BokehIoU↑|\n|-|-|-|-|\n|OSEDiff (ori data)|29.89|0.8175|0.7990|\n|OSEDiff (bokeh-enriched data)|29.99|0.8208|0.8005|\n|Ours|31.46|0.8462|0.8482|"}, {"review_id": "QuQc3w6YKA", "reviewer": "Reviewer_tpZd", "summary": "The author introduces InstructRestore, a novel framework for region-specific image restoration based on natural language instructions. Besides, the authors construct a large-scale dataset,  containing high-quality images, region masks, and corresponding captions. Compared with existing diffusion-based restoration methods that apply uniform processing across the entire image, InstructRestore enables fine-grained control over different image regions. Experiments show promising results on both general and bokeh-aware restoration tasks.", "questions": "- How robust is the model to variations in the instruction format? Some examples or quantitative evaluation would strengthen the claims of instruction-following capability.\n- Have the authors tested the model on instructions involving unseen region types or categories not present in the training set?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### Strengths\n\n- The paper addresses a novel and practical problem—region-customized image restoration—through the lens of instruction-following, which is underexplored in existing research.\n- The authors contribute a large-scale, high-quality dataset and may benefit related research in instruction-guided vision.\n- The proposed framework enables fine-grained, user-controllable restoration, including adjustable fidelity scales for different regions, which is both technically interesting and practically useful.\n- The expression is clear.\n\n### Weaknesses\n\n- The instruction format currently relies on structured templates. While effective, it is unclear how the method would respond to natural or slightly varied user expressions.\n- The paper lacks ablation studies to isolate the effects of key components such as the mask decoder and the fidelity modulation mechanism.\n- The generalization ability of the model to unseen semantic categories or open-domain instructions is not fully explored.", "was_revised": true, "final_justification": "I appreciate the authors’ thoughtful reply, which has successfully resolved my concerns. After carefully considering the responses and reading the other reviewers’ comments, I have decided to raise my rating from Borderline Accept to Accept, as I believe the paper makes a solid contribution and is suitable for publication.", "rebuttal": "We sincerely thank this reviewer for the constructive comments and suggestion. We hope our following point-to-point responses can address this reviewer's concerns.\n\n**[W1] and [Q1]: Variation of Instruction Format**.\n\nWe thank the reviewer for raising the concern about the reliance on structured instruction templates. To answer this question, we designed three instruction variants that were not seen during training to evaluate the robustness of our method to more natural or varied user expressions. Specifically, we tested the following variants:\n\n*v1: “make the entire image sharper while make {region expression} clear”;*\n\n*v2: “enhance the clarity of {region expression}”;*\n\n*v3: “I would like {region expression} to be clear”*.\n\nFor each variant, we set the foreground fidelity scale to either 0.5 or 1.1, the background to 1, and conducted experiments on the Instruct100 set. The test result is shown in below table.  Notably, v1, which is most similar to the original template, achieves nearly identical results to the original instruction. For v2 and v3, which differ more significantly in phrasing, our method still achieves comparable results, suggesting a certain degree of robustness to instruction variations. All the three instruction variants result in significant changes in the specified region and minimal changes elsewhere, with metrics similar to those under the standard instruction. This suggests that, although the phrasing differs, these instructions can still localize the intended region to some extent. However, we acknowledge that the generated masks are not identical across different instructions, which may cause some result variations.\n\n|Fidelity Scale|Target Area| | | | | |Other Area| |\n|-|-|-|-|-|-|-|-|-|\n| |PSNR↑|SSIM↑|LPIPS↓|CLIPIQA↑|MUSIQ↑|MANIQA↑|PSNR↑|SSIM↑|\n|0.5 (our)|29.71|0.7522|0.1610|0.6801|67.86|0.6108|31.27|0.8949|\n|1.1 (our)|30.73|0.8649|0.1253|0.6659|66.92|0.5934|31.56|0.9108|\n|0.5 (v1)|29.82|0.7636|0.1575|0.6772|67.89|0.6084|31.24|0.8929|\n|1.1 (v1)|30.71|0.8641|0.1239|0.6702|67.19|0.5949|31.55|0.9124|\n|0.5 (v2)|29.92|0.7813|0.1483|0.6580|66.73|0.5914|31.52|0.9050|\n|1.1 (v2)|30.90|0.8814|0.1136|0.6229|64.24|0.5551|31.81|0.9238|\n|0.5 (v3)|29.90|0.7751|0.1506|0.6622|67.09|0.5965|31.40|0.8994|\n|1.1 (v3)|30.84|0.8781|0.1154|0.6361|65.16|0.5654|31.73|0.9210|\n\n**[W2]: Ablation Studies**.\n\nWe appreciate the reviewer’s concern about  ablation studies. In our original submission, we did not include experiments that simply remove either the mask decoder or the fidelity modulation mechanism. This is because both modules are essential for achieving region-customized restoration based on user instructions—removing either one would make it impossible to realize the core functionality of localized control.\n\nTo address this reviewer’s suggestion and to better analyze the independent contributions of these modules, we have conducted the following additional experiments:\n\n*1. Mask Decoder*\n\nWe assessed the mask decoder by replacing its predicted masks with ground truth (GT) masks during inference and comparing restoration results on the Instruct100 set under two fidelity settings (0.5 and 1.1). The table below shows that results with predicted masks are very close to those with GT masks across all metrics in both target and remaining areas. This minimal performance gap demonstrates that our mask decoder generates high-quality masks, enabling accurate region-customized restoration. Furthermore, both predicted and GT masks show that the target region changes much with different fidelity scales, while the remaining areas stay stable, confirming precise local control. Another observation is that at a lower fidelity scale (0.5), where more details are generated in the target region, the difference between predicted and GT masks is slightly larger. This suggests that mask quality is more critical when generating finer details. Nevertheless, the performance gap remains small, further confirming the robustness of our mask decoder.\n\n|Fidelity Scale|Target Area| | | | | |Other Area| |\n|-|-|-|-|-|-|-|-|-|\n| |PSNR↑|SSIM↑|LPIPS↓|CLIPIQA↑|MUSIQ↑|MANIQA↑|PSNR↑|SSIM↑|\n|0.5 (predict_mask)|29.71|0.7522|0.1610|0.6801|67.86|0.6108|31.27|0.8949|\n|1.1 (predict_mask)|30.73|0.8649|0.1253|0.6659|66.92|0.5934|31.56|0.9108|\n|0.5 (gt_mask)|29.49|0.7297|0.1663|0.6810|67.76|0.6118|31.32|0.9017|\n|1.1 (gt_mask)|30.74|0.8661|0.1247|0.6663|66.88|0.5930|31.56|0.9107|\n\n*2. Fidelity Modulation Mechanism*\n\nThe fidelity modulation mechanism in our method controls the amount of generated details by adjusting the coefficient for integrating conditional features from ControlNet into the SD backbone. Intuitively, a smaller fidelity coefficient allows the SD backbone to dominate, resulting in richer generated details, while a larger coefficient makes the output closer to the degraded image, increasing fidelity but reducing generated details. \n\nTo analyze the effect of the fidelity modulation mechanism, experiments were conducted within our multi-step inference framework (20 steps in total), with the fidelity scale set to 0.6 for the target region and 1 for the remaining area.  We tested the impact of applying the fidelity modulation starting from different inference steps. In our default setting, we apply the fidelity modulation from the very first step (step 0). The results are summarized in the table below. We observe that the later the modulation is applied (i.e., the larger the starting step t), the higher the fidelity metrics and the lower the no-reference quality metrics. This indicates that delaying the application of fidelity modulation will suppress the model’s ability to generate details, and narrow the adjustable range of effects when different fidelity scales are set according to user instructions. Therefore, to ensure sufficient controllability, we apply fidelity modulation from the very beginning of the inference process.\n\n| t steps | PSNR↑  | SSIM↑  | LPIPS↓ | CLIPIQA↑ | MUSIQ↑ | MANIQA↑ |\n|-------|--------|--------|--------|----------|--------|---------|\n| 15   | 30.52  | 0.8454 | 0.1333 | 0.6821   | 67.65  | 0.6055  |\n| 10   | 30.41  | 0.8325 | 0.1371 | 0.6824   | 67.86  | 0.6083  |\n|  5   | 30.28  | 0.8137 | 0.1431 | 0.6866   | 67.95  | 0.6116  |\n|  0 (Ours)| 30.09  | 0.7922 | 0.1511 | 0.6911   | 68.12  | 0.6152  |\n\n**[W3] and [Q2]: Generalization for Unseen Semantic Categories**.\n\nWe sincerely appreciate this reviewer's insightful question regarding the model’s generalization to unseen semantic categories and open-domain instructions. Actually, our approach leverages Semantic-SAM to generate open-vocabulary masks without pre-defining a closed-set category list, combined with Osprey for region-specific captioning. This pipeline ensures that the training data encompasses diverse semantic regions beyond conventional category boundaries. For example, as shown in Figure 3 of the Appendix, the specified region is “pagoda”, which is not a predefined category in closed-set semantic segmentation. Nevertheless, our method is still able to localize and adjust this region effectively.\n\nWhile this design enhances open-domain capability, we fully acknowledge that certain rare or novel categories may still pose challenges due to their limited presence in the training set. For instance, when given uncommon objects such as “astrolabe” or “scaffold” as the specified region, the mask decoder may generate an all-zero mask, making it impossible to localize or adjust the intended area."}, {"review_id": "Q9UAY6V8oN", "reviewer": "Reviewer_DVDK", "summary": "This paper proposes a method to perform region-adjustable image restoration following human instructions. Specifically, it constructs a dataset to support the training and evaluation of this task. Then it develops a ControlNet-like model to enable region-customized image restoration that aligns with user instructions. Extensive experiments are conducted to evaluate the effect of the proposed method.", "questions": "1. The authors claim in L44-45 that users generally prioritize perceptual quality for irregular texture regions (e.g., trees), favoring more aggressive detail generation. The statement is not reasonable. For image restoration, the whole image should preserve contetn fidelity and accuracy. However, with generation models, it is harder for irregular texture regions compared to structural elements.\n2. Please discuss in detail the effect of the scaling factor \\alpha.\n3. Please discuss the efficiency of the proposed method and compare with the state-of-the-art methods.\n4. It would be better to discuss in detail the effectiveness of the ControlNet features and the restoration mask.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: First, this work advances the investigation of interactive image restoration and enhancement techniques. Second, it constructs a large dataset to support the research of these techniques. Third, it proposes a model to understand user instructions for region-customized restoration.\n\nWeaknesses: The effect of each component in the proposed method is not very clearly demonstrated.", "was_revised": true, "final_justification": "Thanks for the response, which addresses my concerns. I keep my rating and hope the authors could revise the paper according to the rebuttal.", "rebuttal": "We sincerely thank this reviewer for the constructive comments and suggestion. We hope our following point-to-point responses can address this reviewer's concerns.\n\n**[Q1]: The Statement on Irregular Textures.**\n\nThanks for the comments. We agree that, traditionally, image restoration is defined with a primary focus on content fidelity and accuracy. As mentioned by this reviewer, however, it is indeed challenging to accurately recover pixel-wise details in highly irregular texture regions, including state-of-the-art generative restoration models. This point is well supported by prior literature LDL [1], which indicates that “it is difficult to produce high-fidelity results ... because they have much fine-scale details and suffer from signal aliasing in the degradation process, where most high-frequency components are lost.\" In these cases, strictly enforcing fidelity often leads to **over-smoothed or visually unconvincing results**, which can negatively impact the user’s perceptual experience.\n\nIt is also pointed out in LDL [1] that for such regions: \"the pixels are randomly distributed so that the differences between results and ground truth are insensitive to human perception. Therefore, rich details generated can lead to better perceptual quality in these regions.\"  This confirms that for the Real-ISR task, in certain contexts, perceptual quality is more important to users than strict fidelity. DeSRA [2] also indicates that \"fine details in areas with complicated textures are difficult to perceive as artifacts like foliage, hair, and etc, while large pixel-wise differences in areas with smooth or regular textures, such as sea, sky, and buildings, are sensitive to human perception and easy to be seen as artifacts\". These findings collectively suggest that the balance between perceptual quality and fidelity should be adaptively considered according to the semantic characteristics of different regions, as user preferences and perceptual sensitivity can vary significantly across image content.\n\nBased on the above observations, our instruction-based region-customized restoration approach is designed to flexibly balance fidelity and perceptual quality: we maintain strict fidelity in structured regions, while allowing more perceptual detail generation in irregular texture areas. We will add these discussions in the revision.\n\n[1] Liang J, Zeng H, Zhang L. Details or artifacts: A locally discriminative learning approach to realistic image super-resolution. CVPR 2022\n\n[2] Xie L, Wang X, Chen X, et al. Desra: detect and delete the artifacts of gan-based real-world super-resolution models. ICML 2023.\n\n**[Q2]: The Effect of Scaling Factor**.\n\nThanks for the suggestion. The scaling factor plays a central role in our system by controlling the fusion ratio between the SD backbone and the ControlNet branch during inference. By adjusting the scaling factor, the system can flexibly balance between generating new details and preserving fidelity to the degraded input, according to user instructions and regional requirements. \n\nSpecifically, SD backbone receives region-specific captions and generates semantically aligned features (text-guided features), while ControlNet branch extracts degradation-aware features from the low-quality input  (LR-derived features).  The scaling factor governs the fusion ratio between these two pathways. \n\n*For detail enhancement*: Reducing the weighting of LR-derived features allows the text-guided features in the SD backbone (e.g., “the furry dog”) to dominate, enabling the synthesis of plausible new details. Please refer to Figure 4 in the main paper for visual effects.\n\n*For bokeh tuning*: When the SD backbone is guided by blur-related prompts, a lower LR feature contribution allows those blur-oriented features to prevail, while a higher LR contribution enforces structure recovery, which may lead to over-restoration of background blur. The visual effects can be seen in Figure 6 of the main paper.\n\nIn short, a smaller scaling factor allows the SD backbone to dominate, resulting in richer generated details, while a larger scaling factor makes the output closer to the degraded image, increasing fidelity but reducing generated details.\n\nAs for the restoration result, increasing the amount of generated details (i.e., lowering the scaling factor) typically results in lower reference-based metrics (such as PSNR), but higher no-reference perceptual metrics (such as CLIPIQA). This trend can be observed in Table 2 of the main paper: when the fidelity scale (scaling factor) is reduced in the specified region, PSNR and other reference-based metrics decrease, while CLIPIQA and no-reference metrics increase. This demonstrates that the scaling factor provides effective control over the trade-off between fidelity and perceptual quality, allowing users to flexibly adjust restoration outcomes according to their preferences and the semantic context.\n\n**[Q3]: Efficiency**.\n\nThanks for the suggestion. The table below summarizes the inference steps, inference time (in seconds per image), and parameter count (in billions) for several representative methods. Inference time is measured on 512 × 512 LQ images using a single NVIDIA A100 80G GPU.\n\n|                                 | StableSR | DiffBIR | PASD | OSEDiff | SeeSR | Ours |   |   |   |\n|---------------------------------|----------|---------|------|---------|-------|------|---|---|---|\n| Inference steps                 | 200      | 50      | 20   | 1       | 50    | 20   |   |   |   |\n| Inference time（s）/Image numbers | 10.03    | 2.72    | 2.80 | 0.12    | 4.30  | 4.06 |   |   |   |\n| #Param(B)                       | 1.56     | 1.68    | 2.31 | 1.77    | 2.51  | 1.25 | \n\nAlthough our method includes a mask decoder, it is relatively lightweight, resulting in a smaller overall parameter count compared to methods like OSEDiff and SeeSR, which require an additional RAM model. However, during inference, our approach generates a mask at each step and performs feature modulation at multiple scales within the SD-UNet. Since our current implementation executes these operations using a for-loop without code optimization, the inference time is a little longer than PASD, which uses the same number of diffusion steps. In our future work, we will (1) optimize our implementation to speed up the inference and (2) explore one-step or few-step inference to further enhance the efficiency of our framework.\n\n**[Q4]: The Effectiveness of ControlNet Features and Restoration Mask**.\n\nWe thank the reviewer for prompting a discussion on the effectiveness of the ControlNet features and the restoration mask.\n\n*1. ControlNet features*\n\nAs discussed in [Q2], the ControlNet branch extracts degradation-aware features from the low-quality input, providing essential reference information for the restoration process. These features, when fused with the SD backbone via the scaling factor, enable the system to balance between preserving fidelity to the degraded image and generating new details as guided by user instructions.\n\n*2. Restoration mask*\n\nWe validate the effectiveness of predicted restoration masks by replacing it with ground-truth (GT) masks during inference and comparing the results on the Instruct100 set under two fidelity settings (0.5 and 1.1). \n\nThe table below shows that results with predicted masks are very close to those with GT masks across all metrics in both target and background areas. This demonstrates that the predicted mask is effective to enable accurate region-customized restoration. Furthermore, with either predicted or GT masks, the target region changes with different fidelity scales, while the remaining areas stay stable, confirming precise local control. With a lower fidelity scale (0.5), the difference between predicted and GT masks is slightly larger. This is because, in our multi-step inference framework, a mask is generated at each step, and the predicted masks in the early steps may not accurately cover the target region. As a result, the features of the intended area may still be modulated with a fidelity scale of 1 in the initial steps, leading to higher fidelity metrics compared to using GT masks. \n\n|Fidelity Scale|Target Area| | | | | |Other Area| |\n|-|-|-|-|-|-|-|-|-|\n| |PSNR↑|SSIM↑|LPIPS↓|CLIPIQA↑|MUSIQ↑|MANIQA↑|PSNR↑|SSIM↑|\n|0.5 (predict_mask)|29.71|0.7522|0.1610|0.6801|67.86|0.6108|31.27|0.8949|\n|1.1 (predict_mask)|30.73|0.8649|0.1253|0.6659|66.92|0.5934|31.56|0.9108|\n|0.5 (gt_mask)|29.49|0.7297|0.1663|0.6810|67.76|0.6118|31.32|0.9017|\n|1.1 (gt_mask)|30.74|0.8661|0.1247|0.6663|66.88|0.5930|31.56|0.9107|"}, {"review_id": "dGBIDWNQV4", "reviewer": "Reviewer_1V2k", "summary": "This paper investigates a novel problem for real-world general degradation. While existing approaches employ a uniform processing strategy across the entire image, they fail to accommodate users' divergent preferences for different image regions.  The study systematically examines two representative scenarios demanding localized customization, fidelity-perceptual tradeoffs for different semantic regions, and background bokeh control. \n\nTo address the aforementioned challenges, this paper proposes a novel interactive restoration framework that accepts natural language instructions indicating the target region and corresponding enhancement scale. It first addresses data scarcity through a generation engine that curates high-quality images and employs Semantic-Sam, Osprey, and Qwen models to produce training triplets (high-resolution images, semantic region masks, and corresponding captions). It then presents an ingenious architecture that achieves region-specific enhancement. While trained using uniform feature fusion strength, during inference, it first parses user-specified regional requirements to generate corresponding masks, then strategically assigns distinct fusion weights to condition features based on masks, achieving spatially variant restoration effects. Experiments verify the system's precise instruction-following capability. Results show it can make targeted adjustments in specified areas while consistently preserving quality in non-target regions.", "questions": "In addition to the previous questions, two further clarifications would be helpful:\n\n(1) Are the modulation coefficients bounded (e.g., with predefined min/max limits)? If so, how do these bounds affect the output (e.g., under extreme values)?\n\n(2) The current method focuses on regional control—could the framework inherently support global image-level modulation, or would this require architectural modifications?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n+ Pioneers the practical yet under-explored task of instruction-guided region-aware restoration, shifting the paradigm from globally uniform processing to spatially adaptive enhancement. This opens new research directions in user-centric image restoration.\n\n+ The approach of decoupling feature fusion strengths in ControlNet during inference to achieve tunable restoration effects is both intellectually interesting and methodologically novel. The experimental results demonstrate impressively effective outcomes.\n\n+ The data engine is technically sound, and the constructed high-quality triplet dataset (530K+ samples) not only supports the current task but also holds significant potential for future low-level vision tasks requiring region-text alignment or extended grounding capabilities.\n\n+ This paper also has a significant practical impact, as it demonstrates compelling use cases that address unmet needs in professional photo editing, i.e. the regional perceptual-fidelity preference and adjustable bokeh preservation. The natural language interface enables intuitive control through verbal instructions, significantly improving workflow efficiency.\n\n\n**Weakness:**\n\n- This paper demonstrates impressive instruction-guided modulation results, but the underlying mechanism of how the current tuning coefficients achieve such effects could be further analyzed. Notably, in the bokeh effect adjustment, the text prompts fed into the SD backbone differ across instructions. Is this prompt variation inherently linked to the framework's working principle? A deeper discussion would strengthen the theoretical contribution.\n- The current framework extracts region captions directly from instructions for the SD backbone input, which prioritizes simplicity. However, prior works like SPIRE show that using more detailed scene descriptions in the backbone's text prompt can enhance global restoration. Would replacing the current region captions with such enriched descriptions further improve performance? A brief ablation study or discussion on this trade-off between simplicity and potential gains would be valuable.", "was_revised": true, "final_justification": "Most reviewers think this paper is good and give an accept or weak accept. I also believe this paper is novel, and the instruction-based image restoration is interesting, with great potential value in real-world applications. So I give accept.", "rebuttal": "We sincerely thank this reviewer for the constructive comments and suggestion. We hope our following point-to-point responses can address this reviewer's concerns.\n\n**[W1]: Clarity of Working Principle.**\n\nWe sincerely appreciate this reviewer’s insightful question regarding the interplay between prompt variation and tuning coefficients, which indeed touches the core of our framework’s functionality. As the reviewer astutely observed, the system achieves instruction-guided modulation through a dynamic feature interplay between two key pathways:\n\n*1. SD backbone (text-driven prior).* It receives region-specific captions (e.g., \"blurry background\") and generates semantically aligned features (e.g., bokeh patterns when instructed).\n\n*2. ControlNet branch (LR-derived reference).* It extracts degradation-aware features from the low-quality input.\n\nThe tuning coefficients govern their fusion ratio. Specifically:\n\n*For detail enhancement:* Reducing LR feature weighting allows SD's text-guided features (e.g., \"the furry dog\") to dominate, synthesizing plausible details.\n\n*For bokeh preservation:* When the SD backbone processes blur-descriptive prompts, lower LR feature contribution enables these blur-oriented features to prevail, while higher LR contribution forces structure recovery.\n\n**[W2]: The Effect of Region Captions**.\n\nWe sincerely appreciate this reviewer’s valuable suggestion regarding text prompt enrichment. Following this insightful recommendation, we conducted additional experiments using LLaVA to generate comprehensive scene descriptions for Instruct100Set, replacing our original region-focused captions. As shown in the table, using long captions results in slightly lower reference-based metrics (e.g., PSNR, SSIM) on the full image compared to our standard approach, while the overall performance remains comparable. This is expected, as longer captions with more global descriptions may activate some additional background details, leading to minor changes in the global metrics.\n\n| Method  | Target Area |         |        |       |       | Full Image |         |        |         |        |        |\n|---------|-------------|---------|--------|-------|-------|------------|---------|--------|---------|--------|--------|\n|         | PSNR↑       | SSIM↑   | CLIPIQA↑ | MUSIQ↑ | MANIQA↑ | PSNR↑      | SSIM↑   | LPIPS↓ | CLIPIQA↑ | MUSIQ↑ | MANIQA↑ |\n| Long caption   | 30.52       | 0.8352  | 0.6851 | 67.97 | 0.6129 | 25.52      | 0.6944  | 0.3330 | 0.7276   | 71.96  | 0.6849 |\n| Ours    | 30.55       | 0.8368  | 0.6887 | 68.17 | 0.6137 | 25.65      | 0.6999  | 0.3245 | 0.7278   | 71.95  | 0.6809 |\n\n**[Q1]: The Bound of Modulation Coefficients.**\n\nAs analyzed in our responses to [W1], the modulation coefficients govern the feature fusion ratio between the text-guided and LR-derived pathways. The coefficients should maintain a minimum value greater than zero, and our experiments recommended an optimal maximum threshold around 1.5. Exceeding this upper limit tends to introduce artifacts, such as abnormal color saturation effects in the output. \n\nIn practice, we do not recommend setting the minimum coefficient below 0.3, as excessively small values cause the output to be dominated by SD backbone features, resulting in images that lack both fine details and structural similarity to the degraded input. It is also worth noting that the visually acceptable range of modulation coefficients can vary across different images: for some images, values up to 1.3 still yield normal results, while for others, artifacts may already appear at this level. Based on our experiments across the entire dataset, 1.5 can be considered a general upper bound for most cases. Therefore, keeping the coefficient within the range from 0.3 to 1.5 ensures a good balance between detail generation and structural fidelity.\n\n**[Q2]: Global Modulation**.\n\nThe framework inherently supports global image-level modulation without requiring architectural modifications or retraining. By uniformly applying identical modulation coefficients across both masked and unmasked regions, the system can seamlessly transition from regional to global control."}]

Author rebuttal:

Meta-review: The paper presents a practical framework for instruction-guided, region-customized image restoration. Despite initial concerns from one reviewer regarding novelty and baselines, the authors provided thorough and convincing rebuttals, clarifying the distinctions from prior work and addressing evaluation questions. The final decision is to accept the paper, as all reviewers have reached a consensus and recommend acceptance.
Decision: Accept (poster)