Title: InstructRestore: Region-Customized Image Restoration with Human Instructions
Venue: NeurIPS 2025
Accepted: yes
Keywords: real-world general restoration

real-world super-resolution

bokeh tuning

human instruction

region-customized

local enhancement

consistent local control

contineous control
Abstract: Despite the significant progress in diffusion prior-based image restoration for real-world scenarios, most existing methods apply uniform processing to the entire image, lacking the capability to perform region-customized image restoration according to user preferences. In this work, we propose a new framework, namely InstructRestore, to perform region-adjustable image restoration following human instructions. To achieve this, we first develop a data generation engine to produce training triplets, each consisting of a high-quality image, the target region description, and the corresponding region mask. With this engine and careful data screening, we construct a comprehensive dataset comprising 536,945 triplets to support the training and evaluation of this task. We then examine how to integrate the low-quality image features under the ControlNet architecture to adjust the degree of image details enhancement. Consequently, we develop a ControlNet-like model to identify the target region and allocate different integration scales to the target and surrounding regions, enabling region-customized image restoration that aligns with user instructions. Experimental results demonstrate that our proposed InstructRestore approach enables effective human-instructed image restoration, including restoration with controllable bokeh blur effects and region-specific restoration with continuous intensity control. Our work advances the investigation of interactive image restoration and enhancement techniques.  Data, code, and models are publicly available at https://github.com/shuaizhengliu/InstructRestore.git.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a novel problem for real-world general degradation. While existing approaches employ a uniform processing strategy across the entire image, they fail to accommodate users' divergent preferences for different image regions.  The study systematically examines two representative scenarios demanding localized customization, fidelity-perceptual tradeoffs for different semantic regions, and background bokeh control. \n\nTo address the aforementioned challenges, this paper proposes a novel interactive restoration framework that accepts natural language instructions indicating the target region and corresponding enhancement scale. It first addresses data scarcity through a generation engine that curates high-quality images and employs Semantic-Sam, Osprey, and Qwen models to produce training triplets (high-resolution images, semantic region masks, and corresponding captions). It then presents an ingenious architecture that achieves region-specific enhancement. While trained using uniform feature fusion strength, during inference, it first parses user-specified regional requirements to generate corresponding masks, then strategically assigns distinct fusion weights to condition features based on masks, achieving spatially variant restoration effects. Experiments verify the system's precise instruction-following capability. Results show it can make targeted adjustments in specified areas while consistently preserving quality in non-target regions.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n+ Pioneers the practical yet under-explored task of instruction-guided region-aware restoration, shifting the paradigm from globally uniform processing to spatially adaptive enhancement. This opens new research directions in user-centric image restoration.\n\n+ The approach of decoupling feature fusion strengths in ControlNet during inference to achieve tunable restoration effects is both intellectually interesting and methodologically novel. The experimental results demonstrate impressively effective outcomes.\n\n+ The data engine is technically sound, and the constructed high-quality triplet dataset (530K+ samples) not only supports the current task but also holds significant potential for future low-level vision tasks requiring region-text alignment or extended grounding capabilities.\n\n+ This paper also has a significant practical impact, as it demonstrates compelling use cases that address unmet needs in professional photo editing, i.e. the regional perceptual-fidelity preference and adjustable bokeh preservation. The natural language interface enables intuitive control through verbal instructions, significantly improving workflow efficiency.\n\n\n**Weakness:**\n\n- This paper demonstrates impressive instruction-guided modulation results, but the underlying mechanism of how the current tuning coefficients achieve such effects could be further analyzed. Notably, in the bokeh effect adjustment, the text prompts fed into the SD backbone differ across instructions. Is this prompt variation inherently linked to the framework's working principle? A deeper discussion would strengthen the theoretical contribution.\n- The current framework extracts region captions directly from instructions for the SD backbone input, which prioritizes simplicity. However, prior works like SPIRE show that using more detailed scene descriptions in the backbone's text prompt can enhance global restoration. Would replacing the current region captions with such enriched descriptions further improve performance? A brief ablation study or discussion on this trade-off between simplicity and potential gains would be valuable.\n\nQUESTIONS:\nIn addition to the previous questions, two further clarifications would be helpful:\n\n(1) Are the modulation coefficients bounded (e.g., with predefined min/max limits)? If so, how do these bounds affect the output (e.g., under extreme values)?\n\n(2) The current method focuses on regional control—could the framework inherently support global image-level modulation, or would this require architectural modifications?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

comments: SUMMARY:
This paper proposes a method to perform region-adjustable image restoration following human instructions. Specifically, it constructs a dataset to support the training and evaluation of this task. Then it develops a ControlNet-like model to enable region-customized image restoration that aligns with user instructions. Extensive experiments are conducted to evaluate the effect of the proposed method.

STRENGTHS AND WEAKNESSES:
Strengths: First, this work advances the investigation of interactive image restoration and enhancement techniques. Second, it constructs a large dataset to support the research of these techniques. Third, it proposes a model to understand user instructions for region-customized restoration.

Weaknesses: The effect of each component in the proposed method is not very clearly demonstrated.

QUESTIONS:
1. The authors claim in L44-45 that users generally prioritize perceptual quality for irregular texture regions (e.g., trees), favoring more aggressive detail generation. The statement is not reasonable. For image restoration, the whole image should preserve contetn fidelity and accuracy. However, with generation models, it is harder for irregular texture regions compared to structural elements.
2. Please discuss in detail the effect of the scaling factor \alpha.
3. Please discuss the efficiency of the proposed method and compare with the state-of-the-art methods.
4. It would be better to discuss in detail the effectiveness of the ControlNet features and the restoration mask.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author introduces InstructRestore, a novel framework for region-specific image restoration based on natural language instructions. Besides, the authors construct a large-scale dataset,  containing high-quality images, region masks, and corresponding captions. Compared with existing diffusion-based restoration methods that apply uniform processing across the entire image, InstructRestore enables fine-grained control over different image regions. Experiments show promising results on both general and bokeh-aware restoration tasks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The paper addresses a novel and practical problem—region-customized image restoration—through the lens of instruction-following, which is underexplored in existing research.\n- The authors contribute a large-scale, high-quality dataset and may benefit related research in instruction-guided vision.\n- The proposed framework enables fine-grained, user-controllable restoration, including adjustable fidelity scales for different regions, which is both technically interesting and practically useful.\n- The expression is clear.\n\n### Weaknesses\n\n- The instruction format currently relies on structured templates. While effective, it is unclear how the method would respond to natural or slightly varied user expressions.\n- The paper lacks ablation studies to isolate the effects of key components such as the mask decoder and the fidelity modulation mechanism.\n- The generalization ability of the model to unseen semantic categories or open-domain instructions is not fully explored.\n\nQUESTIONS:\n- How robust is the model to variations in the instruction format? Some examples or quantitative evaluation would strengthen the claims of instruction-following capability.\n- Have the authors tested the model on instructions involving unseen region types or categories not present in the training set?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper integrates image super-resolution with the bokeh task, enabling precise restoration of target regions and differentiated background processing based on user instructions. By integrating models like Semantic-SAM, Osprey, and Qwen, it builds an image–mask–text triplet. Using ControlNet, it parses user instructions to generate spatial masks and applies different enhancement scales, achieving fine-grained, end-to-end control over region and intensity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This work combines image super-resolution with bokeh, showing practical application value in scenarios requiring both clear foregrounds and blurred backgrounds.\n2. The constructed triplet dataset is valuable for region-related image editing tasks.\n3. The paper is well-written and easy to read.\n\nWeaknesses：\n1. The novelty is limited and largely based on combinatory innovation. The integration of natural language instructions with region editing in the image editing domain (e.g., IIR-Net [1]) has already been widely explored. Additionally, there are existing works based on natural language instructions in the image super-resolution field (e.g., SPIRE [2], InstructIR [3], PromptSR [4] ). Furthermore, the use of region masks in this paper has also been studied in the image editing field (e.g., DIFFEDIT [5]). Therefore, the paper does not seem to introduce a new contribution in terms of methodology.\n\n2. The title \"Instruction-Customized Image Restoration\" is inappropriate. The paper only addresses one task in image restoration (super-resolution) and one task in image editing (Bokeh). Firstly, Bokeh is not an image restoration task but rather belongs to image beautification or editing. Secondly, common image restoration tasks such as dehazing, denoising, low-light enhancement, and deblurring are not explored in this work. Therefore, the task setting in this paper is not appropriate.\n\n3. There is a lack of comparison with recent works, such as image super-resolution methods like InvSR [6], PiSASR [7], and others. Additionally, there is no comparison with the performance of Bokeh methods like drbokeh [8], bokehme [9], etc.\n\n4. In the image super-resolution task, the authors compare their method on the custom-built Instruct100Set rather than widely-used image super-resolution benchmark datasets, which diminishes the persuasiveness of the results. Additionally, the comparison methods lack the latest image super-resolution techniques mentioned earlier. Furthermore, Figures 10 and 11 in the supplementary materials show that the visual quality of the image super-resolution results does not surpass OSEdiff, and the proposed method lacks detail, appearing blurry and smeared.\n\n[1] Zhang, Zhongping, et al. \"Text-to-image editing by image information removal.\" Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2024.\n\n[2] Qi, Chenyang, et al. \"Spire: Semantic prompt-driven image restoration.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[3] Conde, Marcos V., Gregor Geigle, and Radu Timofte. \"Instructir: High-quality image restoration following human instructions.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[4] Chen, Zheng, et al. \"Image super-resolution with text prompt diffusion.\" arXiv preprint arXiv:2311.14282 (2023).\n\n[5] Couairon, Guillaume, et al. \"Diffedit: Diffusion-based semantic image editing with mask guidance.\" arXiv preprint arXiv:2210.11427 (2022).\n\n[6] Yue, Zongsheng, Kang Liao, and Chen Change Loy. \"Arbitrary-steps image super-resolution via diffusion inversion.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[7] Sun, Lingchen, et al. \"Pixel-level and semantic-level adjustable super-resolution: A dual-lora approach.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[8] Sheng, Yichen, et al. \"Dr. bokeh: differentiable occlusion-aware bokeh rendering.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[9] Peng, Juewen, et al. \"Bokehme: When neural rendering meets classical rendering.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\nQUESTIONS:\n1. Different Bokeh scales should be fully matched with the instructions. Apart from Figure 6(a), more experimental results should be provided to demonstrate this point.\n\n2.  What is the purpose of Figure 6(b)? As the parameter S1 decreases (0.6h and 0.4), more details inconsistent with the real images are generated, suggesting that more imagined components are introduced. This seems to reflect an adjustment of the consistency and detail generation capability of the diffusion model-based image super-resolution, rather than enhancing the clarity of the foreground objects. \n\n3. The experimental comparison in Table 4 is unfair. Even though the two parameters of the model have been set, other comparison methods have not been trained on the Bokeh training set, making it an OOD task for them.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}