Title: User-Instructed Disparity-aware Defocus Control
Venue: NeurIPS 2025
Accepted: yes
Keywords: disparity-aware kernel estimation

joint deblurring and reblurring
Abstract: In photography, an All-in-Focus (AiF) image may not always effectively convey the creator’s intent. Professional photographers manipulate Depth of Field (DoF) to control which regions appear sharp or blurred, achieving compelling artistic effects. 
For general users, the ability to flexibly adjust DoF enhances creative expression and image quality. 
In this paper, we propose UiD, a User-Instructed DoF control framework, that allows users to specify refocusing regions using text, box, or point prompts, and our UiD automatically simulates in-focus and out-of-focus (OoF) regions in the given images.
However, controlling defocus blur in a single-lens camera remains challenging due to the difficulty in estimating depth-aware aberrations and the suboptimal quality of reconstructed AiF images. To address this, we leverage dual-pixel (DP) sensors, commonly found in DSLR-style and mobile cameras. DP sensors provide a small-baseline stereo pair in a single snapshot, enabling depth-aware aberration estimation. Our approach first establishes an invertible mapping between OoF and AiF images to learn spatially varying defocus kernels and the disparity features. These depth-aware kernels enable bidirectional image transformation—deblurring out-of-focus (OoF) images into all-in-focus (AiF) representations, and conversely reblurring AiF images into OoF outputs—by seamlessly switching between the kernel and its inverse form. These depth-aware kernels enable both deblurring of OoF images into AiF representations and reblurring AiF images into OoF representations by flexibly switching its original form to its inverse one. For user-guided refocusing, we first generate masks based on user prompts using SAM, which modulates disparity features in closed form, allowing dynamic kernel re-estimation for reblurring. This achieves user-controlled refocusing effects. Extensive experiments on both common datasets and the self-collected dataset demonstrate that UiD offers superior flexibility and quality in DoF manipulation imaging.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the User-Instructed DoF Control (UID) framework, a novel method for interactive depth-of-field manipulation in out-of-focus (OOF) images. The core innovation is enabling users to specify a desired in-focus region by leveraging the Segment Anything Model (SAM) for intuitive object selection. To accurately estimate the spatially varying blur, the framework utilizes dual-pixel data, which provides crucial sub-pixel disparity information for depth inference. The proposed method unifies two related tasks—OOF deblurring and all-in-focus generation—into a cohesive two-stage architecture that shares a common latent representation, allowing for efficient and effective refocusing and image restoration. Experiments validate the framework's design, demonstrating the significant benefits of dual-pixel inputs and the joint-task approach.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n1. **Novel User Interaction:** The integration of the Segment Anything Model (SAM) for defining the region of interest is a modern and highly effective approach to user interaction. It significantly improves usability over traditional methods like manual scribbles or mask drawing.\n2. **Elegant Unified Framework:** The two-stage architecture that jointly handles OOF deblurring and refocusing is a logical and efficient design. Sharing latent information between these highly related tasks is a promising direction.\n\n\n### Weakness\n1. **Clarity, Notation, and Figure Legibility:** The paper's clarity is significantly hampered by notational inconsistencies and confusing figures. For instance, there is a discrepancy between the notation in the equations ($F_{int}$ in Eq. 5, 6, 7) and the architecture diagram ($F_{init}$ in Figure 2). Furthermore, the annotations used to describe the methodology in Section 3.1 do not align with those presented in Figure 2. These issues make it exceptionally difficult for the reader to follow the model's architecture and data flow.\n\n2. **Justification of Core Methodology (Decoder vs. Physics-Based Model):** The paper's methodology relies on a learned decoder for the final refocusing step, but the motivation for this choice over a more traditional, physics-based approach is unclear. Given that the framework estimates disparity, this information could be used to model a Point Spread Function (PSF) for refocusing. A PSF-based method offers greater interpretability and may be less prone to the hallucination of textures—a known risk in purely generative models.\n\n3. **Unclear Design Rationale:** The manuscript lacks a clear narrative justifying its architectural and component-level design choices. While the ablation studies demonstrate that each component contributes to the final performance, the paper does not adequately explain *why* these specific components were chosen over other well-established alternatives in the literature.\n\n4. **Absence of a Limitations Section:** The paper fails to include a discussion of its limitations. The authors should discuss potential failure cases or negative impacts, such as performance on scenes with extreme blur, subjects where SAM segmentation might be ambiguous, or the method's sensitivity to the quality of the dual-pixel input.\n\n5. **Inadequate Citation Practices:** Citations for baseline methods are missing from several tables and figure captions. This forces the reader to search through the text to identify the compared methods, significantly harming readability. For clarity and proper academic credit, all baseline methods must be cited directly within those elements.\n\n6. **Fairness of Experimental Comparison:** The experimental evaluation compares against single-image (SI) and dual-pixel (DP) defocus deblurring methods. Comparing a method that utilizes dual-pixel (DP) data against those that use only a single image is not a fair assessment of state-of-the-art performance. The paper's central claims would be much stronger if the primary comparison was against other existing methods that also leverage dual-pixel data. Simply cite those SI methods properly in related work and focus more on the ablations studies.\n\nThere are significant and concerning discrepancies between the manuscript and the authors' responses in the NeurIPS checklist. This raises serious questions about the submission process.\n1.  **On Limitations:** Checklist item 2 asserts that limitations are addressed in the future work in the supplementary material. However, the manuscript lacks a dedicated limitations section. The \"Future Work\" section the main paper is not an adequate substitute for a critical analysis of the current method's weaknesses and failure modes.\n2.  **On Data and Code Access:** Checklist item 5 response regarding the public availability of code and data appears to be inconsistent. The author should stated \"No\" if they wish to published after acceptance.\n3.  **On Computational Resources:** Checklist item 8, the submission does not detail the computational resources used. While implementation details are provided, information regarding the hardware (e.g., GPU model), training times, and inference speeds is absent, which is critical for assessing reproducibility and efficiency.\n\nQUESTIONS:\n#### **Regarding Clarity & Methodology**\n\n1.  Could the authors please provide a revised version of Figure 2? The current diagram is difficult to parse due to notational inconsistencies and an unclear depiction of the framework's data flow.\n2.  What is the specific rationale for using a learned decoder over a more interpretable, physics-based Point Spread Function (PSF) model? Could the authors demonstrate or elaborate on the distinct advantages of their chosen approach?\n3.  Could the authors elaborate on the design rationale for their architectural components? For instance, what motivated the choice of specific modules over other common alternatives in the literature?\n\n#### **Regarding Robustness & Limitations**\n\n4.  How robust is the framework to challenging inputs? We ask the authors to provide analysis on failure cases, specifically regarding (a) inaccurate segmentations from SAM, and (b) images with very large amounts of defocus blur.\n5.  Could the authors add the \"Limitations\" section as indicated in their submission checklist? This section should detail the method's specific failure modes and boundary conditions.\n\n#### **Regarding Submission Compliance**\n\n6.  Could the authors provide the missing details on computational resources (GPU model, training/inference times) as stated in the checklist? This information is essential for reproducibility.\n\n7. Could the author update the checklist for \"Open access to data and code\".\n\nLIMITATIONS:\nNo, stated in Weakness 4.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors  present UiD, a novel framework for user-instructed depth-of-field (DoF) manipulation. Users can use the system's point, box, or text prompts to refocus images. The main concept is to use disparity and defocus cues taken from DP image pairs to jointly model invertible deblurring and reblurring processes. Realistic refocusing using user-specified masks derived from SAM is made possible by the method's estimation of disparity-aware defocus kernels. When compared to multiple baselines, the paper shows better results in both synthetic refocusing and defocus deblurring.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- Novel formulation: Treats refocusing as an invertible problem that connects deblurring and reblurring in a mathematically grounded framework.\n- Dual-pixel leverage: Effectively utilizes stereo disparity from DP sensors, offering better depth-aware refocusing.\n- User control: Enables intuitive control using natural prompts, supported by prompt-based segmentation model.\n\nWeaknesses:\n\n- The framework requires dual-pixel (DP) input, which restricts deployment to DP-enabled cameras. It is unclear how the method performs on monocular or stereo-only inputs.\n- UiD use SAM for segmentation masks but does not evaluate the impact of mask quality or conduct robustness tests on prompt errors or segmentation failures.\n- While the method emphasizes user control and artistic intent, it lacks perceptual or user experience studies to validate subjective quality or usability.\n\nQUESTIONS:\n- What happend if the mask produced by SAM is imprecise or unclear? Does the system generate invalid defocus regions or does it degrade gracefully?\n\n- How robust is your method to different DP hardware calibrations or disparities with high noise?\n\n- What is the need for invertibility? When compared to standard encoding-decoding, can you measure the improvement in results?\n\n- Is the method fragile to segmentation boundary errors or small masks?\n\n- Is performance degraded with imprecise or noisy user prompts?\n\nLIMITATIONS:\n- Assumes access to DP sensor data — not all phones/cameras offer this.\n- Refocusing depends on SAM segmentation, but SAM is not evaluated as part of the system.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces User-instructed Disparity-aware Defocus control, a framework that lets everyday photographers refocus still images after capture by simply pointing, boxing, or describing the region they want sharp. Leveraging the tiny stereo baseline of dual-pixel sensors found in many DSLR-style and smartphone cameras, UiD learns an invertible network that couples deblurring with reblurring in a single model. During training, it establishes a disparity-aware feature space and a set of spatially varying blur kernels so that the same kernels can be inverted to go from blurred to sharp or vice-versa. At test time, a user-prompted mask modulates the learned disparity features, enabling seamless refocusing that keeps selected regions crisp while realistically blurring others. Extensive experiments on public DP datasets and a new real-world DP benchmark show UiD achieves state-of-the-art deblurring quality, flexible and visually convincing refocus effects, and lower computation than prior DP or single-image methods.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n1. UiD accepts text, box, or point prompts and turns them into masks with Segment Anything, so anyone can selectively keep subjects sharp while blurring the rest after capture—no depth maps or manual masking needed. \n2. Its invertible disparity-aware network couples deblurring and reblurring with shared spatially varying kernels, delivering state-of-the-art PSNR/SSIM on DP5K and DPD-blur while using the smallest compute budget among dual-pixel methods.\n\n[Weaknesses]\n1. The paper’s main weakness is that its contribution is confined to dual-pixel imagery. There are also deblurring and defocus models [1, 2, 3, 4] that operate on stereo images, which are closely related. Whereas dual-pixel sensors supply only an ultra-small parallax of roughly ±0.5 px, a stereo camera can adjust its baseline to achieve disparities ranging from tens to hundreds of pixels, an advantage that improves both depth accuracy and motion estimation. While the paper rightly targets the typical mobile-device scenario, additional analysis is needed before the method can be extended to settings such as robotics or autonomous driving.\n\n2. The method turns user clicks/boxes into masks with Segment Anything; any over- or under-segmentation directly skews the disparity modulation and produces visible artifacts, yet the paper offers no quantitative analysis of this failure mode. \n\n[1] Pan, Liyuan, et al. \"Joint stereo video deblurring, scene flow estimation and moving object segmentation.\" IEEE Transactions on Image Processing 29 (2019): 1748-1761.\n[2] Zhou, Shangchen, et al. \"Davanet: Stereo deblurring with view aggregation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n[3] Sellent, Anita, Carsten Rother, and Stefan Roth. \"Stereo video deblurring.\" Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14. Springer International Publishing, 2016.\n\nQUESTIONS:\n1. The paper’s main practical limitation is its DP-only scope. Showing either positive evidence of transfer or a principled explanation of the failure modes would greatly improve confidence that the approach is not narrowly engineered. If possible, could you show the results on a more general stereo dataset with larger disparities?\n\n2. For an interactive refocus tool the user experience hinges on latency. Concrete numbers would substantiate the “smallest compute budget” claim. The paper highlights low FLOPs, but latency on mobile hardware is not reported. Can you provide inference time on smartphone SoC or Apple CPU?\n\nLIMITATIONS:\n\"yes\"\nHowever, the main text does not include a separate “Limitations” section; instead, potential improvements (such as the application of a diffusion framework in future research) are indirectly addressed in the Conclusion/Future Work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes UiD, a User-Instructed defocus control framework that enables flexible Depth of Field (DoF) manipulation in images captured by dual-pixel (DP) sensors. The system allows users to specify refocusing regions via text, box, or point prompts, which are converted to masks using SAM. UiD features an invertible deblurring-reblurring architecture that learns spatially varying defocus kernels and disparity features from DP pairs, establishing a closed-form mapping between blur and disparity. This enables both restoration of all-in-focus images from blurred inputs and controlled reblurring for user-specified refocusing. The approach is validated through extensive experiments on standard datasets and a new self-collected DP dataset, demonstrating superior flexibility and quality in DoF manipulation.\n\nSTRENGTHS AND WEAKNESSES:\nThe strengths of this article are described as follows:\nProposes and validates a mathematically-driven invertible deblurring-reblurring framework that achieves SOTA deblurring performance while significantly reducing computational costs, without relying on depth annotations.\nBeyond SAM-powered interactive refocusing, the work contributes a novel real-world DP dataset addressing data scarcity and demonstrates robust out-of-distribution generalization.\n\nThe weaknesses of this article are described as follows: \nThe core framework heavily relies on existing \"deblur-and-reblur\" paradigms (e.g., RefocusGAN, DC²) without introducing significant methodological advancements. Furthermore, the user interaction mechanism depends entirely on off-the-shelf SAM for mask generation, with no task-specific modifications. This unmodified pipeline integration substantially dilutes the claimed contribution of \"user-instructed\" control.\nThe paper claims to support text, box, and point prompts for specifying refocusing regions, yet fails to provide essential implementation details (e.g., text-to-mask conversion, point/box coordinate handling in defocused images) or experimental validation of these interactive features. \nThe self-collected DP dataset lacks a detailed description, including the size and characteristics of the dataset. In addition, ablation studies are insufficient, such as the full range of different variants of the loss function, and exploration of the weights λ is also missing.\nThere are also many formatting errors in the paper, for example, the expression  in Eq. (13) does not match the above, there is a missing period at the end of the title of Table 2, there is a spelling error in “caluclate”, and there is a redundant period in “Figure. 6”, and so on.\n\nQUESTIONS:\nThe framework presented in this paper lacks originality, and in order to demonstrate the novelty of the article, it is necessary to elaborate on the differences from existing methods as well as the implementation details that are fully elaborated.\nAlthough some quantitative results are provided, the qualitative results are not sufficiently presented to visually assess the performance of the UiD framework. In addition, the necessary ablation experiments and detailed implementation details should be added to show the effectiveness of the individual components.\nThere are multiple formatting errors in the article that lead to overall confusion, and the entire text needs to be scrutinized and corrected to improve readability.\n\nLIMITATIONS:\nThe author of this article has fully elaborated on the advantages of the proposed method, but lacks the necessary exploration of its limitations and any potential negative social impacts, which is not conducive to the development of subsequent work. Therefore, it is necessary to supplement the content on limitations to provide direction for future research.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-02"}