User-Instructed Disparity-aware Defocus Control
Authors: Yudong Han
Yan Yang
Hao Yang
Liyuan Pan
Venue: NeurIPS 2025 poster ()
Keywords: disparity-aware kernel estimation
joint deblurring and reblurring
Primary area: deep_learning
TL;DR: disparity-aware joint deblurring and reblurring as a proxy of refocusing

Abstract: In photography, an All-in-Focus (AiF) image may not always effectively convey the creator’s intent. Professional photographers manipulate Depth of Field (DoF) to control which regions appear sharp or blurred, achieving compelling artistic effects. 
For general users, the ability to flexibly adjust DoF enhances creative expression and image quality. 
In this paper, we propose UiD, a User-Instructed DoF control framework, that allows users to specify refocusing regions using text, box, or point prompts, and our UiD automatically simulates in-focus and out-of-focus (OoF) regions in the given images.
However, controlling defocus blur in a single-lens camera remains challenging due to the difficulty in estimating depth-aware aberrations and the suboptimal quality of reconstructed AiF images. To address this, we leverage dual-pixel (DP) sensors, commonly found in DSLR-style and mobile cameras. DP sensors provide a small-baseline stereo pair in a single snapshot, enabling depth-aware aberration estimation. Our approach first establishes an invertible mapping between OoF and AiF images to learn spatially varying defocus kernels and the disparity features. These depth-aware kernels enable bidirectional image transformation—deblurring out-of-focus (OoF) images into all-in-focus (AiF) representations, and conversely reblurring AiF images into OoF outputs—by seamlessly switching between the kernel and its inverse form. These depth-aware kernels enable both deblurring of OoF images into AiF representations and reblurring AiF images into OoF representations by flexibly switching its original form to its inverse one. For user-guided refocusing, we first generate masks based on user prompts using SAM, which modulates disparity features in closed form, allowing dynamic kernel re-estimation for reblurring. This achieves user-controlled refocusing effects. Extensive experiments on both common datasets and the self-collected dataset demonstrate that UiD offers superior flexibility and quality in DoF manipulation imaging.

User-Instructed Disparity-aware Defocus Control
Yudong Han'
Yan Yang² Hao Yang' Liyuan Pan 1.35a
Beijing Institute of Technology, Beijing, China BDSI,, Australian National University, Canberra, Australia
Yangtze Delta Region Academy of Beijing Institude of Technology, Jiaxing, China hanyudong sdu@gmail. com, yan. mangyann@an. edu. au, yaao. yang, liyuan, Liyuan.pan](bir pant edu. cn
Figure 1: An example of refocusing using our method. The input and refocused images are on the left and right. Two regions are highlighted with red and blue boxes, and their zoomed-in views are displayed in the center.
Abstract
In photography, an All-in-Focus (AiF) image may not always effectively convey the creator's intent. Professional photographers manipulate Depth Field (DoF) to control which regions appear sharp or blurred, achieving compelling artistic effects. For general users, the ability to flexibly adjust DoF enhances creative expression and image quality. In this paper, propose UiD, User-Instructed DoF control framework, that allows users to specify refocusing regions using text, box, or point prompts, and our UiD automatically simulates in-focus and out-of-focus (OoF) regions in the given images. However, controlling defocus blur in single-lens camera remains challenging due to the difficulty in estimating depth-aware aberrations and the suboptimal quality of reconstructed AiF images. To address this, we leverage dual-pixel (DP) sensors, commonly found in DSLR- style and mobile cameras. DP sensors provide a small-baseline stereo pair in a single snapshot, enabling depth-aware aberration estimation. Our approach first establishes an invertible mapping between OoF and AiF images to learn spatially varying defocus kernels and the disparity features. These depth-aware kernels enable bidirectional image transformation-ddeluurrtt into all-in-focus (AiF) representations, and conversely reblurring AiF images into OoF outputs by seamlessly switching between the kernel and its inverse form. For user- user-guided refocusing, we first generate masks based on user prompts using SAM, which modulates disparity features in closed form, allowing dynamic kernel re-estimation for reblurring. This achieves user-controlled refocusing effects. Extensive experiments both common datasets and the self-collected dataset demonstrate that UiD offers superior flexibility and quality in DoF manipulation imaging.
out-of of-fu (OoF) images
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Modern cameras utilise sophisticated compound lenses designed to focus light onto the sensor. However, adjusting the aperture has another effect: a larger aperture (lower f-number) introduces depth-dependent blur, rendering only specific regions in focus [20]. This depth of field (DoF) effect is often intentionally used by photographers to direct viewer attention to a subject (e.g., macro photography) or create artistic compositions (e.g.. tilt-shift photography) [2]. Unlike professional photographers, who possess the expertise to control DoF effectively, general users often struggle with achieving the desired DoF composition. Therefore, a framework that enables users to flexibly manipulate sharp and blurred regions in their captured images using simple text or point-based inputs to create visually striking effects would be a valuable application.
Given images with undesired DoF regions captured by general users, effectively controlling depth- dependent blur involves several key steps: (i) estimating the all-in-focus (AiF) image; (ii) computing the corresponding depth/defocus map of the undesired image; (iii) generating mask to designate regions for in-focus and out-of-focce (OoF); and (iv) adjusting the defocus map with the mask for refocusing. Inspired by [52], which employs dual-pixel (DP) sensors [8] to synthetically generate blur on human subjects, we integrate DP sensors into our framework for flexible, user-instructed refocusing.
DP sensors widely used in DSLR and smartphone cameras [22]. These sensors divide each pixel into two photodiodes, capturing separate left and right views of a scene, effectively forming a stereo system with a tiny baseline [39]. The depth information inherently encoded in the DP pair also contains defocus information [6]. which makes DP image pairs highly beneficial for tasks such as defocus deblurring, refocusing, and bokeh effect synthesis [52]. Therefore, in this work, we focus on DP refocusing by incorporating user-specified prompts to identify desired focus regions, thereby enabling our network to effectively address all key steps within a single unified framework.
Different from existing work [52] synthesizes DSL R-yte images from smartphone-captured data by directly treating the input DP image the AiF image during refocusing- our task is more challenging. First, to estimate the latent AiF image and the defocus map, we jointly optimize defocus deblurring and reblurring. In contrast to earlier methods [5, 45, 56, 57] that treat deblurring and reblurring as independent processes or rely on ground truth depth maps for reblurring, we propose an invertible deblurring and reblurring framework that learns the robust defocus map and the spatially- varying associated blur kernels to assist the final refocusing step. Next, to facilitate mask generation, we employ flexible user inputs- such as point or box prompts- and models like SAM [43, 62] to generate masks that guide the refocusing process. Finally, to ensure that the generated mask effectively controls the depth-aware blur, we use the blur kernels learned by the invertible deblurring and reblurring framework to produce the final refocused image.
Specifically, first analyze the relationship between DP disparity, the circle of confusion (CoC, or defocus map), and spatially varying defocus blur, which leads to a closed-form solution for controlling the disparity of a DP pair to achieve refocusing. This insight enables our network to perform controllable refocusing at test time simply by specifying regions of interest-even without explicit training on refocused images or reliance on depth annotations. Building on this understanding, we perform both invertible deblurring and reblurring within a unified framework, enabling the network to learn and regularize the mapping between blurred DP inputs and the sharp image in a self confisonnitt manner via learned disparity features. These learned disparity features are then converted into CoC features, which are used to construct spatially varying deblurring kernels. To ensure consistency, these kernels are further constrained to perform reblurring when inverted, thereby enabling effective simulation of the DP model. During testing, we first restore an all-in-focus image from the input DP pair and then modulate its disparity using user-specified mask to generate the target CoC features, addition to comprehensively
guiding the reblurring process for spatially controllable refocusing. evaluating the robustness and flexibility of the proposed method in real-world refocusing scenarios and to enrich community benchmarking, we further construct a dual-pixel real-world evaluation dataset. The detailed evaluations and visualizations will be provided in the supplementary material. Our contributions are as follows:
A framework that enables general users to flexibly manipulate sharp and blurred regions using simple text or point-based inputs.
An invertible deblurring and reblurring framework that learns the defocus map and associated blur kernels in close alignment with the mathematical formulation of the DP model.

---

Extensive experiments, including self-collected dual-pixel real-world dataset, on defocus deblurring and image refocusing, demonstrate the superior performance of our method.
2 Related Work
Defocus Deblurring. Defocus blur refers to the blur effect when observed objects fall outside the depth-of-ficld (DoF) of a camera lens. To recover lost details from such blurred images, one videly-embraced scheme [1, 26, 58] is two-stage pipeline: (1) estimating explicit defocus map that accurately reflects the level of blur, followed by (2) performing deblurring guided by this map. Earlier approaches often relied hardware assistance [10, 12, 17] to obtain depth map proxy for defocus map in deblurring tasks. To improve flexibility, several studies have explored using pre- pre-trained monocular depth estimators to infer depth maps from single images, which are then used as guidance for defocus-aware deblurring [40, 41J. While these methods offer greater accessibility, they often struggle to generalize beyond their training datasets. To enhance depth estimation accuracy for defocus map recovery, other researchers have incorporated additional cues, such as dual-pixel (DP) disparity information that significantly improves the guidance quality during the deblurring process [42]. Wadhwa et al. [52] propose the first framework that uses DP sensors enlarge the blur effect in the background to sythesize the DSLR-style images, and assume the foreground faces are in focus. Different from the settings in [52], in this paper. our UiD leverages the disparity cues encoded DP image pairs to flexibly control the blur in scalable and effective manner.
data [52]. DP images provide defocus-relaiee
Image Refocus. Achieving accurate refocusing in post-capture images challenging. as it requires both deblurring sharp regions and synthesizing realistic blur effects that align with the desired DoF [45]. Several studies [16, 36, 54] have explored software- and wardware-based approaches to enhance refocusing performance. Prior work [36, 54] often relies on specialized hardware to capture light field information, enabling post-capture focus control. However, light field cameras typically suffer from limited spatial resolution, making it difficult to capture intricate scene dynamics. An alternative solution [7, 48] is to capture focus stack and leverage multi-framm frameffrmee information to estimate the desired focus distance. However, capturing long-term focus stacks inevitably leads to scene evolution, restricting refocusing to static environments [5]. Another line of research focuses on single-image refocusing, where the typical framework first deblurs the image to obtain an all-in-focuu (AiF) representation, followed by reblurring. A recent representative study, RefocusGAN [45], trains two-stage conditional GAN [35] for sequential deblurring and reblurring, enabling flexible single-image refocusing using a focus control vector. The most recent work, DC [5], adopts a dual-camera setup, where the captured fide-frame and altra-wide-frame images first aligned, and a built-in depth sensor is leveraged to guide the image refocusing process. In this work, we follow the leblur-and-reblu paradigm and develop disparity- invertible framework to overcome the limitations of inaccessible hardware gc.g.. built-in depth sensors), the constraints of capturing a focus stack, and the challenge of achieving both defocus deblurring and selective reblurring.
3 User-Instructed Disparity-aware Defocus Control
Motivation. In the DP image pair, two images B, and B, are generated by light rays from the real scene passing through the left and right sub-apertures into two juxtaposed half pixels. The disparity D for DP pair (B,, B,) at location (i,j) is approximately.
D(i,))= = f x B/)( + const = f f B)Z(i.))
(I)
where f is the focal distance, B denotes the shift between two sub- aperttres and Z(i.j) denotes the scene depth. DP disparity is strongly correlated with the degree of defocus blur [42]. Specifically, the blur radius at location (i.j) is given by,
C(i.)) J(} D(i.)).
where J(f) is a function of the focus distance f, controlling the strength of the blur. C known as CoC. When refocusing to objects indicated by a binary mask M. its blur radius follows linear relation with respect to the disparity offset from the target focal disparity. The refocused blur radius (i, becomes,
C(i,) = J(f) (D(i.)) - derg).
(3)
3

---

where the target focal disparity derg is computed as the average disparity within the masked region:
degg
1
Σ
D(",)")
(P ((
with [92] denoting the number of pixels where M(',j') = 1. This formulation enables spatially
adaptive refocusing to user-specified regions of interest defined by the mask.
Overview. Our key idea is to learn disparity-aware feature space that adheres to Eq. 2, enabling a blur- aware representation suitable for both deblurring and refocusing. Leverag- ing standard DP defocus-deblurrin datasets containing DP pairs (B,,B, and their corresponding sharp images I, we train an invertible network that jointly models deblurring and reblur- ring (i.e., red, orange, and green arrow in Figure 2). During training, CoC- based feature F coc is computed in the feature space by combining a disparity feature and blur strength feature ac- cording Eq. 2. This feature used to generate spatially varying deblur- ring kernels K that restore the input DP pair (B,, B,) into a sharp image i. Reblurring is then performed by inverting these kernels, using K- to degrade the sharp image î hack into a blurred DP pair. At test time, we first apply the deblurring process to (B,, B,) recover the sharp image i. To perform refocusing, the disparity feature obtained during deblurring is modulated according to Eq. 4, resulting in a target CoC feature Firg, computed using the user-provided mask M and blur level as in Eq. 2. The mask can be derived from point or box prompts by leveraging the SAM-like model [43, 62]. This target feature guides the reblurring process, transforming the sharp image I into refocused DP image. overview of the network architecture is shown in Figure 2, where deblurring, reblurring, and refocusing operations are carried out in a shared feature space learned by encoder Fe(..)) and decoder Fa(.).
F d
backward
Fe
B
K
M
User Prompt: Red Hat
F coc
inverse
SAM
F
F
FV sam
(B,, B,)
F e
Fa d
forward
Figure 2: Illustration of overall framework. The arrow with red color and green color denotes the deblurring pathway, and the arrow with orange color and green color denotes the reblurring pathway. These two pathways are collabora- tively trained with disparity-awand shared kernel learning (i.e., green arrow). The arrow with gray color indicates the user-instructed mask generation, which is only activated dur- ing inference time. The dotted box with yellow color denotes the early fusion paradigm. Fm can be multimodal encoder or MILMM [29-31], ], proj is the projector, and F denotes the visual encoder of SAM.
3.1 Deblurring
Feature Encoding, We restore the DP pair (Br, B,.) in feature space by first embedding them into base feature representation F B using an encoder Fo(-.)). In parallel, disparity-related feature Fe is extracted using a stereo matching network G(.,-) applied to the DP pair. Within G(-.)), we first use dual visual encoder for feature extraction, forming Fi, and F,,6- For each pixel F1,6, we search for its counterpart F Fr.b within scope d horizontally, where d is defined as the disparity index. We calculate all the possible disparity using widely-used matching costs following [50], where the left and right features concatenated to form the cost volume,
Cala, v) [F1,(0, v), -x d,y),
where x and y are the pixel coordinates. Then, all the obtained disparity feature C. are concate- nated, forming C [C,, Ca.... Cp]. where D denotes the maximum disparity scope. Afterwards, two cascaded 3D convolutional networks are sequentially employed for aggregating the geometry information stored in the cost volumes C, learning the approximate disparity clues in each location. Finally, a lightweight regression block is used on the top of network to compute the expected disparity feature Fc.
While disparity or depth is fundamental to refocus control, directly manipulating the feature space spanned by FG based on these geometric cues is infeasible. In other words, the lack of an interpretable

---

correspondence between feature dimensions and defocus blur prevents meaningful, controllable editing. Therefore, we thus further normalize the DP disparity feature and scale follow,
FD = 2 Tanh (Norm(Fc)/p) 11
(6)
where Tanh(-) denotes the hyperbolic tangent function, Norm(-) is the &g normalization along with channel dimension, and is a temperature parameter that sharp the distribution of the output. By this way, we condense the depth information conveyed in F into a one-channel disparity map F D. which serves direct interface for refocus control. Recognizing that the DP pair also encodes focal mangth-dependem blur information, we extract this information from feature F G using an encoder J-).. Following Eq. 2, the initial CoC-related feature can be rewritten as,
F iinit = ((G)) FD
Next, apply decoder-encoder structure to refine Finit- The refined feature is projected into RGB space via projection head P (rgo()), and projected back to the feature space through a second projection head P-eat(-), with a gating residual connection [47] for numerical stability.
Feoe R @ FG + Peeat (Prgb (F (Fimit).
(8)
where denotes the learnable gate (see supplementary material for details). This recoder-encodee structure enables auxiliary supervision either through ground-truth CoC annotations via self- supervised CoC estimation applied to the RGB-projected output Prgb (Finit).
Feature Restoration. By maintaining set of N learnable blur kemelss (K,)) we use the CoC-related feature F coc to query spatially-varying deblurring kernels from the set, and apply the queried kernels restore the feature FB. Note that keep F ooe and B as the same spatial size. Specifically, for each pixel (i,j), a spatially-varying kernel Kiş is constructed as a weighted sum of the kernel set,
K.j- Σ (n) Kn, a.j Softmax (Pk (Aug( (Kn) @ Pene (F (eoe(i.))))
where a. is a soft attention vector over the N kernels. The function Aug(-) computes the average of each kernel K,, (resulting in a scalar per kernel), forming N-dim vector representation. This vector is element-wise multiplied @ with the CoC-projected feature Peoe (F (i, and passed through a linear layer P.(.) followed by Softmax function to produce the attention weights. The learnable projections Pa()) and (-) are both implemented as linear layers.
We then convolve FR with queried kernels for each pixel (i,j), and refine the feature by using forward mapping of invertible block Trai(-), to get restored feature F,,
F Fi Tnve Σ Fn(i + 5i, + 55) K (5i,
(10)
81,65
where (8i, 6j) is offset of the receptive field of the spatial kernel K,,, Notably, the special design of architecture theoretically guarantees that Jacobian determinant
non-zero, ensuring that invertibility of overall network, irrespective of the specific form of the components within Inw(-). The details can be referred to supplementary materials.
#F
7F
Feature Decoding. We decode the F (i. RGB space for calculating an all-in-focus restoration with a decoder Fal.) by i- FF(F)).
3.2 Reblurring
Our reblurring process is analogous to the deblurring procedure, but it employs the inverse of the queried kernels. For an all-in-focus image I, we first obtain feature F, by F, FF,, Then, we reverse the computation performed by the invertible block Inv(.) by applying its corresponding inverse operation (t). Simultaneously, for each pixel (1, we calculate the inverse of the deblurring kernel K, with the Fourier transform F(.).
K- x-1=ド-1 F(K)
(11)
5

---

outputting an inverse kernel K for reblurring, where F (-) denotes the inverse Fourier transform. The reblurred feature calculated as,
Fi.
Σ Inv' (F)) di + 6j) K (ōi, 85). 82,63
(12)
The resulting reblurred feature map B then decoded into an image B using a decoder Fa(-). The goal is to synthesize the center view of the DP pair B B + B,
3.3 Refocusing
During testing, to perform refocusing on DP pair (B,, B,) using a user-provided focus mask M we first restore an all-in-focus image i following the forward mapping of our invertible network (Eq. 10). We then reblur the image by modulating the original disparity feature Fn by
D = (i
E
Fo(i',j),
(13)
9
This operation shifts the disparity feature such that the selected region appears in focus, effectively implementing a disparity-aware refocusing strategy, as outlined in Eq. 4. We then calculate an updated CoC feature Firg based on Eq. 8, and repeat the kernel querying process defined in Eq. 9. These kernels are applied in the reblurring process according to Eq. 12, ultimately producing the refocused feature decode the target refocused image, denoted B
Training-free Instructed SAM. Given a user instruction T specifying the region of interest, the instructed Segment Anything Model (SAM) generates target mask M semantically aligned with the prompt. A key challenge lies in obtaining a high-quality segmentation mask to effectively guide the refocusing process. While end-to-end joint training of the instructed SAM with our deblur-and- reblur framework is a natural solution, it suffers from two critical drawbacks: (1) the segmentation mask, when suboptimal in early training stages, dynamically affects the refocusing network and may produce erroneous gradients, impeding optimal convergence; and (2) fine-tuning SAM from scratch with andlur--nd-reblur framework entails significant computational overhead, making the overall process resource-intensive. To circumvent these issues, we first forgo the joint training tactics. Instead, we first training our deblur-and-reblur network using mask-free manner, which means that it does not require masks during training. Mask are only provided at test time, where the user inputs a binary mask specifyy the region of interest for refocusing. Therefore, the degradation on mask would not affect the dynamics and robustness of our model training. Given the inaccessibility of clear reference images during testing, adopt an early fusion strategy to deeply exploit the rich semantic cues conveyed by both the user prompt and the input image, enabling more precise mask generation. This approach demonstrates superior performance compared to the conventional late fusion paradigm [62]. Specifically, it takes the restored image that generated via forward mapping in Eq. 10, prompt 'T, and a random initialized [cls] query as input, and using multimodal transformer-Be based encoder F..(.) to enhance the interaction between modalities,
qsts = Fm (i, T, qess);
(14)
the output [cls] query qels encodes the compact multimodal clues, and then fed into the mask decoder together with the SA.-exttrctedd image features F sam. enabling accurate spatial alignment towards robust mask generation,
Mry F, (gets: F somm
(15)
Furthermore, we localize the region interest using structured, template-based prompts that combine number, object color, category, and spatial position, such as "a brown dog on the chair", to enhance spatial grounding precision. For those still imperfect masks, we further apply post-processing techniques such dilation erosion operations for refinement. The detailed implementation of (.) and F-).) can be referred in supplementary material.
3.4 Loss
Our network is trained with a combination of objectives to supervise deblurring, reblurring, and CoC estimation. Specifically, we use deblurring loss Caes (I, i) and a reblurring loss Cres (B, B) based

---

on multi-scale setting following [57], MSE-based CoC regression loss Leoe (C, Prgo (egb( ((Fne)))) and MSE-based CoC gradient loss L (C, Pryble Fin)) to enhance edge precision. The total loss is,
Cintat Ldet Creb [B B) + Xene Looe (C, Prgh imiit) + Agrad Love (C,
P. Prgb Fimtt) (16)
where each À controls the relative weight of the corresponding loss term.
4 Experiment
Dataset. For defocus deblurring task, we evaluate our method on widely-used DPD-blur [2] dataset and recent DP5K [27] dataset. For refocusing task, we use three datasets for evaluation, DP5K dataset, Dis- dataset, and our self-collected DP dataset. Our model trained using the training splits of DPD-blur and DP5K. Refer to our supplementary material for detailed dataset configurations and evaluation protocols.
Evaluation Metries. We evaluate the model with using standard metrics, i.e., peak signal-to noise ratio (PSNR) [19], structural similarity (SSIM) [55], relative error (RMSE rel) [37], mean absolute error (MAE), and learned perceptual image patch similarity (LPIPS) [61].
Implementation Detail. For training,
the Adam W optimizer [24] with 8n 0.9, 82 0.999, a learning rate of 3 x 10-1, and a weight decay of 10-6. A cosine annealing learning rate [32] scheduler with warmup is employed, where the cycle steps, warmup steps, and minimum learning rate are set to 200, 100, and 6 x 10-5, respectively. For the DPD-blur dataset, the model trained for 40k iterations with batch size of 4. For the DP5K dataset, train the model for 64k iterations with batch size of 6. We elaborate our network architecture and more supplementary material.
setting in
4.1 Experimental Results
As mentioned Sec. 3, image refocusing task decomposed into two key stages: (1) defocus deblurring. and then (2) refocusing. In this section, conduct extensive experiments to evaluate the effectiveness of each part in two stages. First, we perform quantitative and qualitative evaluations of both deblurring and refocusing, comparing their performance with several strong baselines. Then, provide detailed component-we analysis of our refocusing framework.
4.1.1 Quantitative and Qualitative Results Image Refocus
Setting. Image refocus takes a user-specified mask and all-in-focus image input, the goal is to generate a new image where the masked region remains sharp while other areas exhibit controlled defocus blur. Therefore, obtaining all-in-focus image from blurred image is the first step to achieve refocusing, and its restored quality greatly determine refocus performance.
Table Deblurring quantitative comparisons on DPD-blur dataset [2] with several SOTA baselines.
Computional Cost
Quantitative Metries
Method
Parums (M)
Flops
PSNR
SSIMI
MAE
MSE ASE_relqu-1)
single-image defocus deblurring
EBDB
23.45
0.683
0.49
DMENet
26.71
4787
23.41
0.714
0.51
0.67
RDPD
24.28
901
25.39
0.772
0.40
0.53
IFAN
10.48
794
25.99
0.804
0.37
0.50
BAMBNet
4.50
1804
26.40
0.821
0.36
0.47
DeepRIT
3682
25.71
Restormer
26.13
4458
26.66
0.833
0.35
0.46
dual- defocus deblurring
DPDNet
31,03
25.13
0.786
DDDNet
6.40
1661
25.36
0.768
0.41
0.54
K3DN
5.00
1033
26.84
0,829
0.35
0.46
Ours
3.12
495
26.89
0.829
0.33
0.47

---

BAMA
Bluryy Image
GT Image
DDDNet [37]
Blurry Image
Restormer [60]
K3DN [58]
Ours
Blurry Image
GT Image
DDDNet [37]
Blurry Image
Restormer [60]
K3DN [58]
Ours
Figure 3: Qualitative comparison on the DP5K-test dataset [3]. We present the blurry image in the first column, and boxed regions with red color are zoomed results.
Table 2:
Comparison of refocusing perfor- mance on DPSK-test. denotes the model version adapted for refocusing
Table 3: Comparison of refocusing performance on DPD-disp for OOD evaluation. denotes the model version adapted for refocusing.
Methods
PSNR1
SSIMI
(0E | LPIPS
Methods
PSNR
SSIM
MAE 1 LPIPS
Omni-Kemei!
25.32
0.816
0.47
23,5
Omni- Meni-Kemel!
18.75
0.692
0.79
31.4
K3DN
30.76
0.943
0.30
14.4
K3DNF
21.72
0.765
0.67
29.7
Ours
31.21
0.953
0.33
13.8
Ours
21.93
0.772
0.56
25.1
Methods. To evaluate the deblurring ability of our framework, we compared with two primary categories of defocus deblurring approaches. (1) Single-image defocus deblurring: KPAC [49],
RDPD [4], DRBNet [44], DeepRFT [34], IFAN [26]. RAMBNet [28], and Restormer [59]. (2) Dual-
pixel defocus deblurring: DPDNet [2], DDDNet [38], K3DN [57]. For refocusing, BokehMe [40] and Dual-Camera [5] are most closely related to our approach. However, they neither release training code nor pretrained models. Thus, we manually modify two SOTA methods on deblurring task, K3DN and Omni-Kernel [11], to enable their refocusing functionality. Refer to our supplementary materials for details.
Evaluation. The deblurring comparisons with state-of-the-arl methods are given in Table I and Table 5. The result reveals that we achieve superior performance, while having smallest computation cost (495 GFlops). Figure 3 showcases the deblurring results of two cases against several state-of- the-art baselines. Our method able to restore severely blurred regions, especially textual region, compared to existing models. For refocusing, we report the qualitative comparison in Table 2 and
26.0
25.0
24.5
-05
0.0
0.5
Fp
Fp =- 0.4
Fp = 0.2
Fp = 0.
FD 0.2
FD 0.4
FD 0.
Fp 0.8
All-in-Foue Focus Image
Figure 4: Visualization of blur effect by setting (F reblurring using Eq. 12. We take the all-in-focus and user-specified mask (a mask referring to unfolding book this case) as input, and observe the blur variation with different F D. The top-right figure shows the overall PSNR variation trend with different F
Stook = when performing

---

Table 4: The effect of joint deblurring- reblurring training on performance.
Table 5: Comparison of deblurring performance on DP5K-test. The best is bold with black.
Model Variants PSNR SSIMI MAE +
Methods
PSNRT
SSIM?
RMSE (110o1) MAE 4
Deblurring
Perforaance
DDDNet
25.59
0.777
W Lras Ldebs
26.89
0.829
0.35
IFAN
28.23
0.875
0.387
0.44
Lieh
26.80
0.823
0.35
BAMBNer
DeepRFT
30.24
30.54
0.915
0.917
0.297
0.307
0.34
0.37
Rebluning Performance
Restoreer
30.71
0.922
0.291
0.33
Lisb Eich
29.18
0.875
0.22
K3DN
30.82
0.923
0.291
0.32
Erch
29.06
0.863
0.28
Ours
30.72
0.926
0,280
0.30
PD
C9
Blurry Image B
Dehlurred Image i
Reblur using FD
Refocus using F
Zoom in
FD
Prgb (Finit)
FD
pfrg rgg imit
F Fb
Figure 5: Visualization on immediate process in refocusing. Notably, FD and Forg denotes the disparity map used in reblurring and refocusing, and Prgb imit) and Pigg (Fint)) are predicted CoC map from their corresponding disparity map (Eq. 8), which both exhibits satisfactory consistency.
Table 3. We also show two cases from DPD-disp dataset to evaluate the dittr---iittibutionn (OOD) generalization capability of our model, shown in Figure 6.
4.1.2 Component-wiss Analysis
Effect of Joint Deblurring and Reblurring Training. Joint deblurrngg-rbbuuring training enables robust blur kernel estimation and exhibits more flexible refocusing. can be seen the Table model with both Fres and Ldet supervision achieves substantial advantages compared to that only with Creb or Ldet- This observation indicates that one process not only does not disturb another branch but also effectively help to optimize the learned blur kernel, thereby enhancing the representation capability learned by our network. Using the shared kernel deblurring process, user-specified refocusing process can be simply achieved by using the with inverse kernel obtained in Eq. 11, which exhibits satisfactory flexibility and viability.
TOY
Blurry Image B
Blurry Image
Dasny FIXAR TOY STORY
Deblurred Image
Deblurred Image
yhsny POXAR TOY STORY
Refoused Image
Refoused Image
Figure Visualization of refocusing on DPD-disp dataset. The bounding box is highlighted for clear observation, and region with red box refers to the user-specified region. The images in the first row refers to blurred image (unified by (B, + B,)), images in the second row are deblurred image recovered by our model (Eq. 10), and the last row showcases the refocused images.

---

Visualization of Kernel Selection Behavior. We provide the visualization map of retrieved ker- nel index in kernel pool of different pixel locations reveal deeper understanding. We first talk from the learned disparity feature Fa. because the retrieved kernel intrinsically rely on the dis- parity feature. As shown in the second row in Figure 7, we use K-means clustering [23] to group the similar disparity feature with different cluster numbers, ize the clustered group index. We can observe that our learned FG conforms with the actual depth or blur cues in B. Done well with Fg, we verify the reliability of using Fg for kernel selection. As shown in kernel selection of Figure 7, the distribution of retrieved kernel index via Eq. 9 tends approximate the blur distribution of the original blurred image. This obser- vation further demonstrate the feasibility of our design: when perform refocusing, specifying the mask of focus would specify the disparity thereby further determining the kernel selection.
= 6, 10 and then visual-
Necessity of Disparity Alignment towards Refocusing Based on Eq. 2, we conclude that CoC map and dis- parity map provide the approximate indication clues for blur kernel estimation. Particularly, a pixel (i,j) with C(i.j) = D(i,j) = 0 indicates to be well-focuser In practice we empirically observe that training network only with disparity feature F Firtt as blur kernel guidance enables faster convergence while achieving better deblurring and reblurring performance than that only using CoC feature. However, indirect supervision of F inat through joint train- ing on deblurring and reblurring tasks may not adequately capture explicit blur cues. Ideally, regions with FD = 0 should exhibit perfect focus. As illustrated in Figure 4, in the absence of CoC map supervision, the disparity value corresponds to minimum PSNR happen to shift. With the aid of CoC supervision, the book is observed to be visu- ally well-focused exactly. {FD (i. (i,,) E Doook = 0 corresponds to minimum PSNR 23.6dB compared with ground-truth blurred image. It indicates that an accurate correspondence between disparity and blur kernel or defocus blur can be explicitly calibrated with the supervision of CoC map C. Moreover, we visualize the correspondence between F FD and predicted CoC map, known as Prot (F (nit), in Figure 5.
Blurry Image B
Kernel Selection
Clustering on F& (c=10 and c=
Figure 7: Illustration of kernel selection and clustered disparity feature FG-
Ablation of Components in Invertible Mapping. To investigate the impact of different invertible component designs towards the robustness of the visual representation, tailor several variants. As shown in Table 6, model v1 denotes adopting the 1 x convolution for channel shuf- fle [13], model v2 adopts the residual invertible network similar to [18], and model u3 incorporates affine transfor- mation operations for more stable invertibility. Among them, model v3 achieves the best performance. It demon- strates that robust invertible mapping enables more effec- tive kernel learning for refocusing. More details of these variants are provided in the supplementary material.
Table 6: The ablation of joint training on both deblurring and reblurring perfor- mance.
Variants
PSNR1
SSIM
MAE
vl [13]
25.56
0.795
0.52
x2 [13]
26.21
0.810
0.35
x:3 [18]
26.89
0.829
0.31
Conclusion and Future Work
Without bells and whistles, we re-frame the task of image refocus as two interconnected sub- tasks-joint reblurring and deblurring using the shared kernel, and the disparity feature extracted from dual-pixel (DP) image pairs is leveraged as an indicator for blur kernel estimation. Specifically, our work experimentally explores several key aspects: (1) We investigate the potential benefits of using DP image pairs to achieve effective defocus control. (2) We mark the first attempt to demon- strate the feasibility of using joint deblurring and reblurring as a proxy of image refocusing, which is non-trivial endeavor. (3) We analyze the behavior of kernel selection and test its sensitivity to adjustment of disparity. (4) We incorporate an early-fusion SAM into our deblur-and-reblu framework to accurately identify the region of interest, enabling robust mask generation and flexible defocus control. In future work, we aim to adapt our method to the diffusion framework to further explore its practical applications and commercial potential.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China under grant No. 62302045, and the Beijing Institute of Technology Special-Zone.
References
[1] Abdullah Abuolaim, Mahmoud Afifi, and Michael S Brown. Improving single-image defocus
deblurring: How dual-pixel images help through multi-task learning. Proceedings of the IEEF/CVE Winter Conference Applications of Computer Vision, pages 1231-1239, 2022. 3 Abdullah Abuolaim and Michael S. Brown. Defocus deblurring using dual-pixel data. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - FCCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X, volume 12355 of Lecture Notes in Computer Science, pages 111-126. Springer, 2020. 2. 7, 8
[2]
[3] Abdullah Abuolaim and Michael S Brown. Defocus deblurring using dual-pixe data. In
European Conference on Computer Vision, pages 111-126. Springer, 2020.
[4] Abdullah Abuolaim, Mauricio Delbracio, Damien Kelly, Michael S. Brown, and Peyman Milanfar. Learning reduce defocus blur by realistically modeling dual-pixel data. In 2021
IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada,
October 10-17, 2021, pages 2269-2278. IEEE, 2021. 8
[5] Hadi Alzayer, Abdullah Abuolaim, Leung Chun Chan, Yang Yang, Ying Chen Lou, Jia-Bin
Huang, and Abhishek Kar. D Dual-camera defocus control by learning to refocus. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver. BC, Canada, June 17-24, 2023, pages 21488-211 214972 IEEF, 2023. 3.
[6] Jonathan T. Barron, Andrew Adams, Yichang Shih, and Carlos Hernández. Fast bilateral-space stereo for synthetic defocus. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 4466-4474 IEEE Computer Society, 2015.
[7] Shiveta Bhat and Deepika Koundal. Multi-focus image fusion techniques: a survey. Artif. Intell. 2021.
Rev., 54(8):57355555)
[8] Myungsub Choi, Hana Lee, and Hyong-Euk Lee. Exploring positional characteristics of dual- pixel data for camera autofocus. In /EEE/CVF International Conference on Computer Vision
ICCV 2023, Paris, France, October 1-6, 2023, pages 13112-13122. IEEE, 2023. 2
[9] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary. Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsu- pervised cross- loos-llnngual representation learning at scale. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pages 8451. Association for Computational Linguistics, 2020. 21
[10] Robert Correll. Digital SIR photography all-in-one for dummies. John Wiley & Sons, 2020. 3 [11] Yuning Cui, Wenqi Ren, and Alois Knoll. Omni-kernel network for image restoration. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Appli- cations of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 1426-1434
AAAI Press, 2024. 8, 22
[12] Harold Davis. Creative Composition: Digital Photography Tips and Techniques. John Wiley
and Sons, 2011. 3
[13] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference on Learning Representationss ICLR 2017, Toulon, France, April
24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. 10
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. 21
Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words:
[15] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes (voc) challenge. International Journal of Computer Vision 88(2):303-338, 2010. 24
11

---

[16] Kashish Galhotra, Azhar Ashraf, and Hiral Singh. Image refocusing-usiie unsupervised learning and depth estimation. In 2024 IEEE 1st Karachi Section Humanitarian Technology
Conference (KHI-HTC), pages 1-6. IEFE, 2024. 3
[17] Rahul Garg, Neal Wadhwa, Sameer Ansari, and Jonathan T Barron. Learning single camera depth estimation using dual-pixels. In Proceedings the IEEF/CVF International Conference on Computer Vision, pages 7628-7637, 2019.
[18] Aidan N. Gomez, Mengye Ren, Raquel Urtasun, and Roger B. Grosse. The reversible residual Backpropagation without storing activations. In Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach. CA, USA. pages
network:
2017. 10
[19] Rafael C Gonzalez and Richard E Woods. Digital Image Processing. Prentice Hall, 2002. 7
[20] Richard Hartley and Andrew Zisserman. Multiple view geometry computer vision. Cambridge
university press, 2003. 2
[21] Eric Jang. Shixiang Gu, and Ben Poole. Categorical reparamcterrization with gumbel-softmax. In Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April
24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. 23
[22] Jinbeum Jang, Yoonjong Yoo, Jongheon Kim, and Joonki Paik. Sensor-based focusiog system using multi-scale feature extraction and phase correlation matching. Sensors, 15(3):5747-
5762, 2015.
[23] Xin Jin and Jiawei Han. K-Means Clustering, pages 563-564. Springer US, Boston, MA, 2010.
10
[24] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9. 2015, Conference Track Proceedings, 2015. 7 Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 con- volutions. In Sany Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- DesaaBiinchi,, and Roman Garnett, editors, Advances in Neural Information Processing Sys- tems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 10236-10245, 2018. 21,
[25]
[26] Junyong Lee, Hyeongseok Son, Jaesung Rim, Sunghyun Cho, and Seungyong Lee. Iterative filter adaptive network for single image defocus deblurring. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2034-2042, 2021. 3,8 [27] Feiran Li. Heng Guo, Hiroaki Santo, Fumio Okura, and Yasuyuki Matsushita. Learning to synthesize photorealistic dual-pixel images from RGBD frames. In IEEE International Conference on Computational Photography, ICCP 2023, Madison, WI, USA, July 28-30, 2023, pages 1-11. IEEE, 2023. 7
[28] Pengwei Liang. Junjun Jiang, Xianming Liu, and Jiayi Ma. Bambnet: A blur-aware multi- branch network for dixel defocus deblurring. JEEF/CAA Journal of Automatica Sinica, 9():8788889 2022. 8
[29] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual
instruction tuning. 2023. 4
[30] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[31]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023.
[32] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations (ICLR), 2017.
[33] David G. Lowe. Object recognition from local scale-invariant features. In Proceedings of the International Conference on Computer Vision, Kerkyra, Corfu, Greece, September 20-25, 1999,
pages 1150-1157. IEEE Computer Society, 1999. 22
[34] Xintian Mao, Yiming Liu, Wei Shen, Qingli Li, and Yan Wang. Deep residual fourier transfor- mation for single image deblurring. CoRR. Abss/111.111
2021. 8, 21
[35] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. CoRR,
hts/1411.17844 2014. 3
[36] Ren Ng. Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz, and Pat Hanrahan. Light field photography with a hand-held plenopic camera. Technical Report CTSR 2005-02, CTSR,
01 2005. 3
[37] Liyunnn Pan, Shah Chowdhury, Richard Hartley, Miaomiao Liu, Hongguang Zhang, and Hong- dong Li. Dual pixel exploration: Simultaneous depth estimation and image restoration. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 2021. 7,8
12

---

[38] Liyuan Pan, Shah Chowdhury, Richard Hartley, Miaomiao Liu, Hongguang Zhang, and Hong- dong Li. Dual pixel exploration: Simultaneous depth estimation and image restoration. In IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2021, virtual, June 19-25,
2021, pages 4340-4349 Computer Vision Foundation IEEE, 2021.
[39] Liyuan Pan, Yuchao Dai, Miaomiao Liu, Fatih Porikli, and Quan Pan. Joint stereo video deblurring, scene flow estimation and moving object segmentation. IEEE Trans. Image Process., 29:1748-1761. 2020. 2
[40] Juewen Peng. Zhiguo Cao, Xianrui Luo, Hao Lu, Ke Xian, and Jianming Zhang. Bokehme: When neural rendering meets classical rendering. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 16262-16271. IEEE, 2022. 3.8
[41] Dominique Piché-Meunier, Yannick Hold-Geoffro, Jianming Zhang, and Françoiss Lalonde. Lens parameter estimation for realistic depth of field modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 499-508, 2023. 3 [42] Abhijith Punnappurath, Abdullah Abuolaim, Mahmoud Afiii and Michael S Brown. Mod- eling defocus-disparity in dual-pixel sensors. In 2020 IEEE International Conference on Compuiational Pholography (ICCP), pages 1-12. IEEE, 2020. 3
[43] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloé Rolland. Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross B. Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM 2: Segment anything in images and videos. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28,
2025. Open 2025. 2, 4
[44] Lingyan Ruan, Bin Chen, Jizhou Li, and Miu-Ling Lam. Learning to deblur using light field generated and real defocus images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 16283-16292. IEEE, 2022.
[45] Parikshit Sakurikar, Ishit Mehta, Vineeth N. Balasubramanian, and P. J. Narayanan. Refocusgan: Scene refocusing using single image. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14. 2018 Proceedings, Part IV, volume 1208 of Lecture Notes in Computer Science, pages 519-535. Springer, 2018 2,
[46] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewiez, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA. USA, June 13-19, 2020, pages 4937-4946. Computer Vision Foundation IEFE, 2020. 22
[47] Pedro H. P. Savarese. Learning identity mappings with residual gates. CoRR, 1..0166
2016.
[48] Mashhour Solh. Real-time focal stack compositing for handheld mobile cameras. In Charles A. Bouman and Ken D. Sauer, editors, Computational Imaging XII, part of the IS&T-SPIE Elec- tronic Imaging Symposium, San Francisco, California, USA, February 2, 2014, Proceedings, volume 9020 of SPIE Proceedings. page 90200Z. IS&T/SPIE, 2014. 3
[49] Hycongseok Son, Junyong Lee, Sunghyun Cho, and Seungyong Lee. Single image defocus deblurring using kernel-sharing parallel atrous convolutions. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021,
pages 2622-2630, IEEE, 2021. 8
[50] Mahmoud Tahmasebi, Saif Huq. Kevin Meehan, and Marion McAfee. Devsmnet: Double cost volume stereo matching network. Neurocomputing, 618: 129002, 2025. 4
[51] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural
networks, 05 2019. 21
[52] Neal Wadhwa, Rahul Garg, David E. Jacobs, Bryan E. Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Atuee
Jonathan T. Barron, Yael Pritch, and Mare Levoy. Synthetic depth-of-fied with a single-camera mobile phone. ACM Trans. Graph. 37(4);64, 2018. 2, [53] Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck. Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal. Subhojit Som, and Furu Wei. Image as foreign language: Beit pretraining for all vision and vision- -angual tasks. CoRR, abs/2208. 10442, 2022. 21
[54] Yingqian Wang, Jungang Yang, Yulan Guo, Chao Xiao, and Wei An. Selective light field refocusing for camera arrays using bokeh rendering and superresolution. IEEE Signal Processing Letters, 26:204 208, 2019. 3
13

---

[55] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing, (4)):600
612, 2004. 7
[56] Shumian Xin, Neal Wadhwa, Tianfan Xue, Jonathan T Barron, Pratul P Srinivasan, Jiawen Chen, loannis Gkioulekas, and Rahul Garg. Defocus map estimation and deblurring from a single dual-pixel image. In Proceedings of the IEEE/CVF International Conference Computer
Vision, pages 2228-2238 2021. 2
[57] Yan Yang. Liyuan Pan, Liu Liu, and Miaomiao Liu. K3DN; disparity- kernel estimation for dual-pixel defocus deblurring. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 13263-13272.
IEEE, 2023, 2. 7,
[58] Yan Yang. Liyuan Pan, Liu Liu, and Miaomiao Liu. K3dn: Disparity -wwre kernel estimation for dual-pixel defocus deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13263-13272, June 2023. 3,
[59] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuae Yang. Restormer: Efficient transformer for high-resolution image restoration. In IEEE/CVF Conference Computer Vision and Pultern Recognition, CVPR 2022, New Orleans,
LA, USA, June 18-24, 2022, pages 5718-5729. IEEE, 2022. 8
[60] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
5728-5739, 2022. 8
[61] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. In IEEE Conference Computer
Vision and Pattern Recognition (CVPR), pages 586-595, 2018. 7
[62] Yuxuan Zhang, Tianheng Cheng, Rui Hu, Lei Liu, Heng Liu, Longjin Ran, Xiaoxin Chen, Wenyu Liu, and Xinggang Wang. Evf-sam: Early vision-language fusion for text-prompted segment anything model. 2024. 2, 4, 6, 21
[63] Yi Zhang, Qixue Yang. Damon M. Chandler, and Xuanqin Mou. Reference-based multi-stage progressive restoration for multi-degraded images. IEEE Trans. Image Process., 33:4982-4999
2024. 21
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper has a clear abstract and introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations the future work section included the supplementary material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We state experimental details in Experiment section in the main paper and Supplementary Material A. We state which datasets used and provide references.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

reproducible, they Answer: [Yes]
Justification: the paper is accepted, the code and data will released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.cm cc/
oniooiicy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips.cc.o
wwwcc//ggg////eeeeeeeeeeeeeeeeee /ooeeSmmmm cioomisssion asionnolll (iionPolccy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We state all experimental details in Experiment section in the main paper and Supplementary Material A.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: ttttpa:/neeee ht////nurippps/ cc. sec/public/Eeee I ines?
Answer: [Yes]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
18

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
19

---

Researchers should communicate the details of the dataset/codeemmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy (https hactpp/seeerrppseccoo/oeeeeeeeee wwt.c///eeeene/////222. /LLM) for what should or should not be described.
20

---

Abstract of Appendix
This appendix provides more implementation details (Appendix A), details of curated DP image dataset (Appendix B), adaptation for refocusing functionality (Appendix C), comparison with existing methods (Appendix D), and more visualization results and analysis (Appendix E).
A More Implementation Details
Network Architecture. Our method adopts a 4-saage UNet-style architecture for progressive feature refinement. As mathematically formulated in Eq. 9 and Eq. of main paper, each stage has two
cascaded parameter-independenn modules: (i) disparity-aware feature convolution, followed by (ii) invertible block [nv(.) and Inv ((). For (i), we use the disparity feature F (f, to retrieve a kernel i. from kernel pool (K, J 1== and then a point-wise convolution [51] with K or K.
is employed for feature refinement to obtain U or U and the architecture of (ii) is elaborated in Table 7. Notably, we parameterize learnable matrix W directly its LU decomposition [25] for efficiency, i.e., W = PL(Q + diag(s)). where P is permutation matrix, L is a lower triangular matrix with ones on the diagonal, Q is upper triangular matrix with zeros on the diagonal, and is a vector.
Towards (-) and (-) in instructed SAM, we primarily adopt BEIT-3 [53] as Fm (-), which implements a multi-way Transformer architecture. The input text is tokenized using the XLM- Roberta Tokenizer [9]. Within each encoder block, image and text tokens are fused through cross- modal attention mechanisms and subsequently processed by separate Feed-Forwar Networks (FFNs). Following the ViT paradigm [14]. we extract the compact multimodal representation by taking the output of the [cls] token. Specifically, the original prompt encoder generates sparse embeddings of shape B P x D, where denotes the batch size, P is the number of input points or boxes, and D represents the embedding dimension. Following [62]. we replace these spatial prompts with the projected multimodal embeddings from the multimodal encoder (.)..). If there are point or box inputs, these are concatenated with an empty sparse embedding tensor and perform alignment with the image feature extracted from sam (-), subsequently fed into the mask decoder.
Table 7: Illustration of invertible block operations Ine(-) and In)=+(()) in our method. In the forward mapping, the input and output to each block are denoted as U² and U During the backward mapping, only Plus (+) and Multiply (c) needs to be inverted. d1. d2, Ø3 and Φ1 do not need to be inverted, which can be any neural networks.
Forward Operation Iun(-)
Backward Operation Ince (()
Specification
L/Q + ding(s)) denotes matrix by LL decomposition [25].
RD
0 = PL(Q (i)g
0141 = (Q + ding(s)) -P--L U011
and P is permutation matrix.
Uy Uj Splin(O")
U 1f Split((++))
Splis()) denotes the channdl-wise split.
U
U! 0 (xpppp (U)) (L)))
U = (U, 6, /// (xp
d dy dy and 6, can be any neural networkks.
R3
U
U. o oxpppp (((
U, (Ug+) (x())) (Up))
the multiply operation.
R4 U441 Coucad U )
Concal(U,, U!)
Concac()) (-) the channel-wiee concatenarion.
As mentioned in Table 6 of main paper, adopt two variants v1 and v2 to investigate the impact of different invertible blocks towards visual representation learning. Their architectures are illustrated in Table 8 and Table 9, respectively.
Table 8: The invertible block of variant vl.
Forward Operation
Backward Operation
Specification
L(Q + diag(s)) denotes the matrix by decomposttion [25], and P is permutation matrix.
R0
Ui+1_
PL(Q + d'
U' (Q + diag(s)) -e---------- u
Hyper-parameter Setting, Our Ldeb and Lreb both uses a combination of Multi- Charbonnier loss Lehar [63], Multi-Scale Edge loss Ledge [63], and Multi-Scale Frequency loss L Lireg [34], i. e., Creb Ldes Lehar d Ledge + Arey. We set A₁ = 5 10-2 and X2 1 x 10 Regarding
21

---

Table 9: The invertible block of variant 22.
Forward Operation
Backward Operation
Specification
L(Q + diag(s)) denotes the matrix by
R0
Ü'- Pl,(Q di
(++1 (Q + diag(s)) 1p--p--
I.U decomposition [25], and P is a permutation matrix.
R1
Ugg U0 (p))n(U")
U!t,, Uf*i Split((/+1))
Split() denotes the channel-wise split.
R2
U Ug + (UU)
U² U -=(UU=1))
P1. d2. Og and da can be any neural networks.
R3
uft! +y(U5"))
U² ug!1 (2((U
R4
U++1 Coneat(U ,,,")) )
U Concat(U,, U3)
Concat(:) () is the channel-wise concatenation.
other two loss supervision Acoe and Agrad, we set Acce 0.5 due to that a excessive large Avee would overwhelm the useful cues in F Finst learned from reblurring and deblurring task, and Agrad = 0.5 to reserve the figh-fryy information and sharp the edge in restored image.
Design of Gate Vector R. in Eq. 7. A computed in Eq. 8 of main paper,
F coc R FG Poent (Prgb (Finit)
(17)
where R serves as the gating vector to balance the learning between the vanilla disparity feature Fc and CoC-aligned feature Prgb(E (ins)).. Specifically, it formulated as,
R = Tanh + PPo(P feat (Prgb (rgb(B (i)))))),
(18)
where P)( and Pc(+) are two simple linear neural layers. We empirically observe that using the gate R could strengthen the gradient of Finat, and adaptively balance the contribution of F Finit and Fc to feature vector coo.
B Details of Curated DP image Dataset
In this paper. We present real-world DP dataset consisting of 10 high-quality pairs with resolution of 6720 x 4480 pixels, for refocusing evaluation. The DP image is captured by Canon EOS 5D MarkIV !. The camera sensor features two independent photodiodes embedded within each pixel, enabling phase-detection autofocus for rapid focusing. During image capture, the left and right sub-pixeh, combine their outputs to generate the final view. After collection, we use software Digital Photo Professional to split the DP view from the captured center one. In our dataset, the image pairs captured using aperture settings corresponding f/2.8, which results in the greatest DoF and thus most defocus blur. As shown in Figure 8, we select two objects 0r and O2 as focal point to form image pair (Bo1. Bo2), and each of pair B. contains a dual-pixel image pair (B. J, B..)). When performing evaluation, we take one dual-pixel image pair as the input, and regard another image (center view) the target image.
Notably, considering that when two objects are far apart, focusing on different objects may introduce noticeable misalignment of the captured scene. To address this issue, we adopt the following strategies during the captured process: (1) Restrict the distance between the two focus targets within a certain range to allow only minimal and acceptable shifts. (2) Manually crop and align the images when slight misalignment still occur. (3) First adopting classical image matching techniques [33, 46] for pixel matching, and then compute corresponding quantitative metrics. All the datasets will be released.
C Adaptation of Refocusing Functionality
We manually modify two SOTA methods on deblurring task, K3DN and Omni-Kernel [11], for refocusing adaption. For K3DN (architecturally similar to our approach), we replace our 4-level UNet with K3DN's backbone while preserving only RO (Table 7) to maintain essential reversibility. For Kernel that has more complex architecture, we simply concatenate disparity feature and DP image features in a channel-wise manner, and take them as input for target image approximation.
'httss//wwwwccamon. www.amoooooooooooo...o wwwps//wwwn....oooooooeee wo..u//eamma///oooSaaaaaaa
httpss//app.ss.iimmgggee wwwpa/iiinnggssannooaooa www../aaw.w.app//////....a.aaaaa http://hhhhd.pphhhhmm////ooo
22

---

Bozi
Buzy
Zoom ta
Figure 8: Three examples of our collected DP image pairs. The red and green circles indicate the focal points, while the orange bounding box highlight the zoomed-in regions.
D Comparison with Existing methods
Compared to existing beblur-and-rebie frameworks (e.g., RefocusGAN), our key improvements are:
Reversible Block with Shared Kernel: We establish an invertible connection between de- blurring and reblurring with shared kernel learning, which strictly follows the mathematical formulation in Eq. 1I while achieving better deblurring and reblurring performance.
Unified Network Architecture: We reuse one network parameter to achieve refocusing task (deblur-and- significantly reducing the total number of parameters and improving training efficiency.
To highlight our contribution, explain the necessity of invertibility in what follows. The network's invertibility ensures that reblurring and deblurring can be jointly trained while sharing the blur kernel, better establishing one-to-one correspondence between disparity feature and blur kernel. Intuitively, an effective disparity- -wware blur kernel should be capable of both restoring an image (deblurring) and blurring it (reblurring). This paves crucial path for the subsequent refocusing, enabling accurate kernel retrieval for defocus control by modifying the disparity (refer to Figure/ 7 for illustration). When empirically adopt traditional encoder-decode architecture, the correspondence between the kernel and disparity becomes difficult to learn effectively, leading to unintended control effects.
E
More Visualization Results and Analysis
Blurry Image
Kernel Selection Preference
Blurry Image B
Kernel Selection Preference
Aac Weig
Figure 9: Kernel selection preference when performing refocusing from all-in-focus image. Accumulated Weight refers to assigned weight score of each blur kernel K,, accumulated across the layer in invertible network.
Kernel Setting, We visualize the kernel selection preference of two cases in Figure 9 for deeper behavior revelation. It can be observed that most kernels has a smaller accumulated weight while only a small part of kernels are activated for blur indication. In these two cases, the commonly used kernels lies in index (56, 57, 65, 73, 86), and kernels with index (3, 17, 22, 80) are sample- specific. We also employ the hard selection of blur kernel, which can be differentiably achieved by Gunbel-SSfttax [21]. However, we empirically observe that hard selection achieves the suboptimal deblurring and reblurring performance. We posit that this issue arises because hard selection induces
23

---

Table 10: Performance with different number of pre-defined kernel (K, / I-
Table 11:
Comparison of refocusing perfor-
mance on self-collected dataset.
32
48
96
128
Variants
PSNRT
SSIM
MAE L LPIPS.
Deblurring Performance
PSNR
25.92 26.14 26.89 26.82
Omni- Kemel
19.50
0.690
0.56
32,94
K3DN
19,52
0.698
0.52
31.91
Reblurring Performance PSNR 28.32 28.46 29.18 29.18
Ours
19.68
0.707
0.52
30.87
an initial bias toward specific kernels during the early stages of training, which subsequently hinders the optimization process for kernel selection. Additionally, we further investigate how different number of pre-defined kernels affect model performance. As shown in Table 10, Both insufficient and excessive numbers of kernels impair model performance. Fewer kernels cannot adequately cover the necessary blur range across the dataset, while too many introduce excessive trainable parameters that compromise model robustness.
Blurred Image
Omni-Kernel
K3DN
Zoom
Ours
Zoom
Figure 10: Visualization comparison of refocusing on self-collected DP image pairs. The red point indicates the focal object, and the yellow bounding box highlights the zoomed-in regions.
More Refocusing Results on Self-Collected DP image pair. We give the quantitative results on our self-collected dataset in Table 11, our method achieves the consistent superior results across all the metries. As shown in Figure 10, we take the first case as the example, the sheep is focused initially, and our goal is to transfer the focal point from sheep to its right bottle. We compare method with two SOTA baselines K3DN and Omni-Kernel, the comparison results show that our refocusing result more scene-realistie.
Evaluation on Mask Quality. To evaluate the mask qual- ity from our adopted SAM, we manually annotate two objects for each image of our collected images as the groundtruth. We compare the average mloU metrics [15] on different version. From Table 12, we observe that early fusion achieves the clear-cut performance compared with late fusion. Furthermore, our structured semplate-based prompt achieves better segmentation result.
Table 12: Comparison of segmentation performance with different prompts. and denote the early-fusion and late- fusion version, respectively. Ours de- notes the template- based prompt.
3 3-Clickt Vanilla Prompt"
Ours"
Ours!
Performance Variation with Different Blur Degree. Be- sides, we also exploit instructed SAM segmentation results towards the image inputs with different degree of defocus blur and our restored image í. We sample ten groups of sample from DP5K-tess -est and each of which has 5 cases with different f-umbeer 1.8, 2.8, 4.0, and 5.6. We use the SAM with early fusion to generate the mask, and mamually annotate the region with interest as the ground-truth. As can be seen from Table 13, we empirically observe that our used SAM could functions well when handling the image with large defocus blur (i.e., f-1.8). and using our restored image the SAM input could achieves the best result. Beyond that, with the blur degree decreasing, the segmentation result becomes saturated.
93.6
87.5
81.6
91.8
Table 13: Segmentation performance with different degrees of defocus blur.
f-1.8
f-2.8
f-4.0
f-5.6
i
85.8
90.3
90.8
91.1
91.8
24


Reviews:
[{"review_id": "xLCV8BoS81", "reviewer": "Reviewer_TUF9", "summary": "This paper proposes UiD, a User-Instructed defocus control framework that enables flexible Depth of Field (DoF) manipulation in images captured by dual-pixel (DP) sensors. The system allows users to specify refocusing regions via text, box, or point prompts, which are converted to masks using SAM. UiD features an invertible deblurring-reblurring architecture that learns spatially varying defocus kernels and disparity features from DP pairs, establishing a closed-form mapping between blur and disparity. This enables both restoration of all-in-focus images from blurred inputs and controlled reblurring for user-specified refocusing. The approach is validated through extensive experiments on standard datasets and a new self-collected DP dataset, demonstrating superior flexibility and quality in DoF manipulation.", "questions": "The framework presented in this paper lacks originality, and in order to demonstrate the novelty of the article, it is necessary to elaborate on the differences from existing methods as well as the implementation details that are fully elaborated.\nAlthough some quantitative results are provided, the qualitative results are not sufficiently presented to visually assess the performance of the UiD framework. In addition, the necessary ablation experiments and detailed implementation details should be added to show the effectiveness of the individual components.\nThere are multiple formatting errors in the article that lead to overall confusion, and the entire text needs to be scrutinized and corrected to improve readability.", "limitations": "The author of this article has fully elaborated on the advantages of the proposed method, but lacks the necessary exploration of its limitations and any potential negative social impacts, which is not conducive to the development of subsequent work. Therefore, it is necessary to supplement the content on limitations to provide direction for future research.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "The strengths of this article are described as follows:\nProposes and validates a mathematically-driven invertible deblurring-reblurring framework that achieves SOTA deblurring performance while significantly reducing computational costs, without relying on depth annotations.\nBeyond SAM-powered interactive refocusing, the work contributes a novel real-world DP dataset addressing data scarcity and demonstrates robust out-of-distribution generalization.\n\nThe weaknesses of this article are described as follows: \nThe core framework heavily relies on existing \"deblur-and-reblur\" paradigms (e.g., RefocusGAN, DC²) without introducing significant methodological advancements. Furthermore, the user interaction mechanism depends entirely on off-the-shelf SAM for mask generation, with no task-specific modifications. This unmodified pipeline integration substantially dilutes the claimed contribution of \"user-instructed\" control.\nThe paper claims to support text, box, and point prompts for specifying refocusing regions, yet fails to provide essential implementation details (e.g., text-to-mask conversion, point/box coordinate handling in defocused images) or experimental validation of these interactive features. \nThe self-collected DP dataset lacks a detailed description, including the size and characteristics of the dataset. In addition, ablation studies are insufficient, such as the full range of different variants of the loss function, and exploration of the weights λ is also missing.\nThere are also many formatting errors in the paper, for example, the expression  in Eq. (13) does not match the above, there is a missing period at the end of the title of Table 2, there is a spelling error in “caluclate”, and there is a redundant period in “Figure. 6”, and so on.", "was_revised": true, "final_justification": "The rebuttal have addressed my main concerns, including novelty, method details, hyperparameter settings. Compared to existing deblur-and-reblur frameworks (e.g., RefocusGAN, $ \\text{DC}^2 $), the proposed method introduces the Reversible Block with Shared Kernel that strictly follows the mathematical formuation (Eq.10), and implements Unified Network Architecture to reduce the total number of parameters and improving training efficiency. This highlights the novelty. Thus, I update my score to borderline accept.", "rebuttal": "We sincerely thank your comments, and we give point-to-point below.\n\n---\n\n## 1. Novelty Highlight\n\nPlease kindly note lines 56–68. The core innovation of our work lies in the adoption of a **unified invertible network** that decomposes the defocus control task into **reversible deblurring and reblurring subtasks**. \nSince deblurring and reblurring are inverse processes, we establish an essential connection between them by constraining these two subtasks to **share the same blur kernel $K$ but with different form** (as formulated in Eq.10). Specifically:\n- The **forward process** (deblurring) uses disparity feature to retrieve kernel $ K $ for convolution.\n- The **reverse process** (reblurring) uses disparity feature to retrieve kernel $ K $, and adopt its inverse kernel $ K^{-1} = \\mathcal{F}^{-1}(\\frac{1}{\\mathcal{F}(K)})$ for convolution, where $\\mathcal{F}$ and $\\mathcal{F}^{-1}$ denote the fourier transformation and its inverse transformation, respectively.\n\nWe also employ the invertible block (elaborated in suppl Tab.1) to better refine the representation. These two processes are trained **collaboratively** using loss in Eq.(13) .\n\nCompared to existing deblur-and-reblur frameworks (e.g., RefocusGAN, $ \\text{DC}^2 $), our key improvements are:\n1. **Reversible Block with Shared Kernel**: We establish an invertible connection between deblurring and reblurring with shared kernel learning, which strictly follows the mathematical formuation (Eq.10).\n2. **Unified Network Architecture**: We reuse one network parameter to achieve refocusing task (deblur-and-reblur), significantly reducing the total number of parameters and improving training efficiency.\n\n---\n\n## 2. Off-the-Shelf SAM\n\n1. **Note that** our method **does not require masks during training**. Masks are only provided at test time, where the user inputs a binary mask (single-channel, $ H $ $\\times$ $ W $, the same size with input image) to specify the region of interest for refocusing. Therefore, **no task-specific modification** would not affect the dynamics and robustness of our model training (train only once).\n2. The language-based SAM, when guided by specially designed prompts, can generate sufficiently robust masks even on deblurred images. Given that the original SAM model was trained with diverse data augmentation, it exhibits inherent robustness to mild noise or blur. To further improve mask quality in our implementation:\n   - For region of interest, we use **language-style prompts**: *number + object color + object name + spatial position* (e.g., a brown dog on the chair).\n   - For those still imperfect masks, we further apply **post-processing techniques** such as dilation or erosion operations for refinement.\n\nTo evaluate the mask quality from our adopted SAM, we manually annotate two objects for each image of our collected images as the groundtruth. We compare their average IOU metrics,\n|3-click | language(only with object name)  |   language (number + object color + object name + spatial position)    |\n|------------|-----------|---------|\n| 93.6      | 91.8     | 87.5   |\n\nWe acknowledge that the quality of refocusing results may be influenced by the quality input mask, we would provide the failure case in our final version.\n\n---\n\n## 3. Implementation Details for SAM\n\nFor **language-based mask generation**:\n- The LLM tokenizer converts user textual prompts into tokens.\n- The global token `[seg]` computes similarity with image features to generate the segmentation mask.\n\nFor **click-based interactive mask generation** (point or box inputs):\n- Each click coordinate is transformed into a **heatmap** (the region with click has large score).\n- This heatmap is fused with the previously generated mask to iteratively refine the segmentation result.\nMore click would further iteratively improve the quality of the generated mask. \n\nWe will provide further technical details in the appendix of the final submission.\n\n---\n\n\n## 4. Detailed Descriptions of Self-Collected DP Dataset\n\n\nPlease refer to Appendix. Additionally, our self-collected DP dataset consists of images with a resolution of **6720 × 4480** pixels. Additional characteristics will be added in the supplementary material of final version.\n\n---\n\n\n## 5. Hyperparameter Settings\n\nThanks for your suggestion. Considering the reversible nature of deblurring and reblurring, we assume **equal importance** to the deblurring loss \n$ L_{\\text{deb}} $ and reblurring loss $ L_{\\text{reb}} $. The gradient regularization term $ \\lambda_{grad}$ is a widely used auxiliary loss in image restoration tasks [1], and is not a primary contribution of our work; hence, we did not emphasize its analysis initially.\n\nIn response to your suggestion, we conduct additional ablation studies on the impact of different hyperparameters. Results on PSNR (dB) are summarized below:\n\n### Ablation on $ \\lambda_{\\text{grad}} $:\n| $ \\lambda_{\\text{grad}} $ | deblur | reblur |\n|----------------------------|-----------|---------|\n| 0.3                        | 26.84     | 29.11   |\n| 0.5                        | 26.89     | 29.18   |\n| 1.0                        | 26.82     | 29.13   |\n\n### Ablation on $ \\lambda_{coc} $:\n| $ \\lambda_{coc} $ | deblur | reblur    |\n|-------------|-----------|---------|\n| 0.3         | 26.86     | 29.16   |\n| 0.5         | 26.89     | 29.18   |\n| 0.7         | 26.74     | 28.86   |\n| 1.0         | 26.73     | 28.86   |\n\nWe will include these ablation studies in the experimental section of the final manuscript.\n\n---\n\n## 6. Formatting Errors\n\nWe appreciate your feedback and will carefully revise all formatting errors in the manuscript accordingly.\n\n\n## Reference:\n[1] Structure-Preserving Super Resolution with Gradient Guidance, CVPR 2020"}, {"review_id": "o4RS7DAjDp", "reviewer": "Reviewer_zAEY", "summary": "The paper introduces User-instructed Disparity-aware Defocus control, a framework that lets everyday photographers refocus still images after capture by simply pointing, boxing, or describing the region they want sharp. Leveraging the tiny stereo baseline of dual-pixel sensors found in many DSLR-style and smartphone cameras, UiD learns an invertible network that couples deblurring with reblurring in a single model. During training, it establishes a disparity-aware feature space and a set of spatially varying blur kernels so that the same kernels can be inverted to go from blurred to sharp or vice-versa. At test time, a user-prompted mask modulates the learned disparity features, enabling seamless refocusing that keeps selected regions crisp while realistically blurring others. Extensive experiments on public DP datasets and a new real-world DP benchmark show UiD achieves state-of-the-art deblurring quality, flexible and visually convincing refocus effects, and lower computation than prior DP or single-image methods.", "questions": "1. The paper’s main practical limitation is its DP-only scope. Showing either positive evidence of transfer or a principled explanation of the failure modes would greatly improve confidence that the approach is not narrowly engineered. If possible, could you show the results on a more general stereo dataset with larger disparities?\n\n2. For an interactive refocus tool the user experience hinges on latency. Concrete numbers would substantiate the “smallest compute budget” claim. The paper highlights low FLOPs, but latency on mobile hardware is not reported. Can you provide inference time on smartphone SoC or Apple CPU?", "limitations": "\"yes\"\nHowever, the main text does not include a separate “Limitations” section; instead, potential improvements (such as the application of a diffusion framework in future research) are indirectly addressed in the Conclusion/Future Work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\n1. UiD accepts text, box, or point prompts and turns them into masks with Segment Anything, so anyone can selectively keep subjects sharp while blurring the rest after capture—no depth maps or manual masking needed. \n2. Its invertible disparity-aware network couples deblurring and reblurring with shared spatially varying kernels, delivering state-of-the-art PSNR/SSIM on DP5K and DPD-blur while using the smallest compute budget among dual-pixel methods.\n\n[Weaknesses]\n1. The paper’s main weakness is that its contribution is confined to dual-pixel imagery. There are also deblurring and defocus models [1, 2, 3, 4] that operate on stereo images, which are closely related. Whereas dual-pixel sensors supply only an ultra-small parallax of roughly ±0.5 px, a stereo camera can adjust its baseline to achieve disparities ranging from tens to hundreds of pixels, an advantage that improves both depth accuracy and motion estimation. While the paper rightly targets the typical mobile-device scenario, additional analysis is needed before the method can be extended to settings such as robotics or autonomous driving.\n\n2. The method turns user clicks/boxes into masks with Segment Anything; any over- or under-segmentation directly skews the disparity modulation and produces visible artifacts, yet the paper offers no quantitative analysis of this failure mode. \n\n[1] Pan, Liyuan, et al. \"Joint stereo video deblurring, scene flow estimation and moving object segmentation.\" IEEE Transactions on Image Processing 29 (2019): 1748-1761.\n[2] Zhou, Shangchen, et al. \"Davanet: Stereo deblurring with view aggregation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n[3] Sellent, Anita, Carsten Rother, and Stefan Roth. \"Stereo video deblurring.\" Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14. Springer International Publishing, 2016.", "was_revised": true, "final_justification": "I have thoroughly reviewed the comments from all reviewers and the authors' rebuttal. I appreciate the authors for presenting various experimental results in such a short period, which has addressed most of my concerns.", "rebuttal": "---\n\n## 1.Confined to Dual-Pixel imagery\n\nOur method is primarily designed for defocus control in **static scenes**. It is worth noting that our focus is not limited to deblurring and reblurring separately, but rather unifying both processes to serve the purpose of refocusing. The works you mentioned, such as [1, 2], primarily address motion deblurring, which is **indeed a different task** from ours.  It is universally known that\nthe crux of refocusing lies in accurately estimating the target disparity. In theory, our method can be extended from dual-pixel to stereo matching, as the disparity estimation network ($G$ network, mentioned in line 155) in our approach essentially functions as a stereo matching network. This network shares the similar architecture as those in articles [3, 4] without pretraining, and this structural prior theoretically would enable it to cover the larger disparities generated by stereo data. Considering that currently there is no dataset specifically designed for stereo image refocusing, we will consider optimizing this aspect in our future work.\n\n## 2.Mobile Efficiency\n\nThank you for your question. Currently, we mainly focus on algorithm design, and have not yet supported deployment on mobile devices. We recorded the average inference time on a single A6000 GPU using our self-collected dual-pixel image pair dataset, with an image resolution of 1152 × 1792. The average time is SAM + refocusing = 0.45s + 0.13s = 0.58s.\n\n## 3.Evaluation on SAM\n\nThe language-based SAM, when guided by specially designed prompts, can generate sufficiently robust masks even on deblurred images. Given that the original SAM model was trained with diverse data augmentation, it exhibits inherent robustness to mild noise or blur. To further improve mask quality in our implementation:\n   - For region of interest, we use **language-style prompts**: *number + object color + object name + spatial position* (e.g., a brown dog on the chair).\n   - For those still imperfect masks, we further apply **post-processing techniques** such as dilation or erosion operations for refinement.\n\nTo evaluate the mask quality from our adopted SAM, we manually annotate two objects for each image of our collected images as the groundtruth. We compare their average IOU metrics,\n|3-click (point and box) | language(only with object name)  |   language (number + object color + object name + spatial position)    |\n|------------|-----------|---------|\n| 93.6      | 91.8     | 87.5   |\n\n## 4. Limitation Section\nThanks. We would add the limitation section and potential improvements of diffusion framework in our final version.\n\n## Reference\n[1] Joint Stereo Video Deblurring, Scene Flow Estimation and Moving Object Segmentation, TIP 2019\n\n[2] Stereo Video Deblurring, ECCV 2016\n\n[3] Hierarchical Neural Architecture Search for Deep Stereo Matching, NeurIPS 20\n\n[4] Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration, CVPR 21"}, {"review_id": "717vtvpXmt", "reviewer": "Reviewer_K4dV", "summary": "The authors  present UiD, a novel framework for user-instructed depth-of-field (DoF) manipulation. Users can use the system's point, box, or text prompts to refocus images. The main concept is to use disparity and defocus cues taken from DP image pairs to jointly model invertible deblurring and reblurring processes. Realistic refocusing using user-specified masks derived from SAM is made possible by the method's estimation of disparity-aware defocus kernels. When compared to multiple baselines, the paper shows better results in both synthetic refocusing and defocus deblurring.", "questions": "- What happend if the mask produced by SAM is imprecise or unclear? Does the system generate invalid defocus regions or does it degrade gracefully?\n\n- How robust is your method to different DP hardware calibrations or disparities with high noise?\n\n- What is the need for invertibility? When compared to standard encoding-decoding, can you measure the improvement in results?\n\n- Is the method fragile to segmentation boundary errors or small masks?\n\n- Is performance degraded with imprecise or noisy user prompts?", "limitations": "- Assumes access to DP sensor data — not all phones/cameras offer this.\n- Refocusing depends on SAM segmentation, but SAM is not evaluated as part of the system.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- Novel formulation: Treats refocusing as an invertible problem that connects deblurring and reblurring in a mathematically grounded framework.\n- Dual-pixel leverage: Effectively utilizes stereo disparity from DP sensors, offering better depth-aware refocusing.\n- User control: Enables intuitive control using natural prompts, supported by prompt-based segmentation model.\n\nWeaknesses:\n\n- The framework requires dual-pixel (DP) input, which restricts deployment to DP-enabled cameras. It is unclear how the method performs on monocular or stereo-only inputs.\n- UiD use SAM for segmentation masks but does not evaluate the impact of mask quality or conduct robustness tests on prompt errors or segmentation failures.\n- While the method emphasizes user control and artistic intent, it lacks perceptual or user experience studies to validate subjective quality or usability.", "was_revised": true, "final_justification": "I  would like to thank the authors for their response to my questions, and I maintain my initial rating.", "rebuttal": "We sincerely thank your comments, and we give point-to-point below.\n\n## 1. Consideration on Monocular or Stereo-only Inputs\nThanks. As the adopted disparity estimation network $G$ in our approach shares similar architecture with [1, 2] (stereo setting), therefore its structural prior theorectically could handle the stereo-only inputs that shares similar pattern with DP inputs. Regarding monocular setting that hard to estimate depth or disparity, I think a possible solution is to leverage a series of powerful pretrained model for depth or disparity, such as DepthAnything [3]. This can be further discussed and explored in our future work. \n\n\n## 2. Evaluation on SAM\n\nThe language-based SAM, when guided by specially designed prompts, can generate sufficiently robust masks even on deblurred images. Given that the original SAM model was trained with diverse data augmentation, it exhibits inherent robustness to mild noise or blur. To further improve mask quality in our implementation:\n   - For region of interest, we use **language-style prompts**: *number + object color + object name + spatial position* (e.g., a brown dog on the chair).\n   - For those still imperfect masks, we further apply **post-processing techniques** such as dilation or erosion operations for refinement.\n\nTo evaluate the mask quality from our adopted SAM, we manually annotate two objects for each image of our collected images as the groundtruth. We compare their average IOU metrics,\n|3-click | language(only with object name)  |   language (number + object color + object name + spatial position)    |\n|------------|-----------|---------|\n| 93.6      | 91.8     | 87.5   |\n\nWe would provide the failure visualization case in our final version.\n\n## 3.User Experience Studies.\nThanks for your suggestion. We would include this consideration in our final version.\n\n## 4. What happend if the mask produced by SAM is imprecise or unclear? Does the system generate invalid defocus regions or does it degrade gracefully?\nIt would degrade gracefully.\n\n## 5. How robust is your method to different DP hardware calibrations or disparities with high noise?\nOur method leverages the structural prior of the stereo matching network $G$ to automatically estimate disparity from the DP image pair (Eq. 7), guided by both deblurring and reblurring losses. **This dual supervision encourages the model to focus on the downstream task**, thereby enhancing its robustness to inherent noise.\n\n## 6. What is the need for invertibility? When compared to standard encoding-decoding, can you measure the improvement in results?\nThe network's invertibility ensures that reblurring and deblurring can be jointly trained while sharing the blur kernel $K$, better establishing **one-to-one correspondence between disparity feature and blur kernel $K$**. Intuitively, an **effective disparity-aware blur kernel** should be capable of both restoring an image (deblurring) and blurring it (reblurring). This paves a crucial path for the subsequent refocusing, enabling accurate kernel retrieval for defocus control by modifying the disparity (refer to Fig. 7 for illustration). When we empiricially adopt a traditional encoder-decoder architecture, the correspondence between the kernel and disparity becomes difficult to learn effectively, leading to unintended control effects (if possible, we would provide results in the rolling stage.).\n\n## 7. Is the method fragile to segmentation boundary errors or small masks?\nSegmentation boundary errors would affect the edges of the refocused image, but the results generally follow reasonableness; Our model exhibits good robustness to small masks.\n\n## 8. Is performance degraded with imprecise or noisy user prompts?\nYes. SAM may generate imprecise masks in response to noisy user prompts, which could have a certain impact on our refocusing process.\n \n## Reference:\n[1] Hierarchical Neural Architecture Search for Deep Stereo Matching, NeurIPS 20\n\n[2] Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration, CVPR 21\n\n[3] Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data, CVPR 24."}, {"review_id": "vgdEG8Ldtg", "reviewer": "Reviewer_2GJ1", "summary": "This paper introduces the User-Instructed DoF Control (UID) framework, a novel method for interactive depth-of-field manipulation in out-of-focus (OOF) images. The core innovation is enabling users to specify a desired in-focus region by leveraging the Segment Anything Model (SAM) for intuitive object selection. To accurately estimate the spatially varying blur, the framework utilizes dual-pixel data, which provides crucial sub-pixel disparity information for depth inference. The proposed method unifies two related tasks—OOF deblurring and all-in-focus generation—into a cohesive two-stage architecture that shares a common latent representation, allowing for efficient and effective refocusing and image restoration. Experiments validate the framework's design, demonstrating the significant benefits of dual-pixel inputs and the joint-task approach.", "questions": "#### **Regarding Clarity & Methodology**\n\n1.  Could the authors please provide a revised version of Figure 2? The current diagram is difficult to parse due to notational inconsistencies and an unclear depiction of the framework's data flow.\n2.  What is the specific rationale for using a learned decoder over a more interpretable, physics-based Point Spread Function (PSF) model? Could the authors demonstrate or elaborate on the distinct advantages of their chosen approach?\n3.  Could the authors elaborate on the design rationale for their architectural components? For instance, what motivated the choice of specific modules over other common alternatives in the literature?\n\n#### **Regarding Robustness & Limitations**\n\n4.  How robust is the framework to challenging inputs? We ask the authors to provide analysis on failure cases, specifically regarding (a) inaccurate segmentations from SAM, and (b) images with very large amounts of defocus blur.\n5.  Could the authors add the \"Limitations\" section as indicated in their submission checklist? This section should detail the method's specific failure modes and boundary conditions.\n\n#### **Regarding Submission Compliance**\n\n6.  Could the authors provide the missing details on computational resources (GPU model, training/inference times) as stated in the checklist? This information is essential for reproducibility.\n\n7. Could the author update the checklist for \"Open access to data and code\".", "limitations": "No, stated in Weakness 4.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strength\n1. **Novel User Interaction:** The integration of the Segment Anything Model (SAM) for defining the region of interest is a modern and highly effective approach to user interaction. It significantly improves usability over traditional methods like manual scribbles or mask drawing.\n2. **Elegant Unified Framework:** The two-stage architecture that jointly handles OOF deblurring and refocusing is a logical and efficient design. Sharing latent information between these highly related tasks is a promising direction.\n\n\n### Weakness\n1. **Clarity, Notation, and Figure Legibility:** The paper's clarity is significantly hampered by notational inconsistencies and confusing figures. For instance, there is a discrepancy between the notation in the equations ($F_{int}$ in Eq. 5, 6, 7) and the architecture diagram ($F_{init}$ in Figure 2). Furthermore, the annotations used to describe the methodology in Section 3.1 do not align with those presented in Figure 2. These issues make it exceptionally difficult for the reader to follow the model's architecture and data flow.\n\n2. **Justification of Core Methodology (Decoder vs. Physics-Based Model):** The paper's methodology relies on a learned decoder for the final refocusing step, but the motivation for this choice over a more traditional, physics-based approach is unclear. Given that the framework estimates disparity, this information could be used to model a Point Spread Function (PSF) for refocusing. A PSF-based method offers greater interpretability and may be less prone to the hallucination of textures—a known risk in purely generative models.\n\n3. **Unclear Design Rationale:** The manuscript lacks a clear narrative justifying its architectural and component-level design choices. While the ablation studies demonstrate that each component contributes to the final performance, the paper does not adequately explain *why* these specific components were chosen over other well-established alternatives in the literature.\n\n4. **Absence of a Limitations Section:** The paper fails to include a discussion of its limitations. The authors should discuss potential failure cases or negative impacts, such as performance on scenes with extreme blur, subjects where SAM segmentation might be ambiguous, or the method's sensitivity to the quality of the dual-pixel input.\n\n5. **Inadequate Citation Practices:** Citations for baseline methods are missing from several tables and figure captions. This forces the reader to search through the text to identify the compared methods, significantly harming readability. For clarity and proper academic credit, all baseline methods must be cited directly within those elements.\n\n6. **Fairness of Experimental Comparison:** The experimental evaluation compares against single-image (SI) and dual-pixel (DP) defocus deblurring methods. Comparing a method that utilizes dual-pixel (DP) data against those that use only a single image is not a fair assessment of state-of-the-art performance. The paper's central claims would be much stronger if the primary comparison was against other existing methods that also leverage dual-pixel data. Simply cite those SI methods properly in related work and focus more on the ablations studies.\n\nThere are significant and concerning discrepancies between the manuscript and the authors' responses in the NeurIPS checklist. This raises serious questions about the submission process.\n1.  **On Limitations:** Checklist item 2 asserts that limitations are addressed in the future work in the supplementary material. However, the manuscript lacks a dedicated limitations section. The \"Future Work\" section the main paper is not an adequate substitute for a critical analysis of the current method's weaknesses and failure modes.\n2.  **On Data and Code Access:** Checklist item 5 response regarding the public availability of code and data appears to be inconsistent. The author should stated \"No\" if they wish to published after acceptance.\n3.  **On Computational Resources:** Checklist item 8, the submission does not detail the computational resources used. While implementation details are provided, information regarding the hardware (e.g., GPU model), training times, and inference speeds is absent, which is critical for assessing reproducibility and efficiency.", "was_revised": true, "final_justification": "I would like to confirm that the following points have been effectively addressed and have greatly strengthened the manuscript:\n\n* **Methodology & Rationale:** The explanation of the invertible network and how it functions within a PSF-based paradigm was a crucial clarification. This resolved my primary misunderstanding regarding the technical foundation of the method. It is now clear that the core contribution is the insightful synthesis of these techniques, motivated by the inverse relationship between deblurring and reblurring. \n\n* **Robustness & Reproducibility:** The new quantitative experiments on SAM's performance are excellent and provide convincing evidence of the framework's robustness. Furthermore, providing the specific computational details has fully addressed my concerns about reproducibility.\n\n* **Promised Revisions:** I acknowledge the commitment to revise the paper's clarity, add the missing citations, include a dedicated limitations section, and update the submission checklist. I trust these changes will be incorporated into the final version.", "rebuttal": "## 1.Clarity and Notation. \nThank you for the feedback. We will revise the data flow and notation in the final version to improve readability.\n\n## 2.Rationale of Method. \nActually, **our network still adheres to the PSF-based framework.** The refocusing process does not employ an additional decoder; instead, it follows the workflow indicated by the **black arrows** and **orange arrows** in Fig. 2. \n\n**black arrows:** Given a deblurred image and a user-specified mask of the region of interest, the modified disparity is first estimated for blur kernel retrieval. \n\n**orange arrows:** Then, blur kernel is used to achieve spatially-variant defocus control using inverse mapping of our invertible network.\n\n## 3.Architectural and Component-level Explanation. \n\nThank you for your suggestion. Here, we clarify the design motivation behind our architecture and core components. \n**Architectural Explanation:**\nIn essence, our goal is to design a framework for refocusing, a process can be typically decomposed into two steps: (1) first deblurring to obtain a sharp image, and (2) then reblurring specific regions according to user needs to achieve the desired defocus effect. \n\nMathematically, deblurring process can be modeled as  $I_{s}$ = $I_{b}$ $\\circledast$ $K$, where $I_{b}$ is the blur image, and $I_{s}$ is the restored image. According to the **convolutional theorm**, we have $\\mathcal{F}(I_{s})$ = $\\mathcal{F}(I_{b})$ $\\odot$ $\\mathcal{F}(K)$. Its inverse process can be represented as a reblurring process,  $I_{b}$ = $I_{s}$ $\\circledast$ $\\mathcal{F}^{-1}(\\frac{1}{\\mathcal{F}(K)})$ =  $I_{s}$ $\\circledast$ $K^{-1}$.\n\nWe observe that the blur kernels $K$ and $K^{-1}$ used in these two processes satisfy a reversibility property. This insight motivates us to learn an invertible network with reversible kernels. **Intuitively, an effective blur kernel should be capable of both restoring an image (deblurring) and blurring it (reblurring)**. Therefore, we supervise the network's joint learning using both the reblurring loss and the deblurring loss. By co-training these two tasks within a single network, **we achieve refocusing efficiently—reducing training time and lowering the model parameter count**. \n\n**Component-level Explanation:**\n As illustrated in Fig. 2, our model follows a common architecture comprising an encoder ($F_{e}$), a refine component, and a decoder ($F_{d}$). Our primary contribution lies within the refine module, which consists of:\n\n**(i) Disparity-aware Feature Convolution:** This component retrieves the corresponding kernel $K$ from the kernel dictionary using the estimated disparity. The forward process (deblurring) uses kernel $K$ for modulation, while the inverse process (reblurring) employs its corresponding inverse kernel  $K^{-1}$ for modulation.\n\n**(ii) Invertible Blocks:**  **To further enhance the representational capacity of network features and strengthen the network's inherent reversibility**, we design invertible blocks, denoted as $Inv(·)$ and $Inv^{-1}(·)$, which has been elaborated in Tab.1 in the appendix. This design ensures strict invertibility of the network (enabling lossless, reversible information flow [3]), where the **Jacobian matrix** of the output with respect to the input is invertible.\n\n## 4.Absence of a Limitations Section. \nThank you for the suggestion. We will include a limitation section in the final version.\n## 5.Citation.  \nThank you. We will add the missing citations in the table.\n\n## 6.Experimental Comparison. \nRegarding this point, our experimental table setup follows the conventions established in most prior works [1, 2]. We will revise the formatting according to your recommendations.\n\n## 6. Could the authors please provide a revised version of Figure 2? The current diagram is difficult to parse due to notational inconsistencies and an unclear depiction of the framework's data flow.\n\nPlease refer to Response 1.\n\n## 7.What is the specific rationale for using a learned decoder over a more interpretable, physics-based Point Spread Function (PSF) model? Could the authors demonstrate or elaborate on the distinct advantages of their chosen approach?\n\nPlease refer to Response 2.\n\n## 8.Could the authors elaborate on the design rationale for their architectural components? For instance, what motivated the choice of specific modules over other common alternatives in the literature?\n\nPlease refer to Response 3.\n\n## 9.How robust is the framework to challenging inputs? We ask the authors to provide analysis on failure cases, specifically regarding (a) inaccurate segmentations from SAM, and (b) images with very large amounts of defocus blur. Could the authors add the \"Limitations\" section as indicated in their submission checklist? This section should detail the method's specific failure modes and boundary conditions.\n\n(a) Inaccurate segmentations from SAM: \nThe language-based SAM, when guided by specially designed prompts, can generate sufficiently robust masks even on deblurred images. Given that the original SAM model was trained with diverse data augmentation, it exhibits inherent robustness to mild noise or blur. To further improve mask quality in our implementation:\n   - For region of interest, we use **language-style prompts**: *number + object color + object name + spatial position* (e.g., a brown dog on the chair).\n   - For those still imperfect masks, we further apply **post-processing techniques** such as dilation or erosion operations for refinement.\n\nTo evaluate the mask quality from our adopted SAM, we manually annotate two objects for each image of our collected images as the groundtruth. We compare their average IOU metrics,\n|3-click | language(only with object name)  |   language (number + object color + object name + spatial position)    |\n|------------|-----------|---------|\n| 93.6      | 91.8     | 87.5   |\n\nWe would provide the failure visualization case in our limitation section in final version.\n\n(b) Images with very large amounts of defocus blur. We empirically observe that our used SAM could functions well when handling the image with large defocus blur (i.e.,  $f$-1.8). We sample two groups of sample from DP5K-test and, each of which has 5 cases with different $f$-number, 1.8, 2.8, 4.0, and 5.6. We use the language-based SAM to generate the mask, and manually annotate the region with interest as the ground-truth. \n\nWe calculate their respective IOU metric.\n\n |$f$-1.8 | $f$-2.8  | $f$-4.0 | $f$-5.6  |\n|------------|-----------|---------|---------|\n| 91.3  |   91.6   | 91.8   |   91.8  |\n\nWe observe mariginal mask degradation.\nWe would include above analysis, and provide more visualization in our final version. Thanks.\n\n## 10. Could the authors provide the missing details on computational resources (GPU model, training/inference times) as stated in the checklist? This information is essential for reproducibility.\nOur model is trained on one A100 80G GPU, and test on GTX A6000.\n| dataset | training time (h) | latency (s) |\n|-------------|-----------|---------|\n| DP5K-model  | 5.02  | 0.64  |\n| DPD-disp-model |   20.58   | 0.56 |\n\n## 11. Could the author update the checklist for \"Open access to data and code\".\nYes, I will update it. Thanks.\n\n## Reference:\n[1] Restormer: Efficient Transformer for High-Resolution Image Restoration, CVPR 22\n\n[2]  K3DN: Disparity-aware Kernel Estimation for Dual-Pixel Defocus Deblurring, CVPR 23\n\n[3] Density estimation using real NVP, ICLR 2017"}]

Author rebuttal:

Meta-review: All four reviewers recommend acceptance (three borderline). The paper presents a user-friendly framework for Depth of Field manipulation in images from dual-pixel sensors. The strengths include the novel user interaction via the Segment Anything Model, the mathematically-driven invertible deblurring-reblurring framework, and the creation of a new real-world dataset. The main weaknesses are the reliance on existing paradigms and off-the-shelf components (like SAM), and the limitation to dual-pixel imagery. The authors' rebuttals were well-received and addressed the reviewers' concerns.
Decision: Accept (poster)