PhoCoLens: Photorealistic and Consistent Reconstruction in Lensless Imaging
Authors: Xin Cai
Zhiyuan You
Hailong Zhang
Jinwei Gu
Wentao Liu
Tianfan Xue
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Lensless Imaging; Computional Imaging;  Generative Prior
Primary area: machine_vision

Abstract: Lensless cameras offer significant advantages in size, weight, and cost compared to traditional lens-based systems. Without a focusing lens, lensless cameras rely on computational algorithms to recover the scenes from multiplexed measurements. However, current algorithms struggle with inaccurate forward imaging models and insufficient priors to reconstruct high-quality images. To overcome these limitations, we introduce a novel two-stage approach for consistent and photorealistic lensless image reconstruction. The first stage of our approach ensures data consistency by focusing on accurately reconstructing the low-frequency content with a spatially varying deconvolution method that adjusts to changes in the Point Spread Function (PSF) across the camera's field of view. The second stage enhances photorealism by incorporating a generative prior from pre-trained diffusion models. By conditioning on the low-frequency content retrieved in the first stage, the diffusion model effectively reconstructs the high-frequency details that are typically lost in the lensless imaging process, while also maintaining image fidelity. Our method achieves a superior balance between data fidelity and visual quality compared to existing methods, as demonstrated with two popular lensless systems, PhlatCam and DiffuserCam.

PhoCoLens: Photorealistic and Consistent Reconstruction in Lensless Imaging
Xin Cai Zhiyuan You', Hailong Zhang', Wentao Liu?., Jinwei Gu', Tianfan Xue
The Chinese University of Hong Kong, *Shanghai Artificial Intelligence Laboratory, TTsinghua University, "SenseTime
{cx023, yz023, tfxue)@ie. cuck. cuhk. edu.hk, hk, jwgu@cuhk. edu. hk, (zhangh121) Smails.mmm......mm.mmmmmmm ttsiisstsissssssssuaa...a edu. cn, 1tueentao@sensetimee com
Abstract
Lensless cameras offer significant advantages in size, weight, and cost compared traditional lens-based systems. Without a focusing lens, lensless cameras rely on computational algorithms recover the scenes from multiplexed measurements. However, current algorithms struggle with inaccurate forward imaging models and insufficient priors to reconstruct high-quality images. To overcome these limita- tions, we introduce novel two-stage approach for consistent and photorealistic lensless image reconstruction. The first stage of our approach ensures data consis- tency by focusing accurately reconstructing the low-frequency content with a spatially varying deconvolution method that adjusts to changes in the Point Spread Function (PSF) across the camera's 's field of view. The second stage enhances pho- torealism by incorporating a generative prior from pre-trained diffusion models. By conditioning the fowffrequeccy content retrieved in the first stage, the diffusion model effectively reconstructs the high-frequency details that are typically lost the lensless imaging process, while also maintaining image fidelity. Our method achieves superior balance between data fidelity and visual quality compared to existing methods, as demonstrated with two popular lensless systems, PhlatCam and DiffuserCam. Project website: phocolens ens wew..iithee.e tithub.io.
(a) Measurement
(b) Decony
(c) FlatNet
(d) FlatNet DiffBIR
(e) Ours
(f) Ground Truth
(a) Measurement
(b) WienerDeconw
(c) FlatNet
(d) FlatNet DiffBIR
(e) Ours
(f) Ground Truth
Figure 1: We introduce PhoCoLens, a lensless reconstruction algorithm that achieves both better visual quality and consistency to the ground truth than existing methods. Our method recovers more details compared to traditional reconstruction algorithms (b) and (c), and also maintains better fidelity to the ground truth compared to the generative approach (d).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

LM
55M
MMCN
Fattett-een
FlatNet CHMBIR
Manioe
PSWR
with spatially varying deconv Figure 2: Comparison between results without (left) and with (right) our spatially varying deconvolution.
w/o spatially varying deconv
Ciipio
MUSI
Figure 3: Evaluation of fidelity and authenticity on PhlatCam [17] dataset.
Introduction
Lensless imaging systems [6] have emerged a groundbreaking solution for ultra-compact, lightweight, and cost-effective imaging. They replace lenses by amplitude [3, 4] or phase masks [1, 5, 20] placed close the sensor to modulate incoming light. This design significantly reduces camera size and weight and enables innovative sensor shapes such as spherical [14] or cylindrical.
Considering the raw measurements from lensless cameras are typically blurry and unrecognizable, it hard to recover high-quality image of the original scene. An example of lensless measurement and recovered images by WienerDeconv [44] is shown in the left column of Fig. 1. Duc to the lack of focusing elements, lensless cameras cannot directly record the scene but encode it into a complex diffraction pattern. This encoding process can be approximated as convolution with large Point Spread Function (PSF). The PSF acts like a low-pass filter applied to the scene thus introducing ambiguity, which means there could be multiple possible recoveries for single measurement. Therefore, powerful algorithms are critical for high-quality reconstruction in lensless systems.
The primary challenge of a lensless reconstruction is to achieve both photorealism and consistency. Photorealism requires high-quality reconstruction with rich details, while consistency further re- quires reconstructed contents to be consistent with the original scene. Traditional techniques, like WienerDeconv, can reconstruct images that align with the ground truth but visual quality is signifi- cantly degraded (Fig. 1b). Learning-base based approaches such FlatNet-gen [17] attempt to enhance visual quality by training with paired images and lensless measurements, yet they often fail to recover high-frequency details (Fig. Ic). The visual quality be further improved by the restoration algorithms using generative priors, like DifBBIR [23] (Fig. Id). While these restoration methods inject rich details, they may also alter image content or insert non-existent objects, breaking consistency. For example, in Fig. 1d, the beak in the top row has the wrong shape compared to the ground truth (Fig. 1f), and the leaves in the bottom row look fake.
Moreover, an inaccurate imaging process simulation may also hurt data consistency. Most existing reconstruction algorithms [17, 27] simplify the imaging process convolution with thift-invariant PSF. However, in practice, PSFs are spatially varying, particularly when the angle of incidence increases. Consequently, these areas experience noticeable drop in the reconstructed similarity to the original scene, especially in the peripheral field of view.
To achieve both photorealism and consistency, we propose two-stage lensless reconstruction based on range-null space decomposition [36]. According to this decomposition, the reconstructed image consists of two components, one from "range space", which can be directly calculated from the lensless measurement (Fig. la), and the other from "null space", which are the detailed textures lost during the lensless imaging process. Therefore, the first stage prioritizes data consistency by recovering the "range space" component, which is the low-frequency content that matches the ground truth. The second stage focuses photorealism by adding more high-frce frgh-frreuency details from the "null space" while maintaining the consistency established in the first stage.
In the first stage, to improve consistency, we propose novel spatially varying deconvolution to reconstruct the "range space" content. Unlike traditional methods that assume shift-invariant PSF, our approach automatically adapts to spatial variations in the PSF across the camera's 's field of view in a data-driven manner. This innovation more accurately models the forward imaging process, leading to better reconstruction of structural integrity and detail in the low-frequency components. Fig. shows improvement with our method (right) against those using a single kernel for deconvolution (left).

---

In the second stage, to improve photorealism, we integrate a generative prior using pre-trained diffusion model, to insert realistic details into the first-stage output. This model specifically targets the recovery of nigh-frequency details lost in the lensless imaging process. We condition the diffusion model on the low-frequency content reconstructed in the first stage, guiding to incorporate missing high-frequency elements from the "null space". Our supervised approach ensures the final images are consistent with actual measurements and also achieve photorealistic quality.
Combining these two stages, our lensless reconstruction achieves a good balance between data consistency and visual quality in the reconstructed images, surpassing existing methods, as shown in Fig. le. To validate this, we compare our method with others on two types of lensless cameras: PhlatCam [17] and DiffuserCam [27]. using metrics that assess both fidelity and visual quality. The qualitative comparison on the PhlatCam dataset, demonstrating improvements in both aspects, is shown in Fig. 3, highlighting the superiority of our technique over current methods.
2 Related Work
Lensless imaging is traditionally addressed by solving regularized least squares problems for convolu- tional imaging models, often incorporating sparsity constraints like those in the gradient or frequency domain [1. 5, 21, 31]. Deep learning has revolutionized lensless imaging by enabling learnable deconvolution parameters and perception enhancement through image-to-image networks. Recent innovations include deep unrolled techniques [19, 27, 49], alongside feed-forward deconvolution methods in image space [17] or in feature space [22]. However, these methods generally assume a constant point spread function in the imaging process. Our proposal introduces spatially varying deconvolution approach to overcome this limitation and achieve more precise image reconstruction.
Spatially- varryng deconvolution has been well-studied area [11, 24, 46] due to the prevalence of imaging systems with PSFs that vary across the FoV. However, these methods [2. 20, 25, 29, 47] often slow, computationally intensive, and result in poor image quality, especially in complex systems. Recently, MultiWienerNet [48] introduced a deep learning approach for fast, spatially varying deconvolution, but it requires tedious multi-location PSF calibrations. In contrast, spatially varying deconvolution method, designed for lensless imaging, leverages a single initial PSF and automatically learns variations across the image, eliminating the need for extensive calibration.
Inverse imaging with diffusion models can be categorized as supervised or zero-shot. Supervised methods train conditional diffusion model [8, 13, 12, 32, 50] with paired images to bridge the gap between input and desired outputs, leveraging generative priors for inverse imaging [35, 39] and controlled generation [28, 34]. On the other hand, zero-shot methods [37] employ guidance to address a wide range of general inverse problems [9. 38]. These techniques typically rely on predefined conditions for guidance. Specific models like DDRM [15] and DDNM [41] focus mathematical decompositions to improve the diffusson process. Our work integrates the strengths of both approaches, combining supervised fine-tuning with theoretical framework based on range-null space decomposition, aiming for a robust solution to inverse imaging challenges.
3 Preliminary
3.1 Range-Null Space Decomposition
The lensless imaging process can be formulated as a linear transformation [10]. Specifically, given a scene plane x E RRM the sensor measurements y E NR is expressed as ý = Ax + n. Here, A E IR.. N x M² is the transfer matrix of the lensless imaging system, and n is the sensor noise. The transfer matrix A essentially encodes how light from each point on the scene plane contributes to each sensor pixel. Given a calibrated camera system, the transfer matrix A is known and the objective of lensless reconstruction is to recover the scene plane image x (like Fig. If) from the measurement (like Fig. aa). For simplicity, below we consider a noise-free case as y = Ax.
We then introduce null space decomposition. Let At E R MM*xN be the pseudo-invers of the linear matrix A, which satisfies AA A A = A. The operation A A can be interpreted as a projection onto the range space of A because for any sample x, we have AAtAx operation (I ATA) acts as projection onto the null space of A, given that A - AfA)x )x = Therefore, any sample can be decomposed into two orthogonal components: one that lies in the
Ax. In contrast, the
3

---

Mask
Sensor
(b)
PSF O
PSF at
50
(d)
Propagation
Point Source A
(c)
1.0
PSF Similarity
Shift
0.9
Point
2
0.8
Source B
(a)
Ug (5.n.
(x,y,d)
-30
20
Field Postion (*)
Figure 4: Characterization of PSFs in Lensless Camera. (a) Illustration of light propagation in the lensless camera: two point sources A and B at infinity emitting parallel light beams. Source emits at angle 9 relative to the optical axis, causing PSF shift on the sensor plane This PSF shift depends on both the incident angle 0 and the distance d between the sensor and the mask. (b) Simulated PSFs for light sources at angles of 0%, 15", and 30% (c) Inner product similarity between the on-axis PSF and off-axis PSFs at different field positions. (d) Reconstruction using PSF at 0", degradation is more significant at the periphery (red box) than the center (green box).
range space of A, and the other in the null space of A. Mathematically, this decomposition is:
x = A Ax + (I- A'A)x,
where A Ax is the range space component, and (I A A)x is the null space component.
Through this decomposition, we derive two distinct elements indicative of consistency and photo- realism. The "range space" term A Ax ensures consistency, as its multiplication by A yields the measurement y, fulfilling the consistency condition Ax = y. Conversely, the "null space" term (I A A') )x ensures that the reconstructed image & aligns with natural image statistics.
3.2 Mismatch in Convolutional Lensless Imaging Model
In lensless literature, most researchers [1, 5, 17, 27] simplify the imaging process as a convolution, because the full transfer matrix A R N ar2 is too large to compute. Specifically, the measurement y is obtained as = h * x, where h is the point spread function (PSF) of the lensless system and * denotes the convolution. Most previous reconstruction algorithms are based this assumption.
However, the real lensless imaging model not simply spatially invariant convolution as shown in Fig. 4a. Consider lensless camera consisting of mask placed at the longitudinal position 0 and a sensor placed at = = d. Let U (xx y, z) be a complex scalar wave field, a complex function of transverse coordinates x, y. and longitudinal position Denote the wave field immediately after propagating through the mask from a point source at infinity with an incident angle & as Uo(E, 77, 0+). Using the Huygens- Huygenn--Fresnl principle [10], the intensity pattern po po y) captured by the sensor is:
/] Uo(S, 0+) exp(jkr)dddn)
Po v) = [U,(1, y. 2)
jAr
where the distance r given by r= dd + {)2 + (y 7722 and À is the wavelength of light.
Therefore, according to Eqn. 2, the spatially invariant convolutional imaging model only satisfies under the Fresnel approximation [5, 10]:
/ d
(a 2)2 (y 11)2 d (x (o2-)))2 + (
This approximation is valid only when the distance d between the lensless mask and the sensor is large enough to satisfy d > v(x E) -)= (y 1/) However, most lensless masks are very close to the sensor (d 2mm in a typical lensless camera), breaking this assumption.
This mismatch can lead to inaccuracies [26, 43, 52] in the convolutional imaging model. To demon- strate that, we simulate the PSF of typical lensless camera (Phlatcam [5]) across various incident angles 0 using Eqn. (2). As shown in Fig. 4b, PSFs at different angles (09, 15", and 30") are visually different. Quantitatively, Fig. shows the similarity between PSFs at the center and -30" drops from 1.0 to 0.7.

---

Measurement y
Stage SVDeconv for Range Space Recon.
Stage Output
Stage 2: Conditional Diffusion for Null Space Recon.
Stage Output
Difusoon
range m + Lapips
L
Spatial Weights
Decov
Add UNet
A'A
Deconv Results
Weighted Deconv Results
Learnable
kernels
Spatially-varying Deconvolution (SVDeconv) Range Space Content A Ax
Ground Truth X
Figure 5: System Overview. The two-stage pipeline begins with spatially varying deconvolution network mapping lensless measurements to range space. Then conditional diffusion model for null space recovery refines details using the first stage output, achieving the final reconstruction.
To further show the consequences of this mismatch, we simulate lensless capture from a clean scene image (Fig. 4d top), with the accurate spatially-varying PSF based on Eqn. (2), but solve the inverse imaging problem using simple convolutional model using the PSF at the center. Fig. 4d bottom shows the result, and there are more artifacts at the boundary (red box) than at the center (green box). This is because the mismatch between the actual PSF and the single PSF used for deconvolution more significant at the periphery field of view.
4 Method
In this section, we introduce PhoCoLens, a method for achieving both Photorealistic and Consistent reconstruction in Lensless imaging. As illustrated in Fig. 5, PhoCoLens consists of two main stages: range space reconstruction and null space recovery. We will first introduce the whole framework and then provide detailed explanation of each stage.
Given an input lensless measurement y = Ax + n. where = Ax is the noise-free part of the lensless measurement, A is the transfer matrix of the lensless camera, x is the original scene aim to reconstruct, and n is the sensor noise. The objective in lensless image reconstruction is to recover photo-realistic image & which satisfies A&
Inspired by the range null space decomposition in section 3.1, we can decompose any potential solution & into two orthogonal components: the range space component that maintains consistency and null space component that maximizes photorealism. Formally, this decomposition given by & A Ax (I - A A1A) & where A Ax is the range space component and (I A A ) is the null space component. Note that any choice of X satisfies the equation A& y, as A A 0.
With this decomposition, illustrated in Fig. 5. In the first stage, we recover the range space content represented by A Ax from the noisy lensless measurement y. Specifically, Varying Deconvolution Network that learns to reconstruct the range space content effectively. The second stage focuses adding the null space content represented by - A A X For this purpose, we introduce null space disfusion, a conditional diffusion model designed for null space recovery. By conditioning on the range space reconstruction obtained from the first stage, the null space diffusion enhances these images by incorporating the null space content, thereby rendering them with more realistic appearance. Below we introduce each stage.
reconstruct the range space and null space components in stages,
propose SVDeconv, physics-inspires Spatially-
4.1 Spatially-varying Deconvolution Network
the first stage, we use SVDeconv, a novel network designed to invert the forward lensless imaging model and reconstruct the range space content A Ax from a lensless measurement ý Ax + n with noise n. SVDeconv is composed two main components: differentiable multi-kernel deconvolution layer and refinement U-Net, as depicted in the gray box at the bottom left corner of Fig. 5.
5

---

Traditional lensless imaging methods often employ single PSF to model the imaging system, which inaccurate for large field of view (FoV) scenarios as analyzed in section 3.2. To address this, SVDeconv utilizes set of learnable K x K PSF kernels, accounting for spatial variations across the field of view. Specifically, we partition the target image region into a grid of K x K segments, and for each, we apply an individual PSF kernel. This process results in K x K' deconvolution
operations, producing K K intermediate deconvolved images, as shown in Fig. 5. The multi-kernel deconvolution is mathematically described Hadamard product in the Fourier domain:
= FF ( o F())), 1.2, ...., K x K,
(4)
where Xie is the i-th deconvolution results of the i-th learnable PSF kernel p F()) and F-1()) are the DFT and the Inverse DFT operations, and O denotes the Hadamard product.
From this deconvolution operation, we obtain K K intermediate deconvolved images. Assuming that each image accurately reconstructs a specific region of the target image corresponding to its PSF's field point, we propose integrate these images into single, unified intermediate image using an innovative interpolation method. The interpolation is expressed as:
x²
Xint (u, 0)
w; wu v) v)x (u, v),
(5)
i=1
(6) where u, are coordinates. The weight (r, u) for each deconvolved image x is inversely proportional to the distance between the point (u, v) and the focal center of each corresponding PSF, formally defined as:
(y v) = d. (u,v) with d((uv)) = (u- u.) (v - (.)
(6)
K2 Ej=1 2
In this way, it ensures that wi(u, v) normalizes the contribution of each deconvolved image based on inverse Euclidean distance to each point (u,, ut). which represents the center of the FoV corresponding to the i-th learnable PSF kernel O This weighted sum effectively interpolates the intermediate images form unified representation with enhanced clarity and detail across the entire FoV.
One advantage of this weight design is that it significantly simplifies the calibration process. Previous multi-kernel methods, such as [48], often require multiple calibrated PSFs at different focal centers, leading to a time-coneee consuminnn calibration process. On the contrary, our approach only requires one calibrated PSF to initialize all kernels. This is because we pre-define the center of the PSF and allow the model to learn its variations automatically.
After the interpolation, the intermediate image Xim is fed into the refinement U-Net [33], removing noise and artifacts the reconstruction to approximate the range space content At Ax. We employ a combination of MSE loss and LPIPS loss [51] to train both the learnable PSFs and U-Net.
4.2
Null Space Content Recovery
With the reconstructed range space content, aim recover final image that maintains consistency and improves photorealism. To ensure consistency, the difference between the final image and the range space content (residual content) should reside in the null space, which ensures the final image aligns with the original lensless measurement. To improve photorealism. the combination of the null space and range space content should appear as a realistic real-world image.
To achieve both objectives, we propose null-space diffusion, which takes the reconstructed range space content as the condition and generates an image that adheres both constraints. It ensures the residual content lies in the null space while maintaining its ability to produce realistic images.
To ensure the outputs from null-space diffusion meet the consistency requirement, we train the model such that the residual content of these samples falls within the null space. Mathematically, if we denote the output as X, it should satisfy the following equation when conditioned by A Ax:
A(X A Ax)
AX Ax
A( x =
(7)
Recall that A is the transfer matrix of the lensless camera and At is its pseudo-inverse. This equation indicates that applying the operator A to the difference between the generated sample x and the original sample x results in zero. This ensures that the residual content resides in the null space of A.

---

Based on this, we design the null space diffusion as follows. Given an image and its corresponding range space content
Ax, we add noise progressively to the image resulting a noisy image Xt, where I. is the noise addition iteration. Similar to other conditional diffusion models [50], we train a network Eg to predict the noise added on the noisy image X with the optimization objective:
min Cuull min (ot.....e-NN....lAAAA....
ł, - lll
(8)
which is derived to align with the goal in Eqn. (7). More details are in the Appendix.
Additionally, to ensure the null-space diffusion produces realistic images according to given condi- tions, we utilize pre-trained diffusion model such as Stable Diffusion [32] with its weights frozen. We focus on training supplementary conditioning modules to guide the generative process using range space conditions while preserving its powerful ability to generate realistic images. Specifically, we follow the StableSR [40] structure, which involves using a conditional encoder to extract multi-scale features from the range space condition and then using them to modulate the intermediate feature maps of the residual blocks in the diffusion model. This modulation can align the generated images with the range space conditions, enhancing the fidelity and realism of output images.
5 Experiments
In this section, we assess the performance of our proposed method using two lensless imaging datasets, collected in real-world environments by two different kinds of lensless cameras: PhlatCam [5] and DiffuserCam [1]. PhlatCam employs a phase mask, while DiffuserCam utilizes a diffuser for its lensless mask. We compare our approach with other methods and conduct a comprehensive ablation study to evaluate the effectiveness of our design.
5.1 Dataset and Metries
The PhlatCam dataset [17] contains 10,000 images across 1,000 classes resized to 384x384 pixels. Images displayed and captured using a lensless PhlatCam [5], generating images at 1280 x 1480 pixels. We use 990 classes for training and 10 classes for testing, following the original protocol.
The DiffuserCam dataset [27] contains 25,000 paired images captured simultaneously using standard lensed camera (ground truth) and a mask-based lensless camera DiffuserCam [1]. These pairs are split into 24,000 images for training and 1,000 for testing. Both cameras utilize sensors with a native resolution of 1080x 9920 pixels, which are first downsampled to 270x480 pixels and then further cropped to final resolution of 210x380 pixels for proper display.
To evaluate both consistency (fidelity) and photorealism (visual quality), we employ two metric sets:
Full-reference metrics to evaluate the consistency. Three full-reference metrics, PSNR, SSIM [42], and LPIPS [51], are used to evaluate the distance between the network output and ground truth, which indicates the fidelity of the reconstruction.
Non-reference metrics to evaluate the photorealism. Three non-reference metrics, ManIQA [45], ClipIQA [39], and MUSIQ [16], are used to evaluate the visual quality of reconstructed images.
5.2 Implementation Details
For SVDeconv, utilize 3 3 PSF kernels for deconvolution. We initialize the 9 kernels using single calibrated PSF from both the DiffuserCam and PhlatCam datasets. The U-Net component in SVDeconv is adapted from the U-Net used in Le-ADMM-U U for training on the DiffuserCam, and from the U-Net in FlatNet-gen for the PhlatCam. We train the network for 100 epochs with a batch size 5, using the Adam optimizer [18]. The learning rate is set 3c-5 for U-Net training in both datasets. For deconvolution kernel training, the learning rate is set at 4c-9 for PhlatCam and 3e-5 for DiffuserCam. We set the MSE and LPIPS loss weights to be 1 and 0.05, respectively. In the DiffuserCam dataset, where the measurements are cropped by the sensor's 's limited size, we employ replicate padding [17] which expands the width and height of the measurements by a factor of two.
For null-space diffusion, we use the range space content reconstructed by SVDeconv as input condi- tions. We train it for 200 epochs, following the training and inference settings used in [ableSR[[9]].
In the first stage, SVDeconv is trained using range-space content derived from ground truth images. SVDeconv consists of two main parameter components: a learnable deconvolution kernel initialized
7

---

Table 1: Performance comparison of methods on DiffuserCam and PhlatCam
Dataset
PhlarCam
DiffuserCam
Metries
PSNR SSIM LPIPS ManIQA ClipiQA MUSIQ
PSNR SSIM LPIPS} ManIQA ClipIQA
MUSIQ
Ground Truth
0.431
0.583
63.40
0.160
0.333
31.21
WienerDeconv [44]
12.19
0.270
0.922
0.111
0.149
15.47
10.59
0.275
0.843
0.184
0,180
19.43
ADMM
13.45
0.301
0.877
0.132
0.159
16.45
12.87
0.305
0.705
0.141
0.136
16.51
Le-ADMM-U [27)
20.12
0.515
0.405
0.138
0.180
24.64
22.35
0.668
0.253
0.110
0,185
20.16
MMCN [49]
20.39
0.524
0.346
0.145
0.206
27.65
24.09
0.744
0.238
0.121
0.183
21.47
UPDN [19]
20.48
0.533
0.352
0.158
0.215
29.32
24.67
0.747
0.256
0.139
0.178
22.56
FlaiNet-gen [17]
20.53
0.549
0.375
0.203
0.287
44.94
21.43
0.696
0.254
0.134
0.244
23.15
DDNM+[41]
15.22
0.485
0.623
0.297
45.32
18.36
0.539
0.516
0.162
0.201
27.83
FlaNNet+DiffBIR [23]
19.96
0.544
0.391
0.335
0.523
57.13
18.97
0.517
0,505
0.312
0.548
51.20
PhoCol.ens(Ours)
22.07
0.601
0.215
0.357
0.565
62.20
24.12
0.748
0.161
0.172
0.339
32.84
with known PSFs, and a U-Net initialized with standard weights without pretraining. Once trained, SVDeconv processes input lensless measurements to estimate the range-space content of training samples. Subsequently, via null-space diffusion. During diffusion fine-tuning, we utilize a pre-trained diffusion model with frozen weights. We only train the supplementary conditioning modules like StableSR [35], to guide the reconstruction process effectively.
this estimated range-space content input conditions for fine-tuning
WienerDeconv Le-ADMM-U
FlatNet-gen
MatNet+DiffBIn
Ours
Groundtruth
Figure 6: Qualitative comparison between method and others the PhlatCam dataset.
Le-ADMM-U
UPDN
FlatNet+DiffBIR
Ours
Groundtruth
Figure 7: Qualitative comparison between our method and others on the DiffuserCam dataset. 5.3 Comparison with Other Approaches
We evaluate the performance our proposed method by comparing it against both traditional and learningbbased approaches, using qualitative and quantitative measures on the two datasets.
We compare traditional methods like Tikhonov regularized reconstruction in the Fourier domain (WienerDecony [44]) and total variation regularization via ADMM [7]. Additionally, we evaluate learning-based methods like the unrolled network Le-ADMM-U [27], MMCN [49], and UPDN [19] and the feedforward deconvolution method FlatNet-gen. Additionally, we assess two diffusion-based methods: one uses pre-trained Stable Diffusion [32] with zero-shot inverse imaging sampling method DDNM+ [41], and another enhances outputs from FlatNet-gen using pre-traing pre-trained blind

---

Table 2: Comparison of deconvolution methods for range
space reconstruction and original content reconstruction.
Reconstruction Method
PhlaiCam
DiffuserCam
Target
PSNR SSIM I.PIS.
PSNR SSIM I.PISS
FlatNet-gen
MultiWinenerNet
Le-ADMM-U
23.87 0.835 0.307
24.24 0,748
0.282
Range Space
MultiWienerNet
SingleDecony
26.43
25.61 0.874
0.880
0.256
0.263
26.32
24.94 0.799
0.823
0.189
0.230
SVDeconv (Ours)
26.97
0.894
0.253
28.84
0.856
0.144
Le-ADMM-U
20.12 0.515
0.405
22.35
0.668
0.253
Original Content MultiWienerNet
SingleDecony
20.82
20.43 0,545
0.562
0.374
0,381
23.02
21.97 0.689
0,725
0.262
0.214
Figure 8: Comparison deconvolution
SVDencanv(Ours)
Groundtruth
SVDecony (Ours) 21.01 0.571
0.363
25.55
0.781
0.179
methods on DiffuserCam example.
Table 3: Comparison of null space recovery methods
Table 4: Comparison of different diffusion conditions
Method
PSNR LPIPS] ManIQA ClipIQA MUSIQ
PhlarCam
DiffuserCam
Condition
StableSR [39] 14,93 0.446 0.624
0.64
0.235
33.84
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS]
DiIBIR [23] 16.21 0.432
0,502
0,353
0.512
56.65
WienerDecony
20.38
0.551
0.301
20.42
0.632
0.282
DDNM
17.62 0.539
0.517
0.148
0.201
27.82
FlaNettgeen
SVD.OC OC
21.16
20.89
0.556
0.286
0.276
21.67
0.647
0.679
0.230
0.185
Ours
22.07
0.215
0.357
62.20
Ours
22.07
0.601
0.215
24.12
0,748
0.161
Groundtruth Figure 9: Comparison between our null space diffu- sion and other methods.
DDNM
DiffBIR
Ours
FlatNet-gen
SVD-OC
Ours
Groundtruth
Figure 10: Comparison between different diffusion conditions.
image restoration model DiffBIR [23]. Quantitative results are summarized in Tab. 1. Qualitative comparisons for PhlatCam and DiffuserCam are shown in Figures 6 and 7, respectively.
On the PhlatCam dataset, method achieves the best performance on both fidelity (full-reference metrics) and visual quality (non-reference metrics) according to Tab. 1, This is further supported by the qualitative results in Fig. 6, where our reconstruction are the closest the ground truth.
For the DiffuserCam dataset, our method achieves superior performance in consistency metrics (full-reference metrics, SSIM, and LPIPS). While UPDN achieves slightly higher PSNR, the visual quality its outputs suffers from artifacts, as shown Fig. This slight PSNR drop in our method is likely due artifacts within the DiffuserCam ground truth itself, such moiré patterns. These artifacts can penalize high-quality images with a lower PSNR, despite their good visual consistency. This is further evidenced by the visual comparison in Fig. 7. Conversely, SiatNet+DiffBi though achieving higher quality metrics (non-reference metrics, ManIQA, ClipIQA, and MUSIQ), fails to maintain data consistency (low full-reference metrics, PSNR, SSIM, and LPIPS). Also, shown Fig. 7, FlatNet+DiffBIIR sometimes badly distorts the original image content, like transferring a method achieves the best trade-off between the
human hand into clothes in the top row. Overall, data consistency and the visual quality. And thanks to the introduction of SVDencov, our method can reconstruct the field-of-view periphery better than other methods.
5.4 Analysis and Discussion
Effectiveness of Spatially-varying Deconvolution. To verify the effectiveness of the proposed SVDe- conv, we compare it with other learnable deconvolution approaches. Baselines include Le-ADMM-U deep unrolled network implementing ADMM), SingleDeconv (using a single deconvolution kernel), and MultiWienerNet (emplloyg multiple deconvolution kernels without weighted interpolation, initialized similarly to our approach with a single PSF). We focus on two types of content reconstruc- tion: range space content reconstruction, where range space content is the target, and original content

---

reconstruction. using the original ground truth as the target. For a fair comparison, all methods utilize the same U-Net for denoising and perceptual enhancement and the same loss for training. Results in Tab. 2 show the efficacy of the proposed spatially-varying deconvolution in the faithful reconstruction of both range space content and original content. Further, Fig. shows our method outperforms others, particularly in reconstructing fine details at the periphery (circled by white dashed boxes).
Effectiveness of Null-space Difftusion. Following the reconstruction of range space content, we perform null space content recovery conditioned on the previously reconstructed range space content. We compare our null- -pacce diffusion model with other approaches, including conditional diffusion models like DiffBIR [23] and StableSR [39], as well as the zero-shot inverse imaging method DDNM [41]. Quantitative results in Tab. 3 and qualitative comparison in Fig. demonstrate that our null-space diffusion performs the best in both consistency with the ground truth and visual quality.
Effectiveness of Range Content Conditions. One important design of our framework is first stage reconstruction as the condition for the second stage diffusion. To evaluate the effectiveness of these conditions, we perform ablation study where we replace our reconstructed range content with the outputs from WienerDeconv, FlatNet-gen, and the SVD-OC (SVDencov trained with original content), and re-train the conditional diffusion models using these conditions. As evident from Fig. 10, our range content condition significantly improves reconstruction consistency compared to others. This because using output from models like FlatNet- -enn introduces additional artifacts that hinder the second stage from recovering the original image. Furthermore, quantitative analysis in Tab. 4 confirms this observation. Our method outperforms these conditions in all fidelity metries.
the
6 Conclusion and Limitation
We present PhoCoLens, an innovative two-stage approach for achieving photorealistic and consistent reconstructions in lensless imaging. The method leverages the strengths of two complementary approaches: spatially varying deconvolution for the range space ensures consistency, while null-space diffusion for the null space guarantees photorealism. Our experiments on two lensless cameras demonstrate PhoCoLens' effectiveness in reconstructing realistic scenes while preserving fidelity. One limitation of our approach is that the two-stage nature of PhoCoLens and the diffusion model's sampling time hinder its real-time applicability. Additionally, although PhoCoLens achieves the best fidelity, the diffusion model we used may still introduce high-frequency details that deviate from the original scene, particularly for smooth scenes lacking in detail. Future work will focus on two main aspects: accelerating the diffusion model sampling process to enable real-time photo and video capture with lensless cameras, and exploring 3D spatially varying point spread function (PSF) effects to capture 3D scene information.
Acknowledgment
We thank the reviewers for their valuable comments. This work is supported by the Shanghai Artificial Intelligence Laboratory, the RGC Early Career Scheme (ECS) No. 24209224, and CUHK Direct Grants (RCFUS) No. 4055189.
References
[1] Nick Antipa, Grace Kuo, Reinhard Heckel, Ben Mildenhall, Emrah Bostan, Ren Ng, and Laura Waller. Diffusercam: lensless single-exposure 3d imaging. Optica, 5(1):1-9, 2018.
[2] Muthuvel Arigovindan, Joshua Shaevitz, John McGowan, John W Sedat, and David A Agard. A parallel product-convolution approach for representing depth varying point spread functions in 3d widefield microscopy based on principal component analysis. Optics express, 18(7):6461-
6476, 2010.
[3] M Salman Asif, Ali Ayremlou, Aswin Sankaranarayanan, Ashok Veeraraghavan, and Richard G Baraniuk. Flatcam: Thin, lensless cameras using coded aperture and computation. IEEE Transactions on Computational Imaging, 3(3):384-399) 2016.
[4] Eric Bezzam, Martin Vetterli, and Matthieu Simeoni. Privacy-enhancing optical embeddings for lensless classification. arXiy preprint arXiv:2211.1286/ 2022.
10

---

[5] Vivek Boominathan, Jesse K Adams, Jacob T Robinson, and Ashok Veeraraghavan. Phlatcam: Designed phase-man thase-mask based thin lensless camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, (2(7):1618-166)
2020.
[6] Vivek Boominathan, Jacob T Robinson, Laura Waller, and Ashok Veeraraghavan. Recent advances in lensless imaging. Optica, 9(1):1-16, 2022.
[7] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed opti- mization and statistical learning via the alternating direction method of multipliers. Foundations and Trends@ in Machine learning, 3(1):1-122, 2011.
[8] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. llvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2/08.02953 02938, 2021.
[9] Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems, 35:25683-25666 25696, 2022.
[10] Joseph Goodman. Introduction to Fourier optics. Roberts and Company publishers, 2005.
[11] Felix Heide, Mushfiqur Rouf, Matthias B Hullin, Bjorn Labitzke, Wolfgang Heidrich, and Andreas Kolb. High-quality computational imaging through simple lenses. ACM Transactions on Graphics (ToG), 32(5):1-14, 2013.
[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[13] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 1259,, 2022.
[14] Yi Hua, Yongyi Zhao, and Aswin Sankaranarayanan. Angle sensitive pixels for lensless imaging spherical sensors. arXiv preprint WxXiv:2306.15953, 2023.
[15] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 35:23593-23660 2022.
[16] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar. and Feng Yang. Musiq: Multi-scal image quality transformer. In Proceedings of the IEEE/CVF international conference computer vision, pages 5148-5157, 2021.
[17] Salman Siddique Khan, Varun Sundar, Vivek Boominathan, Ashok Veeraraghavan, and Kaushik Mitra. Flatnet: Towards photorealistic scene reconstruction from lensless measurements. IEEE Trunsuctions on Puttern Analysis and Machine Intelligence, 1934,11934 2020.
[18] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 141 2014.
[19] Oliver Kingshott, Nick Antipa, Emrah Bostan, and Kaan Akşit. Unrolled primal-dual networks for lensless cameras. Optics Express, 30(26):4632444663 2022.
[20] Grace Kuo, Fanglin Linda Liu, Irene Grossrubatscher. Ren Ng, and Laura Waller. On-chip fluorescence microscopy with random microlens diffuser. Optics Express, 28(6):838488399 2020.
[21] Chengbo Li, Wotao Yin, Hong Jiang, and Yin Zhang. An efficient augmented lagrangian method with applications to total variation minimization. Computational Optimization and Applications, 56:507-530, 2013.
[22] Ying Li, Zhengdai Li, Kaiyu Chen, Youming Guo, and Changhui Rao. Mwdns: reconstruction in multi-scale feature spaces for lensless imaging. Optics Express, 31(23):39088-39100 2023.
[23] Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Ben Fei, Bo Dai, Wanli Ouyang, Yu Qiao, and Chao Dong. Diffbir: Towards blind image restoration with generative diffusion prior, arXiv preprint arXiv:2308.10 15070, 2023.
11

---

[24] Jun Luo, Yunfeng Nie, Wenqi Ren, Xiaochun Cao, and Ming Hasan Yang. Correcting optical aberration via depth-aware point spread functions. IEEE transactions on pattern analysis and machine intelligence, 2024.
[25] Elie Maalouf, Bruno Colicchio, and Alain Dieterlen. Fluorescence microscopy three- dimensional depth variant point spread function interpolation using zernike moments. JOSA A, 28(9):1864-1700 2011.
[26] Kyoji Matsushima. Shifted angular spectrum method for off-axis numerical propagation. Optics Express, 18(17)118453-1846 2010.
[27] Kristina Monakhova, Joshua Yurtsever, Grace Kuo, Nick Antipa, Kyrollos Yanny, and Laura Waller. Learned reconstructions for practical mask-based lensless imaging. Optics express, 27(20);28055-28999, 2019.
[28] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi. and Ying Shan. T2i-adapter:
Learning adapters dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4296-4304, 2024.
[29] Nurmohammed Patwary and Chrysanthe Preza. Image restoration for firee-dimensional fluo- rescence microscopy using an orthonormal basis for efficient representation of depth-variant point-spread functions. Biomedical optics express, 6(10):38263388 2015.
[30] Ken Perlin. Improving noise. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques, pages 681-682, 2002.
[31] Dikpal Reddy, Ashok Veeraraghavan, and Rama Chellappa. P2c2: Programmable pixel com- pressive camera for high speed imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 329-336. IEEE, 2011.
[32] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[33] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted iitervention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234 241. Springer, 2015.
[34] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-in to-imagee diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[35] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE transactions on pattern analysis and machine intelligence, 45(4):4713-4726 2022.
[36] Johannes Schwab, Stephan Antholzer, and Markus Haltmeier. Deep null space learning for inverse problems: convergence analysis and rates. Inverse Problems, (5(2)x025008, 2019.
[37] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances neural information processing systems, 32, 2019.
[38] Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. arXiv preprint arXiv:2111.080 2021.
[39] Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2555 2265-2553,222 2023.
[40] Jianyi Wang. Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Ex- ploiting diffusion prior for weal-world image super-resolution. arXiv preprint 0,0002230..07015 2023.
12

---

[41] Yinhuai Wang, Jiwen Yu, and Jian Zhang, Zero-shot image restoration using denoising diffusion null-space model. In International Conference on Learning Representations. 2022.
[42] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions image processing, 13(4)6000-
612, 2004.
[43] Haoyu Wei, Xin Liu, Xiang Hao, Edmund Y Lam, and Yifan Peng. Modeling off-axis diffraction with the least-sampling angular spectrum method. Optica, 10(7):959 902,99622222 2023.
[44] Norbert Wiener. Extrapolation. interpolation, and smoothing of stationary time series: with engineering applications. The MIT press, 1949.
[45] Sidi Yang, Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujuu Yang. Maniqa: Multi-dimension attention network for no-reference image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1191-1200, 2022.
[46] Xinge Yang, Qiang Fu, Mohamed Elhoseiny, and Wolfgang Heidrich. Aberration-awarc depth- from-focus. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
[47] Kyrollos Yanny, Nick Antipa, William Liberti, Sam Dehaeck, Kristina Monakhova, Fan- glin Linda Liu, Konlin Shen, Ren Ng, and Laura Waller. Miniscope3d: optimized single-shot miniature 3d fluorescence mieroscopy. Light: Science & Applications, 9(1):171, 2020.
[48] Kyrollos Yanny, Kristina Monakhova, Richard W Shuai, and Laura Waller. Deep learning for fast spatially varying deconvolution. Optica, 9(1::66-99 99, 2022.
[49] Tianjiao Zeng and Edmund Y Lam. Robust reconstruction with deep learning to handle model mismatch in lensless imaging. IEEE Transactions on Computational Imaging, 7:1080-1092. 2021.
[50] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[51] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[52] Wenhui Zhang, Hao Zhang, Kyoji Matsushima, and Guofan Jin. Shifted band-extended angular spectrum method for off-axis diffraction calculation. Optics Express, 1907))10089-10111 2021.
13

---

Appendix
A PhlatCam Simluation
In section 3.2, we simulate a PhlatCam to interpret the detail of mismatch in a convolutional lensless imaging model. Here we explain how to simulate lensless camera like PhlatCam.
Following previous work [5], our simulation begins by constructing a desired point spread function (PSF) based on the contours generated from random Perlin noise [30]. Given a specific distance between the mask and sensor, we optimize the phase mask to produce PSF that matches the target. For this optimization, we employ the near-field phase retrieval algorithm (NIPR) [5]. This iterative algorithm alternates between the fields at the mask plane and the sensor plane, enforcing specific constraints at each location: the field amplitude at the mask plane set to unity, and the intensity of the field at the sensor plane conforms to the target PSF. The algorithm utilizes forward Fresnel propagation for computing the field from the mask to the sensor plane and applies backward Fresnel propagation to compute from the sensor back to the mask. The inputs required for phase mask optimization include the target PSF, the mask-to-sense wavelength, which in this case is set to 532 nm, a typical wavelength in the mid-visible spectrum.
distance (Imm in our setup), and the light
Target PSF
Optimized Phase Mask
Simulated PSF
Figure Al: Visual examples for results of the range space content simulation.
Fig. Al illustrates the target PSF, the optimized phase mask, and the simulated PSF resulting from the Fresnel propagation method applied the designed phase mask. Given a desired phase mask, we can accurately simulate the lensless imaging process under various situations.
B Range Space Content Approximation
In the first stage of our method (section 4.1), we want to reconstruct the range-space projection At from the lensless measurement y. This reconstruction requires the known target A Ax corresponding to the given image x. However, for real-world lensless imaging systems, directly obtaining the transfer matrix A E x is computationally infeasible due to its considerable size. Consequently, we resort to simulating the realistic lensless imaging process using an approximate form of A and emulate its pseudoinverse AT by inverting this approximation.
In cases where single point spread function (PSF) h is calibrated, we approximate the forward imaging process as follows:
ysim = C(h x) + n,
(A1)
where denotes the cropping operation due to the limited sensor size, * represents the convolution operator, and represents both sensor and quantization noise. In our practical implementation, the simulated ysim is quantized to 12 bits, and sensor noise introduced to achieve a signal-to-noise ratio (SNR) of 30.
The approximation for A is derived by reversing the forward process, specifically by applying replicate padding to counteract the cropping and employing a Wiener deconvolution to reverse the convolution:
Xyange D(P(yaim) h)
(A2)
14

---

where the P represent replicate padding and the D represent Wiener deconvolution.
If multiple calibrated PSFs available across different fields of view (FoV), the imaging forward process better approximated by employing a low-rank model to mimic the spatially varying convolution. Correspondinglyy, the spatially varying deconvolution model discussed in section 4.1 could be utilized to simulate A!.
In PhlatCam and DiffuserCam setups, we only have single calibrated PSF Therefore, constitute the range-space content of image x using the simple convolutions. It's important to note that even without multiple calibrated PSFs for more accurate simulation, the generated range-space content remains good approximation the real counterpart. As shown in Section 5.4, our simulation still serves as better supervision compared to the original image content in the first stage.
Range space
content A Ax
Original x
5
Figure A2: Visual examples for results of the range space content simulation.
Ground truth
FlatNet output
Range space content AT Ax
Estimated range space content
Figure A3:
Examples of range space content estimated by method.
Fig. shows the visual examples of the original ground truth x and corresponding simulated range space content A Ax of PhlatCam. It is shown that the range of space content A Ax keeps the low-frequency structure of the original scene and lost some high-frequency details from the lensless imaging process.
15

---

Fig. A3 shows the visual examples of range space content estimated by our method. The outputs from SVDeconv exhibit high visual consistency with the ground truth, as highlighted in the green boxes, in contrast those from FlatNet. FlatNet alters the original scene's content and introduces incorrect high-frequency details. Conversely, our method preserves only the oowfrequency content in the range space, aligning closely with the original ground truth.
C Training Target for the Null-space Diffusion Model
In section 6, we establish that the goal articulated in Eqn. (7) corresponds directly to the optimization objective described in Eqn. (8) used for diffusion training. We provide detailed derivation to support this equivalence.
Diffusion models [32] can be characterized by signal-to-noise ratio defined by two sequences (an))=1 and (o.))=1- Given data sample Xo. the forward diffusion process & can expressed as:
(xx
xx) = Nxx 0: If
(A3)
where N denotes the normal distribution.
The Markov property of the diffusion is given as:
((x: x.) = N(xx dsj oia.l),
(A4)
for 8 < 1, with:
O///
and o/j o/ 0
(A5)
0g
Given X = 0xXo + o,e, and a range space condition e AtAx, the diffusion output x can be modeled by x xx(,,, t). To ensure reconstruction consistency such that A xx) 0 in Eqn. (7), the optimization objective is defined as:
min Ex Exp. [IX, c,t) xx)
(A6)
Following previous work [12], we employ the reparametetization of the reconstruction term as denoising objective:
Xs - axxo c,
cx(x,,
(A7)
Ox
minimize:
min Cmull (xx,,......((...1
(A8)
which is exactly the Eqn. (8).
Due the substantial computational cost associated with simulating the transfer matrix A, an alternative optimization objective can be employed in practice. Specifically, we can approximate the goal described in Eqn. (8) using the following optimization formulation:
min Lnullapp Exp.te,..
[1l(ex(x,, c,t) -
(A9)
D Qualitative Results
We present additional qualitative results demonstrating the superior performance of our methods in Fig. A4 for the PhlatCam dataset and in Fig. A5 for the DiffuserCam dataset.
16

---

WienerDeconv Le-ADMM-U
FlatNet-Ren
FlatNet+ DiffBIR
Ours
Groundtruth
Figure A4: Qualitative comparison between our method and others the PhlatCam dataset.
Le-ADMM-U
UPDN
FlatNet+DiffBIR
Ours
Groundtruth
Figure A5: Qualitative comparison between our method and others on the DiffuserCam dataset.
17

---

E
Broader Impacts
Lensless imaging algorithms represent significant advancement camera technology, offering both promising benefits and potential drawbacks for society. On the positive side, these algorithms facilitate the development of cameras that are smaller, lighter, and more energy-effccien. This can greatly expand their applicability in various fields, particularly in medical contexts where the miniaturization of imaging devices is crucial. For example, tiny lensless cameras could be integrated into endoscopic tools for less invasive procedures, improving patient outcomes and recovery times. Furthermore, such compact cameras could be used in wearable technologies to monitor health conditions continuously, providing real-time data both patients and healthcare providers.
However, the proliferation of small, lensless cameras also introduces potential negative societal impacts, particularly concerning privacy. The ease with which these discreet cameras can be incorpo- rated into everyday environments raises the possibility misuse, such as unauthorized surveillance or covert photography. This concern is particularly acute in private spaces where individuals expect high level of privacy. As such, while lensless imaging technology promises significant advancements in fields like healthcare, it also necessitates stringent regulations and ethical guidelines to prevent invasive use and protect individual privacy rights.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals motivation long as is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations discussed in section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification,
asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't I acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does include theoretical results.
Guidelines:
19

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
For details on reproducing the experimental results, please refer to section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the case of closed-souse source models, it may be that access to the model is limited in some way (e.g.. to registered users). but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code and data will be available on the project website.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
20

---

Please see the NeurIPS code and data submission guidelines (tttpe://nipss.cccpbbii cc/pubicc/ guides/Code htth//ooeeSeuiiiiceee ssionpolicy)) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https : /ntps.ipppppp/// cc/public/ guides/ Cae///ooeSuubii ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The implementation details of the experiments are provided in section 5.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We report the experiment results using the standard conventions in lensless imaging research, consistent with previous studies.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of eπrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
21

---

8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed reproduce the experi- ments?
Answer: [Yes]
Justification: Evrey experiment can be completed within two days using server equipped with eight NVIDIA RTX 3090 GPUs, a 64-core CPU, 256GB of RAM, and 1TB of storage.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code Ethics https: http::://neuripppp nwwrips....e....eeeee.... wwtpaapppp..ppppc/eeeeiaea cc/pubicc/hhicceeieeel......
Answer: [Yes]
Justification:
We strictly adhere to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts of our work are explained in appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
legitimate to point out that
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
22

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g.. pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
We will not release data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer:
[Yes]
Justification:
We cite all original papers and make sure that our usage is legal.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasses has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged reach out the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
23

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Sub- jects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer:
[NA]
Justification: Our work does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
24


Reviews:
[{"review_id": "Zfw8Ry3Ur3", "reviewer": "Reviewer_FAra", "summary": "This paper proposes a novel two-stage approach for lensless image reconstruction. The approach ensures data consistency with a spatially varying deconvolution method and enhances photorealism using pre-trained diffusion models. This method outperforms existing techniques in data fidelity and visual quality on PhlatCam and DiffuserCam systems.", "questions": "1) This is minor, but why is Wiener Doconvolution referred to as WinnerDeconv everywhere in the paper?\n2) Although the paper compares the proposed approach with several other methods for qualitative evaluation, it is unclear how these approaches compare when it comes to computational complexity. Typically, there is a trade-off between quality and compute. Therefore I would like to see a comparison of the computational complexity involved in reconstruction.\n3) I would recommend citing the following paper:\nV. Boominathan, J. T. Robinson, L. Waller, and A. Veeraraghavan, “Recent advances in lensless imaging,” Optica 9(1), 1 (2022). This paper gives an overview of the common modulation schemes used for lenseless imaging, and I believe your approach is only applicable for phase modulation masks and not for amplitude modulation masks?", "limitations": "1) Although the authors mention that real-time reconstruction is not possible, it is unclear how much time reconstruction actually takes. It would be great if this is explained as well.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) The biggest strength of this paper lies in the quality of results displayed. The reconstructed images closely resemble the ground truth and are structurally very similar, unlike some of the other comparable methods.\n2) Extensive evaluation has been performed. The method has been tested on two different lensless imaging datasets, which helps build confidence. The method outperforms peers on 6 difference metrics, 3 for quality and 3 for photorealism. The evaluation has been conducted on both range space recovery method and null space recovery method, demonstrating the merit of this approach.\n3) Great clarity has been provided around the range space - null space decomposition, including mathematical derivations. \n4) The method seems reproducible because of the great details provided in the paper, for both range space and null space reconstruction.", "weaknesses": "1) The paper mentions usage of 3X3 PSF kernels for Deconvolution. They arrive at this conclusion using experiments with different sizes. However, no mathematical reasoning or intuition has been provided as to why this is a good choice, and under what circumstances, it will break down. Without this explanation, it is very difficult to reuse the same model for a dataset captured with a different lenseless system. More work/reasoning is needed to support this choice of sampling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Choice of the number of deconvolution kernels**\n\nThe rationale for using 3x3 PSF kernels is based on the assumption that the central PSF effectively represents the region of the field of view (FoV) where PSFs change smoothly, which is typical in lensless settings. The 3x3 choice balances performance and cost: by discretely sampling a small number of the central region and their corresponding PSFs, we approximate the continuous PSF across the entire FoV while minimizing computational and memory costs. \n\nFurthermore, opting for an odd number of PSFs (in both width and height dimensions) is recommended when employing a single calibrated PSF at the center of the FoV ($\\theta$ = 0). This choice ensures that the initialized deconvolution kernel at the FoV center aligns accurately with the calibrated PSF, establishing a reliable starting point for the deconvolution process. In contrast, using an even number of kernels may result in inaccurate initializations across all deconvolution kernels, potentially compromising the effectiveness of the deconvolution. With an odd number, at least one correct initialization can be assured.\n\n**2. Comparison of the computational complexity**\n\n| Method | FlatNet [15] | Le-ADMM-U [23]    | DDNM [37] | SVDeconv (first stage)| Null-space diffusion (second stage)  | PhoColens (two stages total) |\n|-|-|-|-|-|-|-|\n| **Inference time (in sec)** | 0.013 | 0.047 | 3.447 | 0.025 | 0.781 | 0.806| \n\nWe evaluated the computational efficiency of the proposed method on a machine equipped with Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz and RTX 4090 GPU. The above table presents the Inference time results. Future work will explore optimal trade-offs between computational speed and performance for various application scenarios.\n\n\n\n**3. Usage of WinnerDeconv**\n\nThanks for the reminder! We agree that we should use the correct term 'Wiener Deconvolution' in the main context to ensure clarity and accuracy and we apologize for any confusion caused.\n\n\n**4. Phase modulation masks v.s. amplitude modulation masks**\n\nOur model is applicable to all lensless cameras using a convolution imaging model, whether they employ phase modulation masks or amplitude modulation masks. However, for amplitude modulation lensless cameras like Flatcam [3], which utilize a separate imaging model, further exploration is needed to assess the effectiveness of the proposed method.\n\n**5. Other comments**\n\nIn the revised version, we will include a citation to the paper 'Recent Advances in Lensless Imaging,' which contributes to our research by offering a comprehensive review of the definition and evolution of lensless imaging.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VpqUd9sDyq", "reviewer": "Reviewer_Ernw", "summary": "This paper proposed a deep learning-based approach for lensless imaging. To address the problem of model mismatch that simple convolutional models cannot accurately describe the lensless imaging process, this paper introduced a spatially-varying devolution module that reweights the deconvolution results from multiple kernels using spaitally-varying weights. In addition, a two-stage model based on range-null space decomposition. In the first stage, a spatially-varying deconvolution network is designed to reconstruct the low-frequency content; The second stage uses the output of the previous stage as a condition to guide the pre-trained diffusion model in reconstructing fine high-frequency image details associated to the null space of the measurement operator. The experiments were conducted on two datasets: PhlatCam and DiffuserCam.", "questions": "1 . Why to simulate the PSF that has already been calibrated in the dataset, and what effect does the simulated PSF have in the algorithm?\n2. Section 3 mentions that model mismatch is mainly caused by the presence of the incident angle θ. However, in the first stage of weight calculation, the error caused by only considering distance. Why?\n3. What exactly does spatially-varying represent? How is the FoV center of the learnable kernel determined in weight calculation? What does the distance from point (u, v) to the FoV center exactly mean? Illustration with one figure could facilitate the understanding.\n4. How to set the size and quantity of the learnable kernels?", "limitations": "The paper mentioned several limitations, such as computational cost due to introducing diffusion-based sampling and two-stage processing, as well as false details introduced by the diffusion model.\n\nOne possible limitation that is not mentioned in the paper is, the two-stage framework may have the robustness issue, the error from the first stage may affect the accuracy in the second stage, compared to an iterative framework. I suggest the authors to have a discussion on this issue.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Introducing diffusion model to reconstruct fine details of high-frequency content associated to the null space the measurement operator.\n2. To some extent,  the spatially-varying deconvolution solved the problem of model mismatch.", "weaknesses": "1. Using diffusion model to recover the null-space related image information is not new. E,g,,\n[a] Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model. ICLR 2023.\n2. Lack of experimental or deductive analysis processes to demonstrate the effectiveness or accuracy of using range-null space decomposition to describe the process of lensless image restoration.\n3. The PSFs in lenless imaging are usually having a very large size (even larger than the image). Accordingly, the deconvolution kernels should be large. However, the deconvoluton kernel size seems small in the first stage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Comparison with DDNM**\n\nOur paper presents notable advancements compared to DDNM, in both the reconstruction quality and inference speed. Training-free methods like DDNM depend on an accurate imaging model to recover null space, but acquiring such a model for lensless imaging is difficult. In contrast, our null space diffusion learns also utilizes real-world training data pairs, improving its performance on real captures where the imaging formulation model is imperfect.\n As demonstrated in Tab. 3 and Fig. 9 in the original paper, our approach consistently outperforms DDNM on real captures. Additionally, DDNM requires 3x more time than ours for the reconstruction Under lensless imaging setting. Unlike DDNM, which requires complex calculations at each sampling step to maintain consistency with the original measurement, our feed-forward model minimizes additional computational overhead while delivering superior results. Moreover, we introduce a novel imaging model that enhances the accuracy of estimating range-space content, crucially improving the fidelity of final reconstruction.\n\n**2. Effectiveness of range-null space decomposition**\n\nIn Tab. 4 and Fig. 10 of the original paper, we conduct a comparative analysis of our diffusion model under various conditions. Specifically, for the SVD-OC method, we utilize a similar deconvolution approach to recover the original content, contrasting with our method that incorporates range-null space decomposition. The results consistently demonstrate our method's superiority over SVD-OC, highlighting the clear advantage of integrating range-null space decomposition in enhancing lensless image restoration.\n\n**3. Clarification about deconvolution kernel size**\n\nThe deconvolution kernel size is as large as the PSF size of the lensless camera. For example, in the PhlatCam dataset (1280 × 1480, same as the lensless measurement), and the DiffuserCam dataset (540 × 960, larger than the original lensless measurement). For DiffuserCam, we employ replicate padding to align the PSF size with the padded measurements, as detailed in our paper. This ensures our method effectively handles the large PSF sizes typical in lensless imaging, maintaining accuracy in image restoration.\n\n**4. The function of the simulated lensless camera and PSF**\n\nThe simulation of the lensless PSF is solely used to demonstrate the mismatch in widely used spatial invariant convolution models. We do not use the simulation in our lensless imaging experiments.  For lensless datasets captured in the real world we used, like the DiffuserCam and PhlatCam datasets, we only have a calibrated PSF in the center (zero angle of incidence light). Therefore, we use the calibrated PSF to initialize all the learnable deconvolution kernels.\n\n**5. Relation between the incident angle and the FoV center**\n\nIn a lensless setup with a 2-dimensional imaging plane, the Huygens-Fresnel principle [9] establishes a direct relationship between the incident angles ($\\theta$,  $\\phi$) and the center shift of the PSF — known as the focus center. Specifically, if the PSF center for zero angles of incidence light is located at ($c_x$, $c_y$), then for incident angles ($\\theta$,  $\\phi$), the PSF center approximately shifts to ($c_x - d \\sin \\theta$, $c_y - d \\sin \\phi$) according to the Fresnel propagation approximation [9], where $d$ denotes the mask-sensor distance. For detailed derivation, please refer to the Phlatcam paper [5], which we will further elaborate upon in the revised version. Reviewers can also refer to *Fig. a in the rebuttal PDF* for an illustration. This approach allows us to model the PSF variation across different focus centers on the imaging plane, reflecting the variation due to incident angles.\n\nIn our formulation, the center of the learnable kernel corresponds to a specific incident angle as well as coordinates on the imaging plane. We define this specific incident angle as the FoV center of the learnable kernel and the coordinates as the focus center of the kernel. Therefore, the weights can be determined by the distance between the pixel coordinates (u, v) and the focus center.\n\n**6. Robustness of the two-stage Framework**\n\nWe appreciate the concern about the potential robustness issues of a two-stage framework. Actually, our training scheme is designed to mitigate errors introduced by inaccurate image formation in the first stage. Our range-space reconstruction in the first stage is a deterministic process, focused on recovering information directly observable from the lensless measurement. This reduces the risk of introducing artifacts or errors that would hinder the subsequent diffusion process. Additionally, our SVDencov component is specifically designed to handle the challenges of lensless image reconstruction, improving the accuracy of the first stage. We conducted ablation studies (Tab. 4 and Fig. 10) comparing different intermediate outputs, demonstrating that using the full estimated image reconstruction as input to the diffusion model can indeed amplify first-stage errors. But our range-space-based approach alleviates this issue. This is because recovering range-space content is easier than the original content in the first stage, therefore we achieve better data fidelity and fewer artifacts. *Fig. b in the rebuttal PDF* also shows the data fidelity of our first stage.\n\nMoreover, iterative frameworks that project the output onto the lensless measurement space generally rely on precise imaging models, which are difficult to establish accurately in the context of lensless imaging. However, inaccurate modeling may introduce additional errors in the reconstruction process. Moreover, iterative methods often require more computational resources, such as DDNM and similar approaches.\n\nGiven these considerations, our method achieves a balanced trade-off between performance and efficiency.\n\n**7. Choice of the number of deconvolution kernels**\n\nPlease refer to the reply to Reviewer FAra (R1).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M7Q7AWT6YK", "reviewer": "Reviewer_pMG3", "summary": "This paper proposes a method for reconstructing photorealistic images in lensless imaging. The reconstruction process, which aims to be consistent with observations while achieving photorealism, is based on range-null space decomposition. To accommodate realistic cameras, the method introduces SVDeconv, which learns the deconvolution process with a spatially-varying PSF simultaneously. Additionally, the reconstruction in the null space is performed using a pretrained diffusion model. Quantitative and qualitative evaluations are conducted using two datasets, PhlatCam and DiffuserCam.", "questions": "If there are any misunderstandings in the weaknesses pointed out, please clarify them.", "limitations": "Limitations have been addressed well.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ For the generally challenging task of reconstructing high-frequency details in lensless imaging, introducing the concept of range-null space decomposition to achieve photorealistic and measurement-consistent image reconstruction is a very rational and technically sound approach.\n+ The idea of using a generative approach solely for reconstruction in the null space, rather than relying entirely on generative priors for restoration, addresses the issue of hallucination. This approach showcases originality.\n+ The effectiveness is also commendable as it achieves generally good results both quantitatively and qualitatively when compared with various other methods.", "weaknesses": "1. There is a lack of detail regarding the training process, making it difficult to understand correctly. It is unclear whether fine-tuning is performed using the input images at the test time or the PSF and parameters are freezed using a pretrained network.\n1. The method of dividing the Spatially-Varying PSF (SV-PSF) into a 3×3 grid is somewhat naive. Particularly, I doubt that the spatial dependency of the PSF is also influenced by the target scene depth, which does not appear to be considered.\n1. The analysis of the results is also insufficient. It is unclear how close the estimated PSF is to the accurately calibrated SV-PSF. The comparison between the reconstructed images using the accurate SV-PSF and the proposed method is not discussed. Additionally, for the dataset used in the evaluation, the inference results of the range-null content are not provided (as shown in Figure A2). The data fidelity in the reconstructed images is also not confirmed.\n+ Comment: In eq(4), $N\\times N$ is used, while previously $K\\times K$ was used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Details of the training process**\n\nIn the first stage, SVDeconv is trained using range-space content derived from ground truth images. SVDeconv consists of two main parameter components: a learnable deconvolution kernel initialized with known PSFs, and a U-Net initialized with standard weights without pretraining.  Once trained, SVDeconv processes input lensless measurements to estimate the range-space content of training samples. Subsequently, we use this estimated range-space content as input conditions for fine-tuning via null-space diffusion. During diffusion fine-tuning, we utilize a pre-trained diffusion model with frozen weights. We only train the supplementary conditioning modules like StableSR [35], to guide the reconstruction process effectively.\n\n**2. Variation of spatially-varying PSF related to the depth**\n\n\nIn our lensless camera setup, the scene-to-camera distance significantly exceeds the sensor size (less than 1cm), typically around 40cm, which is practical for everyday capture scenarios. This distance effectively disregards spatial variance along the depth axis (from 30cm to infinity), treating the PSF at such distances as equivalent to that from an infinitely distant point source. Our simulations validate this by showing a 0.995 similarity score between the PSF of a light point at 30cm and 100cm distances. Such scenarios are widely accepted in the relevant literature [15,17,23,43] and align with the lensless imaging datasets used in our study.\n\nConsideration of depth-dependent spatial variance in the PSF becomes critical only when the scene-to-camera distance is comparable to the camera size, typically between 1cm and 5cm. However, these scenarios require capturing scenes very close to our camera, which is beyond the scope of this paper. Future work will explore these 3D spatially varying PSF effects.\n\n**3. Comparison of reconstruction using accurate SV-PSF and our method**\n\nWe conducted an additional experiment below, which shows that our method achieves comparable performance to SV-deconvolution methods when accurate PSFs are provided.\n\nIn real-world lensless camera datasets like the PhlatCam and DiffuserCam, accurately calibrated Spatially-Varying Point Spread Functions (SV-PSFs) for different incident angles are typically unavailable. To validate the effectiveness of our proposed method, we simulated a dataset using the simulated lensless camera discussed in our paper, incorporating known SV-PSFs. This dataset comprises 2000 images with 20dB noise. We evaluate four methods for SV-deconvolution: 1)  spatially varying FISTA [a], 2) MultiWinnerNet [b] using a 3x3 grid of known PSFs, 3) our method using a single known PSF, and 4) our method using a 3x3 grid of known PSFs. Results in the following table demonstrate that our method achieves comparable performance to SV-deconv methods utilizing accurately calibrated Spatially-Varying Point Spread Functions (SV-PSFs).\n\n| Methods | PSNR | SSIM | LPIPS |\n|---|---|---|---|\n| Spatially-varying FISTA [a] | 24.19 | 0.787 | 0.288 |\n| MultiWinnerNet [b] (1 known PSF) | 24.72 | 0.796 | 0.273 |\n| MultiWinnerNet [b] (3x3 known PSFs) | 25.88 | 0.832 | 0.261 |\n| Ours (1 known PSF) | 25.47 | 0.811 | 0.265 |\n| **Ours (3x3 known PSFs)** | **26.02** | **0.837** | **0.258** |\n\n**4. Data fidelity in the reconstructed range-space contents**\n\nFor data fidelity in the reconstructed range-space contents, the results in Tab.2 of the original paper compare various deconvolution methods, showing that our approach achieves superior fidelity quantitatively. Reviewers can also refer to the qualitative examples in *Fig. b in the rebuttal PDF* for further confirmation.\n\n**5. Other comments**\n\nThank you for pointing out the $𝑁$×$𝑁$ typo, we will revise it.\n\n\n[a]. Yanny, Kyrollos, et al. \"Miniscope3D: optimized single-shot miniature 3D fluorescence microscopy.\" Light: Science & Applications 9.1 (2020): 171.\n\n[b]. Yanny, Kyrollos, et al. \"Deep learning for fast spatially varying deconvolution.\" Optica 9.1 (2022): 96-99.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QP8epbLx3v", "reviewer": "Reviewer_4jsT", "summary": "This paper addresses the challenge of spatially varying Point Spread Function (PSF) in lensless imaging and introduces a two-stage approach for reconstructing lensless images.", "questions": "1. Fig. 4(a) should be revised. Light coming from an off-axis angle should not be shown as being ‘refracted’ to produce parallel output light, as this is inconsistent with the proposed spatially-varying PSF model. Instead, I suggest that the author illustrate two point sources: one originating from the axis with output also parallel to the axis (in cyan), and another from an off-axis point with output also at an off-axis angle (in blue).\n\n2. In Fig. 1(1), vignetting is observed, while the reconstructed images show no signs of vignetting. It would be better to explain in the paper how the vignetting effect was removed.\n\n3. It would be beneficial to provide more explanations from the optical perspective in Section 3.2 to strengthen the paper's soundness in optics. The paraxial imaging model (spatially invariant convolution) is commonly used to simplify the forward imaging model, but it is inherently inaccurate, particularly for large field-of-view imaging. It would be good to add references to corresponding off-axis wave propagation models, such as “Shifted angular spectrum method for off-axis numerical propagation” by Matsushima, “Modeling Off-Axis Diffraction with the Least-Sampling Angular Spectrum Method” by Wei et al., and “Shifted band-extended angular spectrum method for off-axis diffraction calculation” by Zhang et al.\n\n4. Spatially varying PSFs have also been used in other applications with improved results. Therefore, I think it is important to add references to these works. For example, “Aberration-aware depth-from-focus” by Yang et al. demonstrated improvement in depth estimation when considering spatially varying PSFs. “Correcting Optical Aberration via Depth-Aware Point Spread Functions” by Luo et al. proposed spatially varying PSFs for optical aberration correction and depth estimation. Additionally, “High-Quality Computational Imaging through Simple Lenses” by Heide et al. proposed spatially varying PSF imaging for simple lenses.", "limitations": "The limitations of this paper are well discussed.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper is well-written, presenting a detailed statement of the problem, the physical imaging model, an innovative reconstruction method, and thorough experimental results. I believe this paper will significantly impact the fields of both lensless imaging and the broader area of spatially-varying computational imaging, given its contributions to both the imaging model and reconstruction methods. The demonstrated reconstruction results are impressive, showing significant improvements over existing methods.\n\nI believe this paper will make a significant contribution to both lensless imaging and the broader field of spatially varying imaging reconstruction. The proposed learnable spatially varying PSF will provide valuable insights for many other applications.", "weaknesses": "The only two weaknesses of this paper are: (1) the lack of real-world experiments, but this is mitigated by the use of a standard lensless imaging dataset for evaluation, and (2) the paper would benefit from incorporating more knowledge and discussion from the optics field to strengthen its soundness.\n\nThere are two suggestions to make this paper more robust: (1) Expand Fig. 4(b) and Fig. A1 to illustrate how PSFs change with different fields of view. (2) Expand Fig. 2 and Fig. 4(d) to better demonstrate the significance of considering spatially varying PSFs, possibly including pixel-wise error maps of the reconstruction results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Real-world experiments**\n\nThe DiffuserCam and PhlatCam datasets utilized in our study are collected from real-world lensless camera prototypes. These datasets are widely recognized and extensively used in lensless imaging literature, providing robust and reliable benchmarks for evaluation. Thus, while our study focuses on standard datasets, they effectively represent real-world scenarios and ensure a comprehensive evaluation of our proposed methods.\n\n**2. Discussion from the optics field**\n\nWe appreciate the reviewer's suggestion to incorporate deeper insights from optics. We agree that a more thorough optical analysis can strengthen our model's foundation. In the revised manuscript, we will provide a detailed derivation of the imaging model mismatch from an optical perspective to enhance the theoretical basis of our approach.\n\n**3. Mitigating vignetting in WinnerDeconv outputs**\n\nThe appearance of vignetting is primarily due to the limitations of the shift-invariant convolution model which motivates us to propose a spatial-varying formuation. Our method effectively addresses vignetting through two stages. First, the proposed SVDeconv component in the first stage corrects for the inaccuracies of the shift-invariant model, mitigating vignetting artifacts. Second, the subsequent neural network stage learns a natural image distribution that is typically devoid of vignetting. This observation is supported by the fact that even a trained U-Net architecture, as employed in FlatNet, exhibits the ability to reduce vignetting.\n\n**4. Revision on Fig. 4**\n\nWe appreciate the reviewer's suggestion. Figure 4(a) has been revised to accurately depict the light propagation as suggested. The updated figure is presented in *Fig. a of the rebuttal PDF*. \n\n**5. Other comments**\n\nThanks for the suggestions on adding references. We will incorporate the recommended references to enhance the discussion on the theoretical foundations and applications of spatially varying PSFs in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the time and effort dedicated by the ACs and reviewers in evaluating our work. We have carefully considered all comments and suggestions, and our detailed responses can be found in the rebuttal box below.

Meta-review: The paper received highly positive reviews, which became even stronger after the authors' rebuttal. Reviewers appreciated quality of results (4jsT, pMG3, FAra), the novelty (4jsT, pMG3), thorough evaluation (4jsT, FAra), and clear presentation (4jsT, FAra). Although there were initial concerns about missing important details and a lack of analysis, these issues were effectively addressed in the authors' rebuttal.

After carefully reviewing the paper, the reviews, and the authors' rebuttal, the AC agrees with the reviewers' highly positive consensus and therefore recommends the paper for acceptance (spotlight).

For the camera-ready version, the authors should ensure to incorporate all discussions from the rebuttal into the main paper and supplementary materials. The specific changes that need to be implemented are:

**1. Improved presentation**: Include a discussion from the optics field (4jsT) and a discussion on the variation of the spatially-varying PSF related to depth (pMG3). Revise Figure 4 to accurately illustrate light propagation (4jsT) and improve Table 2 for data fidelity in the reconstructed range-space contents (pMG3).

**2. Additional experiment**: Add a comparison of reconstruction using the accurate SV-PSF and the proposed method (pMG3).

**3. Other corrections**: Include additional references (4jsT, FAra) and correct the typo from "Winner" to "Wiener"  (FAra).
Decision: Accept (spotlight)