Title: PhoCoLens: Photorealistic and Consistent Reconstruction in Lensless Imaging
Venue: NeurIPS 2024
Accepted: yes
Keywords: Lensless Imaging; Computional Imaging;  Generative Prior
Abstract: Lensless cameras offer significant advantages in size, weight, and cost compared to traditional lens-based systems. Without a focusing lens, lensless cameras rely on computational algorithms to recover the scenes from multiplexed measurements. However, current algorithms struggle with inaccurate forward imaging models and insufficient priors to reconstruct high-quality images. To overcome these limitations, we introduce a novel two-stage approach for consistent and photorealistic lensless image reconstruction. The first stage of our approach ensures data consistency by focusing on accurately reconstructing the low-frequency content with a spatially varying deconvolution method that adjusts to changes in the Point Spread Function (PSF) across the camera's field of view. The second stage enhances photorealism by incorporating a generative prior from pre-trained diffusion models. By conditioning on the low-frequency content retrieved in the first stage, the diffusion model effectively reconstructs the high-frequency details that are typically lost in the lensless imaging process, while also maintaining image fidelity. Our method achieves a superior balance between data fidelity and visual quality compared to existing methods, as demonstrated with two popular lensless systems, PhlatCam and DiffuserCam.

PhoCoLens: Photorealistic and Consistent Reconstruction in Lensless Imaging

Abstract

Lensless cameras offer significant advantages in size, weight, and cost compared to traditional lens-based systems. Without a focusing lens, lensless cameras rely on computational algorithms to recover the scenes from multiplexed measurements. However, current algorithms struggle with inaccurate forward imaging models and insufficient priors to reconstruct high-quality images. To overcome these limitations, we introduce a novel two-stage approach for consistent and photorealistic lensless image reconstruction. The first stage of our approach ensures data consistency by focusing on accurately reconstructing the low-frequency content with a spatially varying deconvolution method that adjusts to changes in the Point Spread Function (PSF) across the camera's field of view. The second stage enhances photorealism by incorporating a generative prior from pre-trained diffusion models. By conditioning on the low-frequency content retrieved in the first stage, the diffusion model effectively reconstructs the high-frequency details that are typically lost in the lensless imaging process, while also maintaining image fidelity. Our method achieves a superior balance between data fidelity and visual quality compared to existing methods, as demonstrated with two popular lensless systems, PhlatCam and DiffuserCam. Project website: phocolens.github.io .

(a) Measurement

(b) WienerDeconv

(c) FlatNet

(d) FlatNet + DiffBIR

(e) Ours

(a) Measurement

(b) WienerDeconv

(c) FlatNet

(d) FlatNet + DiffBIR

(e) Ours

TA

1 Introduction

Lensless imaging systems [6] have emerged as a groundbreaking solution for ultra-compact, lightweight, and cost-effective imaging. They replace lenses by amplitude [3, 4] or phase masks [1, 5, 20] placed close to the sensor to modulate incoming light. This design significantly reduces camera size and weight and enables innovative sensor shapes such as spherical [14] or cylindrical.

Considering the raw measurements from lensless cameras are typically blurry and unrecognizable, it is hard to recover a high-quality image of the original scene. An example of lensless measurement and recovered images by WienerDeconv [44] is shown in the left column of Fig. 1. Due to the lack of focusing elements, lensless cameras cannot directly record the scene but encode it into a complex diffraction pattern. This encoding process can be approximated as a convolution with a large Point Spread Function (PSF). The PSF acts like a low-pass filter applied to the scene thus introducing ambiguity, which means there could be multiple possible recoveries for a single measurement. Therefore, powerful algorithms are critical for high-quality reconstruction in lensless systems.

The primary challenge of a lensless reconstruction is to achieve both photorealism and consistency. Photorealism requires high-quality reconstruction with rich details, while consistency further requires reconstructed contents to be consistent with the original scene. Traditional techniques, like WienerDeconv, can reconstruct images that align with the ground truth but visual quality is significantly degraded (Fig. 1b). Learning-based approaches such as FlatNet-gen [17] attempt to enhance visual quality by training with paired images and lensless measurements, yet they often fail to recover high-frequency details (Fig. 1c). The visual quality can be further improved by the restoration algorithms using generative priors, like DiffBIR [23] (Fig. 1d). While these restoration methods can inject rich details, they may also alter image content or insert non-existent objects, breaking consistency. For example, in Fig. 1d, the beak in the top row has the wrong shape compared to the ground truth (Fig. 1f), and the leaves in the bottom row look fake.

Moreover, an inaccurate imaging process simulation may also hurt data consistency. Most existing reconstruction algorithms [17, 27] simplify the imaging process as a convolution with a shift-invariant PSF. However, in practice, PSFs are spatially varying, particularly when the angle of incidence increases. Consequently, these areas experience a noticeable drop in the reconstructed similarity to the original scene, especially in the peripheral field of view.

To achieve both photorealism and consistency, we propose a two-stage lensless reconstruction based on range-null space decomposition [36]. According to this decomposition, the reconstructed image consists of two components, one from 'range space', which can be directly calculated from the lensless measurement (Fig. 1a), and the other from 'null space', which are the detailed textures lost during the lensless imaging process. Therefore, the first stage prioritizes data consistency by recovering the 'range space' component, which is the low-frequency content that matches the ground truth. The second stage focuses on photorealism by adding more high-frequency details from the 'null space' while maintaining the consistency established in the first stage.

In the first stage, to improve consistency, we propose a novel spatially varying deconvolution to reconstruct the 'range space' content. Unlike traditional methods that assume a shift-invariant PSF, our approach automatically adapts to spatial variations in the PSF across the camera's field of view in a data-driven manner. This innovation more accurately models the forward imaging process, leading to better reconstruction of structural integrity and detail in the low-frequency components. Fig. 2 shows improvement with our method (right) against those using a single kernel for deconvolution (left).

LPIPS

SSIM

MMCN

0.21

0.60

FlatNet-gen

FlatNet+DiffBIR

Ours

0.35

0.38

0-52

ManIQA360.34

020-014039

27.5

19.96-20.3920.53

2207PSNR

0.21

0.29

44.94

0.56

0.52

57.13

62.20

ClipIQA

MUSIQ

In the second stage, to improve photorealism, we integrate a generative prior using a pre-trained diffusion model, to insert realistic details into the first-stage output. This model specifically targets the recovery of high-frequency details lost in the lensless imaging process. We condition the diffusion model on the low-frequency content reconstructed in the first stage, guiding it to incorporate missing high-frequency elements from the 'null space'. Our supervised approach ensures the final images are consistent with actual measurements and also achieve photorealistic quality.

Combining these two stages, our lensless reconstruction achieves a good balance between data consistency and visual quality in the reconstructed images, surpassing existing methods, as shown in Fig. 1e. To validate this, we compare our method with others on two types of lensless cameras: PhlatCam [17] and DiffuserCam [27], using metrics that assess both fidelity and visual quality. The qualitative comparison on the PhlatCam dataset, demonstrating improvements in both aspects, is shown in Fig. 3, highlighting the superiority of our technique over current methods.

2 Related Work

Lensless imaging is traditionally addressed by solving regularized least squares problems for convolutional imaging models, often incorporating sparsity constraints like those in the gradient or frequency domain [1, 5, 21, 31]. Deep learning has revolutionized lensless imaging by enabling learnable deconvolution parameters and perception enhancement through image-to-image networks. Recent innovations include deep unrolled techniques [19, 27, 49], alongside feed-forward deconvolution methods in image space [17] or in feature space [22]. However, these methods generally assume a constant point spread function in the imaging process. Our proposal introduces a spatially varying deconvolution approach to overcome this limitation and achieve more precise image reconstruction.

Spatially-varying deconvolution has been a well-studied area [11, 24, 46] due to the prevalence of imaging systems with PSFs that vary across the FoV. However, these methods [2, 20, 25, 29, 47] are often slow, computationally intensive, and result in poor image quality, especially in complex systems. Recently, MultiWienerNet [48] introduced a deep learning approach for fast, spatially varying deconvolution, but it requires tedious multi-location PSF calibrations. In contrast, our spatially varying deconvolution method, designed for lensless imaging, leverages a single initial PSF and automatically learns variations across the image, eliminating the need for extensive calibration.

Inverse imaging with diffusion models can be categorized as supervised or zero-shot. Supervised methods train a conditional diffusion model [8, 13, 12, 32, 50] with paired images to bridge the gap between input and desired outputs, leveraging generative priors for inverse imaging [35, 39] and controlled generation [28, 34]. On the other hand, zero-shot methods [37] employ guidance to address a wide range of general inverse problems [9, 38]. These techniques typically rely on predefined conditions for guidance. Specific models like DDRM [15] and DDNM [41] focus on mathematical decompositions to improve the diffusion process. Our work integrates the strengths of both approaches, combining supervised fine-tuning with a theoretical framework based on range-null space decomposition, aiming for a robust solution to inverse imaging challenges.

3 Preliminary

3.1 Range-Null Space Decomposition

The lensless imaging process can be formulated as a linear transformation [10]. Specifically, given a scene plane x ∈ R M 2 , the sensor measurements ˆ y ∈ R N 2 is expressed as ˆ y = Ax + n . Here, A ∈ R N 2 × M 2 is the transfer matrix of the lensless imaging system, and n is the sensor noise. The transfer matrix A essentially encodes how light from each point on the scene plane contributes to each sensor pixel. Given a calibrated camera system, the transfer matrix A is known and the objective of lensless reconstruction is to recover the scene plane image x (like Fig. 1f) from the measurement (like Fig. 1a). For simplicity, below we consider a noise-free case as y = Ax .

We then introduce range-null space decomposition. Let A † ∈ R M 2 × N 2 be the pseudo-inverse of the linear matrix A , which satisfies AA † A ≡ A . The operation A † A can be interpreted as a projection onto the range space of A because for any sample x , we have AA † Ax = Ax . In contrast, the operation ( I -A † A ) acts as a projection onto the null space of A , given that A ( I -A † A ) x = 0 . Therefore, any sample x can be decomposed into two orthogonal components: one that lies in the

PSF at 0°

PSF at 15°

PSF at 30°

(a)

(b)

(c)

(d)

Propagation

𝑈𝜃

(𝜉, 𝜂, 0

+ )

𝑈(𝑥, 𝑦, 𝑑)

𝑧 = 𝑑

Mask

Sensor

𝜉

𝑥

𝜂

𝑦

𝜃

Point

Source A

Point

Source B

Shift

1.0

0.9

0.8

-30

-20

-10

0

10

20

30

Field Postion(°)

一

V

range space of A , and the other in the null space of A . Mathematically, this decomposition is:

where A † Ax is the range space component, and ( I -A † A ) x is the null space component.

Through this decomposition, we derive two distinct elements indicative of consistency and photorealism. The 'range space' term A † Ax ensures consistency, as its multiplication by A yields the measurement y , fulfilling the consistency condition Ax = y . Conversely, the 'null space' term ( I -A † A ) x ensures that the reconstructed image ˆ x aligns with natural image statistics.

3.2 Mismatch in Convolutional Lensless Imaging Model

In lensless literature, most researchers [1, 5, 17, 27] simplify the imaging process as a convolution, because the full transfer matrix A ∈ R N 2 × M 2 is too large to compute. Specifically, the measurement y is obtained as y = h ∗ x , where h is the point spread function (PSF) of the lensless system and ∗ denotes the convolution. Most previous reconstruction algorithms are based on this assumption.

However, the real lensless imaging model is not simply a spatially invariant convolution as shown in Fig. 4a. Consider a lensless camera consisting of a mask placed at the longitudinal position z = 0 and a sensor placed at z = d . Let U ( x, y, z ) be a complex scalar wave field, a complex function of transverse coordinates x, y , and longitudinal position z . Denote the wave field immediately after propagating through the mask from a point source at infinity with an incident angle θ as U θ ( ξ, η, 0 + ) . Using the Huygens-Fresnel principle [10], the intensity pattern p θ ( x, y ) captured by the sensor is:

where the distance r is given by r = √ d 2 +( x -ξ ) 2 +( y -η ) 2 and λ is the wavelength of light.

Therefore, according to Eqn. 2, the spatially invariant convolutional imaging model only satisfies under the Fresnel approximation [5, 10]:

This approximation is valid only when the distance d between the lensless mask and the sensor is large enough to satisfy d ≫ √ ( x -ξ ) 2 +( y -η ) 2 . However, most lensless masks are very close to the sensor ( d = 2mm in a typical lensless camera), breaking this assumption.

This mismatch can lead to inaccuracies [26, 43, 52] in the convolutional imaging model. To demonstrate that, we simulate the PSF of a typical lensless camera (Phlatcam [5]) across various incident angles θ using Eqn. (2). As shown in Fig. 4b, PSFs at different angles (0°, 15°, and 30°) are visually different. Quantitatively, Fig. 4c shows the similarity between PSFs at the center and -30° drops from 1.0 to 0.7.

Add &

UNet

Measurement

Stage 1 Output

Stage 2 Output

Spatially-varying Deconvolution (SVDeconv)

Learnable

Deconv Kernels

Add

Spatial

Weights

Weighted Deconv Results

Decov

Deconv Results

Ground Truth

Range Space Content

ℒ

!"##

ℒ

$%!&'

= ℒ

()'

+ℒ

#*+*)

Stage 2

: Conditional Diffusion

for

Null Space

Recon.

Stage 1

: SVDeconv

for

Range Space

Recon.

Diffusion

Condition

To further show the consequences of this mismatch, we simulate lensless capture from a clean scene image (Fig. 4d top), with the accurate spatially-varying PSF based on Eqn. (2), but solve the inverse imaging problem using a simple convolutional model using the PSF at the center. Fig. 4d bottom shows the result, and there are more artifacts at the boundary (red box) than at the center (green box). This is because the mismatch between the actual PSF and the single PSF used for deconvolution is more significant at the periphery field of view.

4 Method

In this section, we introduce PhoCoLens , a method for achieving both Pho torealistic and Co nsistent reconstruction in Lens less imaging. As illustrated in Fig. 5, PhoCoLens consists of two main stages: range space reconstruction and null space recovery. We will first introduce the whole framework and then provide a detailed explanation of each stage.

Given an input lensless measurement ˆ y = Ax + n , where y = Ax is the noise-free part of the lensless measurement, A is the transfer matrix of the lensless camera, x is the original scene we aim to reconstruct, and n is the sensor noise. The objective in lensless image reconstruction is to recover a photo-realistic image ˆ x which satisfies Aˆ x = y .

Inspired by the range null space decomposition in section 3.1, we can decompose any potential solution ˆ x into two orthogonal components: the range space component that maintains consistency and null space component that maximizes photorealism. Formally, this decomposition is given by ˆ x = A † Ax +( I -A † A ) ¯ x , where A † Ax is the range space component and ( I -A † A ) ¯ x is the null space component. Note that any choice of ¯ x satisfies the equation Aˆ x = y , as A ( I -A † A ) ¯ x = 0 .

With this decomposition, we reconstruct the range space and null space components in two stages, as illustrated in Fig. 5. In the first stage, we recover the range space content represented by A † Ax from the noisy lensless measurement ˆ y . Specifically, we propose SVDeconv , a physics-inspired S patiallyV arying Deconv olution Network that learns to reconstruct the range space content effectively. The second stage focuses on adding the null space content represented by ( I -A † A )¯ x . For this purpose, we introduce null space diffusion , a conditional diffusion model designed for null space recovery. By conditioning on the range space reconstruction obtained from the first stage, the null space diffusion enhances these images by incorporating the null space content, thereby rendering them with a more realistic appearance. Below we introduce each stage.

4.1 Spatially-varying Deconvolution Network

In the first stage, we use SVDeconv, a novel network designed to invert the forward lensless imaging model and reconstruct the range space content A † Ax from a lensless measurement ˆ y = Ax + n with noise n . SVDeconv is composed of two main components: a differentiable multi-kernel deconvolution layer and a refinement U-Net, as depicted in the gray box at the bottom left corner of Fig. 5.

Traditional lensless imaging methods often employ a single PSF to model the imaging system, which is inaccurate for large field of view (FoV) scenarios as analyzed in section 3.2. To address this, SVDeconv utilizes a set of learnable K × K PSF kernels, accounting for spatial variations across the field of view. Specifically, we partition the target image region into a grid of K × K segments, and for each, we apply an individual PSF kernel. This process results in K × K deconvolution operations, producing K × K intermediate deconvolved images, as shown in Fig. 5. The multi-kernel deconvolution is mathematically described as a Hadamard product in the Fourier domain:

where x ( i ) de is the i -th deconvolution results of the i -th learnable PSF kernel p ( i ) , F ( . ) and F -1 ( . ) are the DFT and the Inverse DFT operations, and ⊙ denotes the Hadamard product.

From this deconvolution operation, we obtain K × K intermediate deconvolved images. Assuming that each image accurately reconstructs a specific region of the target image corresponding to its PSF's field point, we propose to integrate these images into a single, unified intermediate image using an innovative interpolation method. The interpolation is expressed as:

where u, v are coordinates. The weight w i ( u, v ) for each deconvolved image x ( i ) de is inversely proportional to the distance between the point ( u, v ) and the focal center of each corresponding PSF, formally defined as:

In this way, it ensures that w i ( u, v ) normalizes the contribution of each deconvolved image based on inverse Euclidean distance to each point ( u i , v i ) , which represents the center of the FoV corresponding to the i -th learnable PSF kernel p ( i ) . This weighted sum effectively interpolates the intermediate images to form a unified representation with enhanced clarity and detail across the entire FoV.

One advantage of this weight design is that it significantly simplifies the calibration process. Previous multi-kernel methods, such as [48], often require multiple calibrated PSFs at different focal centers, leading to a time-consuming calibration process. On the contrary, our approach only requires one calibrated PSF to initialize all kernels. This is because we pre-define the center of the PSF and allow the model to learn its variations automatically.

After the interpolation, the intermediate image x int is fed into the refinement U-Net [33], removing noise and artifacts in the reconstruction to approximate the range space content A † Ax . We employ a combination of MSE loss and LPIPS loss [51] to train both the learnable PSFs and U-Net.

4.2 Null Space Content Recovery

With the reconstructed range space content, we aim to recover a final image that maintains consistency and improves photorealism. To ensure consistency, the difference between the final image and the range space content (residual content) should reside in the null space, which ensures the final image aligns with the original lensless measurement. To improve photorealism, the combination of the null space and range space content should appear as a realistic real-world image.

To achieve both objectives, we propose null-space diffusion, which takes the reconstructed range space content as the condition and generates an image that adheres to both constraints. It ensures the residual content lies in the null space while maintaining its ability to produce realistic images.

To ensure the outputs from null-space diffusion meet the consistency requirement, we train the model such that the residual content of these samples falls within the null space. Mathematically, if we denote the output as ˜ x , it should satisfy the following equation when conditioned by A † Ax :

Recall that A is the transfer matrix of the lensless camera and A † is its pseudo-inverse. This equation indicates that applying the operator A to the difference between the generated sample ˜ x and the original sample x results in zero. This ensures that the residual content resides in the null space of A .

Based on this, we design the null space diffusion as follows. Given an image x and its corresponding range space content c = A † Ax , we add noise progressively to the image resulting in a noisy image x t , where t is the noise addition iteration. Similar to other conditional diffusion models [50], we train a network ϵ θ to predict the noise added on the noisy image x t with the optimization objective:

which is derived to align with the goal in Eqn. (7). More details are in the Appendix.

Additionally, to ensure the null-space diffusion produces realistic images according to given conditions, we utilize a pre-trained diffusion model such as Stable Diffusion [32] with its weights frozen. We focus on training supplementary conditioning modules to guide the generative process using range space conditions while preserving its powerful ability to generate realistic images. Specifically, we follow the StableSR [40] structure, which involves using a conditional encoder to extract multi-scale features from the range space condition and then using them to modulate the intermediate feature maps of the residual blocks in the diffusion model. This modulation can align the generated images with the range space conditions, enhancing the fidelity and realism of output images.

5 Experiments

In this section, we assess the performance of our proposed method using two lensless imaging datasets, collected in real-world environments by two different kinds of lensless cameras: PhlatCam [5] and DiffuserCam [1]. PhlatCam employs a phase mask, while DiffuserCam utilizes a diffuser for its lensless mask. We compare our approach with other methods and conduct a comprehensive ablation study to evaluate the effectiveness of our design.

5.1 Dataset and Metrics

The PhlatCam dataset [17] contains 10,000 images across 1,000 classes resized to 384×384 pixels. Images are displayed and captured using a lensless PhlatCam [5], generating images at 1280 × 1480 pixels. We use 990 classes for training and 10 classes for testing, following the original protocol.

The DiffuserCam dataset [27] contains 25,000 paired images captured simultaneously using a standard lensed camera (ground truth) and a mask-based lensless camera DiffuserCam [1]. These pairs are split into 24,000 images for training and 1,000 for testing. Both cameras utilize sensors with a native resolution of 1080×1920 pixels, which are first downsampled to 270×480 pixels and then further cropped to a final resolution of 210×380 pixels for proper display.

To evaluate both consistency (fidelity) and photorealism (visual quality), we employ two metric sets:

Full-reference metrics to evaluate the consistency. Three full-reference metrics, PSNR, SSIM [42], and LPIPS [51], are used to evaluate the distance between the network output and ground truth, which indicates the fidelity of the reconstruction.

Non-reference metrics to evaluate the photorealism. Three non-reference metrics, ManIQA [45], ClipIQA [39], and MUSIQ [16], are used to evaluate the visual quality of reconstructed images.

5.2 Implementation Details

For SVDeconv, we utilize 3 × 3 PSF kernels for deconvolution. We initialize the 9 kernels using a single calibrated PSF from both the DiffuserCam and PhlatCam datasets. The U-Net component in SVDeconv is adapted from the U-Net used in Le-ADMM-U for training on the DiffuserCam, and from the U-Net in FlatNet-gen for the PhlatCam. We train the network for 100 epochs with a batch size of 5, using the Adam optimizer [18]. The learning rate is set to 3e-5 for U-Net training in both datasets. For deconvolution kernel training, the learning rate is set at 4e-9 for PhlatCam and 3e-5 for DiffuserCam. We set the MSE and LPIPS loss weights to be 1 and 0.05, respectively. In the DiffuserCam dataset, where the measurements are cropped by the sensor's limited size, we employ replicate padding [17] which expands the width and height of the measurements by a factor of two.

For null-space diffusion, we use the range space content reconstructed by SVDeconv as input conditions. We train it for 200 epochs, following the training and inference settings used in StableSR[39].

In the first stage, SVDeconv is trained using range-space content derived from ground truth images. SVDeconv consists of two main parameter components: a learnable deconvolution kernel initialized

with known PSFs, and a U-Net initialized with standard weights without pretraining. Once trained, SVDeconv processes input lensless measurements to estimate the range-space content of training samples. Subsequently, we use this estimated range-space content as input conditions for fine-tuning via null-space diffusion. During diffusion fine-tuning, we utilize a pre-trained diffusion model with frozen weights. We only train the supplementary conditioning modules like StableSR [35], to guide the reconstruction process effectively.

Le-ADMM-U

WienerDeconv

FlatNet-gen

Ours

FlatNet+DiffBIR

Le-ADMM-U

UPDN

Ours

FlatNet+DiffBIR

5.3 Comparison with Other Approaches

We evaluate the performance of our proposed method by comparing it against both traditional and learning-based approaches, using qualitative and quantitative measures on the two datasets.

We compare traditional methods like Tikhonov regularized reconstruction in the Fourier domain (WienerDeconv [44]) and total variation regularization via ADMM [7]. Additionally, we evaluate learning-based methods like the unrolled network Le-ADMM-U [27], MMCN [49], and UPDN [19] and the feedforward deconvolution method FlatNet-gen. Additionally, we assess two diffusion-based methods: one uses a pre-trained Stable Diffusion [32] with a zero-shot inverse imaging sampling method DDNM+ [41], and another enhances outputs from FlatNet-gen using a pre-trained blind

SVDenconv(Ours)

FlatNet-gen

MultiWinenerNet

Method

PSNRSSIMLPIPS

↓

ManIQAClipIQAMUSIQ

StableSR [39] 14.93 0.446

0.624

0.64

0.235

33.84

DiffBIR [23]

16.21 0.432

0.502

0.353

0.512

56.65

DDNM[41]

17.62 0.539

0.517

0.148

0.201

27.82

Ours

22.07 0.601

0.215

0.357

0.565

62.20

Condition

PhlatCam

DiffuserCam

PSNR

SSIM

LPIPS

↓

PSNR

SSIM

LPIPS

↓

WienerDeconv

20.38

0.551

0.301

20.42

0.632

0.282

FlatNet-gen

20.89

0.556

0.286

21.67

0.647

0.230

SVD-OC

21.16

0.572

0.276

22.43

0.679

0.185

Ours

22.07

0.601

0.215

24.12

0.748

0.161

DDNM

Ours

DiffBIR

FlatNet-gen

Ours

SVD-OC

image restoration model DiffBIR [23]. Quantitative results are summarized in Tab. 1. Qualitative comparisons for PhlatCam and DiffuserCam are shown in Figures 6 and 7, respectively.

On the PhlatCam dataset, our method achieves the best performance on both fidelity (full-reference metrics) and visual quality (non-reference metrics) according to Tab. 1, This is further supported by the qualitative results in Fig. 6, where our reconstruction are the closest to the ground truth.

For the DiffuserCam dataset, our method achieves superior performance in consistency metrics (full-reference metrics, SSIM, and LPIPS). While UPDN achieves a slightly higher PSNR, the visual quality of its outputs suffers from artifacts, as shown in Fig. 7. This slight PSNR drop in our method is likely due to artifacts within the DiffuserCam ground truth itself, such as moiré patterns. These artifacts can penalize high-quality images with a lower PSNR, despite their good visual consistency. This is further evidenced by the visual comparison in Fig. 7. Conversely, FlatNet+DiffBIR, though achieving higher quality metrics (non-reference metrics, ManIQA, ClipIQA, and MUSIQ), fails to maintain data consistency (low full-reference metrics, PSNR, SSIM, and LPIPS). Also, as shown Fig. 7, FlatNet+DiffBIR sometimes badly distorts the original image content, like transferring a human hand into clothes in the top row. Overall, our method achieves the best trade-off between the data consistency and the visual quality. And thanks to the introduction of SVDencov, our method can reconstruct the field-of-view periphery better than other methods.

5.4 Analysis and Discussion

Effectiveness of Spatially-varying Deconvolution. To verify the effectiveness of the proposed SVDeconv, we compare it with other learnable deconvolution approaches. Baselines include Le-ADMM-U (a deep unrolled network implementing ADMM), SingleDeconv (using a single deconvolution kernel), and MultiWienerNet [48](employing multiple deconvolution kernels without weighted interpolation, initialized similarly to our approach with a single PSF). We focus on two types of content reconstruction: range space content reconstruction, where range space content is the target, and original content

reconstruction, using the original ground truth as the target. For a fair comparison, all methods utilize the same U-Net for denoising and perceptual enhancement and the same loss for training. Results in Tab. 2 show the efficacy of the proposed spatially-varying deconvolution in the faithful reconstruction of both range space content and original content. Further, Fig. 8 shows our method outperforms others, particularly in reconstructing fine details at the periphery (circled by white dashed boxes).

Effectiveness of Null-space Diffusion . Following the reconstruction of range space content, we perform null space content recovery conditioned on the previously reconstructed range space content. We compare our null-space diffusion model with other approaches, including conditional diffusion models like DiffBIR [23] and StableSR [39], as well as the zero-shot inverse imaging method DDNM[41]. Quantitative results in Tab. 3 and qualitative comparison in Fig. 9 demonstrate that our null-space diffusion performs the best in both consistency with the ground truth and visual quality.

Effectiveness of Range Content Conditions . One important design of our framework is to use the first stage reconstruction as the condition for the second stage diffusion. To evaluate the effectiveness of these conditions, we perform an ablation study where we replace our reconstructed range content with the outputs from WienerDeconv, FlatNet-gen, and the SVD-OC (SVDencov trained with original content), and re-train the conditional diffusion models using these conditions. As evident from Fig. 10, our range content condition significantly improves reconstruction consistency compared to others. This is because using output from models like FlatNet-gen introduces additional artifacts that hinder the second stage from recovering the original image. Furthermore, quantitative analysis in Tab. 4 confirms this observation. Our method outperforms these conditions in all fidelity metrics.

6 Conclusion and Limitation

We present PhoCoLens, an innovative two-stage approach for achieving photorealistic and consistent reconstructions in lensless imaging. The method leverages the strengths of two complementary approaches: spatially varying deconvolution for the range space ensures consistency, while null-space diffusion for the null space guarantees photorealism. Our experiments on two lensless cameras demonstrate PhoCoLens' effectiveness in reconstructing realistic scenes while preserving fidelity. One limitation of our approach is that the two-stage nature of PhoCoLens and the diffusion model's sampling time hinder its real-time applicability. Additionally, although PhoCoLens achieves the best fidelity, the diffusion model we used may still introduce high-frequency details that deviate from the original scene, particularly for smooth scenes lacking in detail. Future work will focus on two main aspects: accelerating the diffusion model sampling process to enable real-time photo and video capture with lensless cameras, and exploring 3D spatially varying point spread function (PSF) effects to capture 3D scene information.

Acknowledgment

Wethank the reviewers for their valuable comments. This work is supported by the Shanghai Artificial Intelligence Laboratory, the RGC Early Career Scheme (ECS) No. 24209224, and CUHK Direct Grants (RCFUS) No. 4055189.

References

Nick Antipa, Grace Kuo, Reinhard Heckel, Ben Mildenhall, Emrah Bostan, Ren Ng, and Laura Waller. Diffusercam: lensless single-exposure 3d imaging. Optica , 5(1):1-9, 2018.

Muthuvel Arigovindan, Joshua Shaevitz, John McGowan, John W Sedat, and David A Agard. A parallel product-convolution approach for representing depth varying point spread functions in 3d widefield microscopy based on principal component analysis. Optics express , 18(7):64616476, 2010.

MSalman Asif, Ali Ayremlou, Aswin Sankaranarayanan, Ashok Veeraraghavan, and Richard G Baraniuk. Flatcam: Thin, lensless cameras using coded aperture and computation. IEEE Transactions on Computational Imaging , 3(3):384-397, 2016.

Eric Bezzam, Martin Vetterli, and Matthieu Simeoni. Privacy-enhancing optical embeddings for lensless classification. arXiv preprint arXiv:2211.12864 , 2022.

Vivek Boominathan, Jesse K Adams, Jacob T Robinson, and Ashok Veeraraghavan. Phlatcam: Designed phase-mask based thin lensless camera. IEEE Transactions on Pattern Analysis and Machine Intelligence , 42(7):1618-1629, 2020.

Vivek Boominathan, Jacob T Robinson, Laura Waller, and Ashok Veeraraghavan. Recent advances in lensless imaging. Optica , 9(1):1-16, 2022.

Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning , 3(1):1-122, 2011.

Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.02938 , 2021.

Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems , 35:25683-25696, 2022.

Joseph W Goodman. Introduction to Fourier optics . Roberts and Company publishers, 2005.

Felix Heide, Mushfiqur Rouf, Matthias B Hullin, Bjorn Labitzke, Wolfgang Heidrich, and Andreas Kolb. High-quality computational imaging through simple lenses. ACM Transactions on Graphics (ToG) , 32(5):1-14, 2013.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Yi Hua, Yongyi Zhao, and Aswin C Sankaranarayanan. Angle sensitive pixels for lensless imaging on spherical sensors. arXiv preprint arXiv:2306.15953 , 2023.

Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems , 35:23593-23606, 2022.

Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer. In Proceedings of the IEEE/CVF international conference on computer vision , pages 5148-5157, 2021.

Salman Siddique Khan, Varun Sundar, Vivek Boominathan, Ashok Veeraraghavan, and Kaushik Mitra. Flatnet: Towards photorealistic scene reconstruction from lensless measurements. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(4):1934-1948, 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Oliver Kingshott, Nick Antipa, Emrah Bostan, and Kaan Ak¸ sit. Unrolled primal-dual networks for lensless cameras. Optics Express , 30(26):46324-46335, 2022.

Grace Kuo, Fanglin Linda Liu, Irene Grossrubatscher, Ren Ng, and Laura Waller. On-chip fluorescence microscopy with a random microlens diffuser. Optics Express , 28(6):8384-8399, 2020.

Chengbo Li, Wotao Yin, Hong Jiang, and Yin Zhang. An efficient augmented lagrangian method with applications to total variation minimization. Computational Optimization and Applications , 56:507-530, 2013.

Ying Li, Zhengdai Li, Kaiyu Chen, Youming Guo, and Changhui Rao. Mwdns: reconstruction in multi-scale feature spaces for lensless imaging. Optics Express , 31(23):39088-39101, 2023.

Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Ben Fei, Bo Dai, Wanli Ouyang, Yu Qiao, and Chao Dong. Diffbir: Towards blind image restoration with generative diffusion prior. arXiv preprint arXiv:2308.15070 , 2023.

Jun Luo, Yunfeng Nie, Wenqi Ren, Xiaochun Cao, and Ming-Hsuan Yang. Correcting optical aberration via depth-aware point spread functions. IEEE transactions on pattern analysis and machine intelligence , 2024.

Elie Maalouf, Bruno Colicchio, and Alain Dieterlen. Fluorescence microscopy threedimensional depth variant point spread function interpolation using zernike moments. JOSA A , 28(9):1864-1870, 2011.

Kyoji Matsushima. Shifted angular spectrum method for off-axis numerical propagation. Optics Express , 18(17):18453-18463, 2010.

Kristina Monakhova, Joshua Yurtsever, Grace Kuo, Nick Antipa, Kyrollos Yanny, and Laura Waller. Learned reconstructions for practical mask-based lensless imaging. Optics express , 27(20):28075-28090, 2019.

Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 4296-4304, 2024.

Nurmohammed Patwary and Chrysanthe Preza. Image restoration for three-dimensional fluorescence microscopy using an orthonormal basis for efficient representation of depth-variant point-spread functions. Biomedical optics express , 6(10):3826-3841, 2015.

Ken Perlin. Improving noise. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques , pages 681-682, 2002.

Dikpal Reddy, Ashok Veeraraghavan, and Rama Chellappa. P2c2: Programmable pixel compressive camera for high speed imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 329-336. IEEE, 2011.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACMSIGGRAPH 2022 conference proceedings , pages 1-10, 2022.

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE transactions on pattern analysis and machine intelligence , 45(4):4713-4726, 2022.

Johannes Schwab, Stephan Antholzer, and Markus Haltmeier. Deep null space learning for inverse problems: convergence analysis and rates. Inverse Problems , 35(2):025008, 2019.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. arXiv preprint arXiv:2111.08005 , 2021.

Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 2555-2563, 2023.

Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. arXiv preprint arXiv:2305.07015 , 2023.

Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In International Conference on Learning Representations , 2022.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600612, 2004.

Haoyu Wei, Xin Liu, Xiang Hao, Edmund Y Lam, and Yifan Peng. Modeling off-axis diffraction with the least-sampling angular spectrum method. Optica , 10(7):959-962, 2023.

Norbert Wiener. Extrapolation, interpolation, and smoothing of stationary time series: with engineering applications . The MIT press, 1949.

Sidi Yang, Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujiu Yang. Maniqa: Multi-dimension attention network for no-reference image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1191-1200, 2022.

Xinge Yang, Qiang Fu, Mohamed Elhoseiny, and Wolfgang Heidrich. Aberration-aware depthfrom-focus. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Kyrollos Yanny, Nick Antipa, William Liberti, Sam Dehaeck, Kristina Monakhova, Fanglin Linda Liu, Konlin Shen, Ren Ng, and Laura Waller. Miniscope3d: optimized single-shot miniature 3d fluorescence microscopy. Light: Science & Applications , 9(1):171, 2020.

Kyrollos Yanny, Kristina Monakhova, Richard W Shuai, and Laura Waller. Deep learning for fast spatially varying deconvolution. Optica , 9(1):96-99, 2022.

Tianjiao Zeng and Edmund Y Lam. Robust reconstruction with deep learning to handle model mismatch in lensless imaging. IEEE Transactions on Computational Imaging , 7:1080-1092, 2021.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3836-3847, 2023.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Wenhui Zhang, Hao Zhang, Kyoji Matsushima, and Guofan Jin. Shifted band-extended angular spectrum method for off-axis diffraction calculation. Optics Express , 29(7):10089-10103, 2021.

Appendix

A PhlatCam Simluation

In section 3.2, we simulate a PhlatCam to interpret the detail of mismatch in a convolutional lensless imaging model. Here we explain how to simulate a lensless camera like PhlatCam.

Following previous work [5], our simulation begins by constructing a desired point spread function (PSF) based on the contours generated from random Perlin noise [30]. Given a specific distance between the mask and sensor, we optimize the phase mask to produce a PSF that matches the target. For this optimization, we employ the near-field phase retrieval algorithm (NfPR) [5]. This iterative algorithm alternates between the fields at the mask plane and the sensor plane, enforcing specific constraints at each location: the field amplitude at the mask plane is set to unity, and the intensity of the field at the sensor plane conforms to the target PSF. The algorithm utilizes forward Fresnel propagation for computing the field from the mask to the sensor plane and applies backward Fresnel propagation to compute from the sensor back to the mask. The inputs required for phase mask optimization include the target PSF, the mask-to-sensor distance (1mm in our setup), and the light wavelength, which in this case is set to 532 nm, a typical wavelength in the mid-visible spectrum.

Target PSF

Optimized Phase Mask

Simulated PSF

3

Fig. A1 illustrates the target PSF, the optimized phase mask, and the simulated PSF resulting from the Fresnel propagation method applied to the designed phase mask. Given a desired phase mask, we can accurately simulate the lensless imaging process under various situations.

B Range Space Content Approximation

In the first stage of our method (section 4.1), we want to reconstruct the range-space projection A † Ax from the lensless measurement ˆ y . This reconstruction requires the known target A † Ax corresponding to the given image x . However, for real-world lensless imaging systems, directly obtaining the transfer matrix A ∈ R N 2 × M 2 is computationally infeasible due to its considerable size. Consequently, we resort to simulating the realistic lensless imaging process using an approximate form of A and emulate its pseudoinverse A † by inverting this approximation.

In cases where a single point spread function (PSF) h is calibrated, we approximate the forward imaging process as follows:

where C denotes the cropping operation due to the limited sensor size, ∗ represents the convolution operator, and n represents both sensor and quantization noise. In our practical implementation, the simulated y sim is quantized to 12 bits, and sensor noise is introduced to achieve a signal-to-noise ratio (SNR) of 30.

The approximation for A † is derived by reversing the forward process, specifically by applying replicate padding to counteract the cropping and employing a Wiener deconvolution to reverse the convolution:

where the P represent replicate padding and the D represent Wiener deconvolution.

If multiple calibrated PSFs are available across different fields of view (FoV), the imaging forward process is better approximated by employing a low-rank model to mimic the spatially varying convolution. Correspondingly, the spatially varying deconvolution model discussed in section 4.1 could be utilized to simulate A † .

In our PhlatCam and DiffuserCam setups, we only have a single calibrated PSF. Therefore, we constitute the range-space content of image x using the simple convolutions. It's important to note that even without multiple calibrated PSFs for a more accurate simulation, the generated range-space content remains a good approximation of the real counterpart. As shown in Section 5.4, our simulation still serves as better supervision compared to the original image content in the first stage.

Range space

content

Original

NewSlideRule

Like

Ground truth

Range space

content

Estimated range

space content

FlatNet output

Fig. A2 shows the visual examples of the original ground truth x and the corresponding simulated range space content A † Ax of PhlatCam. It is shown that the range of space content A † Ax keeps the low-frequency structure of the original scene and lost some high-frequency details from the lensless imaging process.

Fig. A3 shows the visual examples of range space content estimated by our method. The outputs from SVDeconv exhibit high visual consistency with the ground truth, as highlighted in the green boxes, in contrast to those from FlatNet. FlatNet alters the original scene's content and introduces incorrect high-frequency details. Conversely, our method preserves only the low-frequency content in the range space, aligning closely with the original ground truth.

C Training Target for the Null-space Diffusion Model

In section 6, we establish that the goal articulated in Eqn. (7) corresponds directly to the optimization objective described in Eqn. (8) used for diffusion training. We provide a detailed derivation to support this equivalence.

Diffusion models [32] can be characterized by a signal-to-noise ratio defined by two sequences ( α t ) T t =1 and ( σ t ) T t =1 . Given a data sample x 0 , the forward diffusion process q can be expressed as:

where N denotes the normal distribution.

The Markov property of the diffusion is given as:

for s < t , with:

Given x t = α t x 0 + σ t ϵ , and a range space condition c = A † Ax , the diffusion output ˜ x can be modeled by ˜ x = x θ ( x t , c , t ) . To ensure reconstruction consistency such that A ( ˜ x -x 0 ) = 0 in Eqn. (7), the optimization objective is defined as:

Following previous work [12], we employ the reparameterization of the reconstruction term as a denoising objective:

to minimize:

which is exactly the Eqn. (8).

Due to the substantial computational cost associated with simulating the transfer matrix A , an alternative optimization objective can be employed in practice. Specifically, we can approximate the goal described in Eqn. (8) using the following optimization formulation:

D Qualitative Results

We present additional qualitative results demonstrating the superior performance of our methods in Fig. A4 for the PhlatCam dataset and in Fig. A5 for the DiffuserCam dataset.

Le-ADMM-U

WienerDeconv

FlatNet-gen

Ours

FlatNet+DiffBIR

Le-ADMM-U

UPDN

Ours

FlatNet+DiffBIR

E Broader Impacts

Lensless imaging algorithms represent a significant advancement in camera technology, offering both promising benefits and potential drawbacks for society. On the positive side, these algorithms facilitate the development of cameras that are smaller, lighter, and more energy-efficient. This can greatly expand their applicability in various fields, particularly in medical contexts where the miniaturization of imaging devices is crucial. For example, tiny lensless cameras could be integrated into endoscopic tools for less invasive procedures, improving patient outcomes and recovery times. Furthermore, such compact cameras could be used in wearable technologies to monitor health conditions continuously, providing real-time data to both patients and healthcare providers.

However, the proliferation of small, lensless cameras also introduces potential negative societal impacts, particularly concerning privacy. The ease with which these discreet cameras can be incorporated into everyday environments raises the possibility of misuse, such as unauthorized surveillance or covert photography. This concern is particularly acute in private spaces where individuals expect a high level of privacy. As such, while lensless imaging technology promises significant advancements in fields like healthcare, it also necessitates stringent regulations and ethical guidelines to prevent invasive use and protect individual privacy rights.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately reflect the contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are discussed in section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: For details on reproducing the experimental results, please refer to section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code and data will be available on the project website.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The implementation details of the experiments are provided in section 5.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We report the experiment results using the standard conventions in lensless imaging research, consistent with previous studies.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Evrey experiment can be completed within two days using a server equipped with eight NVIDIA RTX 3090 GPUs, a 64-core CPU, 256GB of RAM, and 1TB of storage.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We strictly adhere to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The broader impacts of our work are explained in appendix E.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We will not release data or models that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all original papers and make sure that our usage is legal.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not introduce new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of spatially varying Point Spread Function (PSF) in lensless imaging and introduces a two-stage approach for reconstructing lensless images.\n\nSTRENGTHS:\nThis paper is well-written, presenting a detailed statement of the problem, the physical imaging model, an innovative reconstruction method, and thorough experimental results. I believe this paper will significantly impact the fields of both lensless imaging and the broader area of spatially-varying computational imaging, given its contributions to both the imaging model and reconstruction methods. The demonstrated reconstruction results are impressive, showing significant improvements over existing methods.\n\nI believe this paper will make a significant contribution to both lensless imaging and the broader field of spatially varying imaging reconstruction. The proposed learnable spatially varying PSF will provide valuable insights for many other applications.\n\nWEAKNESSES:\nThe only two weaknesses of this paper are: (1) the lack of real-world experiments, but this is mitigated by the use of a standard lensless imaging dataset for evaluation, and (2) the paper would benefit from incorporating more knowledge and discussion from the optics field to strengthen its soundness.\n\nThere are two suggestions to make this paper more robust: (1) Expand Fig. 4(b) and Fig. A1 to illustrate how PSFs change with different fields of view. (2) Expand Fig. 2 and Fig. 4(d) to better demonstrate the significance of considering spatially varying PSFs, possibly including pixel-wise error maps of the reconstruction results.\n\nQUESTIONS:\n1. Fig. 4(a) should be revised. Light coming from an off-axis angle should not be shown as being ‘refracted’ to produce parallel output light, as this is inconsistent with the proposed spatially-varying PSF model. Instead, I suggest that the author illustrate two point sources: one originating from the axis with output also parallel to the axis (in cyan), and another from an off-axis point with output also at an off-axis angle (in blue).\n\n2. In Fig. 1(1), vignetting is observed, while the reconstructed images show no signs of vignetting. It would be better to explain in the paper how the vignetting effect was removed.\n\n3. It would be beneficial to provide more explanations from the optical perspective in Section 3.2 to strengthen the paper's soundness in optics. The paraxial imaging model (spatially invariant convolution) is commonly used to simplify the forward imaging model, but it is inherently inaccurate, particularly for large field-of-view imaging. It would be good to add references to corresponding off-axis wave propagation models, such as “Shifted angular spectrum method for off-axis numerical propagation” by Matsushima, “Modeling Off-Axis Diffraction with the Least-Sampling Angular Spectrum Method” by Wei et al., and “Shifted band-extended angular spectrum method for off-axis diffraction calculation” by Zhang et al.\n\n4. Spatially varying PSFs have also been used in other applications with improved results. Therefore, I think it is important to add references to these works. For example, “Aberration-aware depth-from-focus” by Yang et al. demonstrated improvement in depth estimation when considering spatially varying PSFs. “Correcting Optical Aberration via Depth-Aware Point Spread Functions” by Luo et al. proposed spatially varying PSFs for optical aberration correction and depth estimation. Additionally, “High-Quality Computational Imaging through Simple Lenses” by Heide et al. proposed spatially varying PSF imaging for simple lenses.\n\nLIMITATIONS:\nThe limitations of this paper are well discussed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for reconstructing photorealistic images in lensless imaging. The reconstruction process, which aims to be consistent with observations while achieving photorealism, is based on range-null space decomposition. To accommodate realistic cameras, the method introduces SVDeconv, which learns the deconvolution process with a spatially-varying PSF simultaneously. Additionally, the reconstruction in the null space is performed using a pretrained diffusion model. Quantitative and qualitative evaluations are conducted using two datasets, PhlatCam and DiffuserCam.\n\nSTRENGTHS:\n+ For the generally challenging task of reconstructing high-frequency details in lensless imaging, introducing the concept of range-null space decomposition to achieve photorealistic and measurement-consistent image reconstruction is a very rational and technically sound approach.\n+ The idea of using a generative approach solely for reconstruction in the null space, rather than relying entirely on generative priors for restoration, addresses the issue of hallucination. This approach showcases originality.\n+ The effectiveness is also commendable as it achieves generally good results both quantitatively and qualitatively when compared with various other methods.\n\nWEAKNESSES:\n1. There is a lack of detail regarding the training process, making it difficult to understand correctly. It is unclear whether fine-tuning is performed using the input images at the test time or the PSF and parameters are freezed using a pretrained network.\n1. The method of dividing the Spatially-Varying PSF (SV-PSF) into a 3×3 grid is somewhat naive. Particularly, I doubt that the spatial dependency of the PSF is also influenced by the target scene depth, which does not appear to be considered.\n1. The analysis of the results is also insufficient. It is unclear how close the estimated PSF is to the accurately calibrated SV-PSF. The comparison between the reconstructed images using the accurate SV-PSF and the proposed method is not discussed. Additionally, for the dataset used in the evaluation, the inference results of the range-null content are not provided (as shown in Figure A2). The data fidelity in the reconstructed images is also not confirmed.\n+ Comment: In eq(4), $N\\times N$ is used, while previously $K\\times K$ was used.\n\nQUESTIONS:\nIf there are any misunderstandings in the weaknesses pointed out, please clarify them.\n\nLIMITATIONS:\nLimitations have been addressed well.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a deep learning-based approach for lensless imaging. To address the problem of model mismatch that simple convolutional models cannot accurately describe the lensless imaging process, this paper introduced a spatially-varying devolution module that reweights the deconvolution results from multiple kernels using spaitally-varying weights. In addition, a two-stage model based on range-null space decomposition. In the first stage, a spatially-varying deconvolution network is designed to reconstruct the low-frequency content; The second stage uses the output of the previous stage as a condition to guide the pre-trained diffusion model in reconstructing fine high-frequency image details associated to the null space of the measurement operator. The experiments were conducted on two datasets: PhlatCam and DiffuserCam.\n\nSTRENGTHS:\n1. Introducing diffusion model to reconstruct fine details of high-frequency content associated to the null space the measurement operator.\n2. To some extent,  the spatially-varying deconvolution solved the problem of model mismatch.\n\nWEAKNESSES:\n1. Using diffusion model to recover the null-space related image information is not new. E,g,,\n[a] Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model. ICLR 2023.\n2. Lack of experimental or deductive analysis processes to demonstrate the effectiveness or accuracy of using range-null space decomposition to describe the process of lensless image restoration.\n3. The PSFs in lenless imaging are usually having a very large size (even larger than the image). Accordingly, the deconvolution kernels should be large. However, the deconvoluton kernel size seems small in the first stage.\n\nQUESTIONS:\n1 . Why to simulate the PSF that has already been calibrated in the dataset, and what effect does the simulated PSF have in the algorithm?\n2. Section 3 mentions that model mismatch is mainly caused by the presence of the incident angle θ. However, in the first stage of weight calculation, the error caused by only considering distance. Why?\n3. What exactly does spatially-varying represent? How is the FoV center of the learnable kernel determined in weight calculation? What does the distance from point (u, v) to the FoV center exactly mean? Illustration with one figure could facilitate the understanding.\n4. How to set the size and quantity of the learnable kernels?\n\nLIMITATIONS:\nThe paper mentioned several limitations, such as computational cost due to introducing diffusion-based sampling and two-stage processing, as well as false details introduced by the diffusion model.\n\nOne possible limitation that is not mentioned in the paper is, the two-stage framework may have the robustness issue, the error from the first stage may affect the accuracy in the second stage, compared to an iterative framework. I suggest the authors to have a discussion on this issue.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel two-stage approach for lensless image reconstruction. The approach ensures data consistency with a spatially varying deconvolution method and enhances photorealism using pre-trained diffusion models. This method outperforms existing techniques in data fidelity and visual quality on PhlatCam and DiffuserCam systems.\n\nSTRENGTHS:\n1) The biggest strength of this paper lies in the quality of results displayed. The reconstructed images closely resemble the ground truth and are structurally very similar, unlike some of the other comparable methods.\n2) Extensive evaluation has been performed. The method has been tested on two different lensless imaging datasets, which helps build confidence. The method outperforms peers on 6 difference metrics, 3 for quality and 3 for photorealism. The evaluation has been conducted on both range space recovery method and null space recovery method, demonstrating the merit of this approach.\n3) Great clarity has been provided around the range space - null space decomposition, including mathematical derivations. \n4) The method seems reproducible because of the great details provided in the paper, for both range space and null space reconstruction.\n\nWEAKNESSES:\n1) The paper mentions usage of 3X3 PSF kernels for Deconvolution. They arrive at this conclusion using experiments with different sizes. However, no mathematical reasoning or intuition has been provided as to why this is a good choice, and under what circumstances, it will break down. Without this explanation, it is very difficult to reuse the same model for a dataset captured with a different lenseless system. More work/reasoning is needed to support this choice of sampling.\n\nQUESTIONS:\n1) This is minor, but why is Wiener Doconvolution referred to as WinnerDeconv everywhere in the paper?\n2) Although the paper compares the proposed approach with several other methods for qualitative evaluation, it is unclear how these approaches compare when it comes to computational complexity. Typically, there is a trade-off between quality and compute. Therefore I would like to see a comparison of the computational complexity involved in reconstruction.\n3) I would recommend citing the following paper:\nV. Boominathan, J. T. Robinson, L. Waller, and A. Veeraraghavan, “Recent advances in lensless imaging,” Optica 9(1), 1 (2022). This paper gives an overview of the common modulation schemes used for lenseless imaging, and I believe your approach is only applicable for phase modulation masks and not for amplitude modulation masks?\n\nLIMITATIONS:\n1) Although the authors mention that real-time reconstruction is not possible, it is unclear how much time reconstruction actually takes. It would be great if this is explained as well.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-25"}