Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes using a shallow network with a large convolutional kernel for end-to-end (E2E) learning of computational imaging systems. The proposed large convolutional kernel (used only in the first layer) is implemented in the Fourier domain. The proposed network is used to reconstruct experimentally captured diffuserCam data, where it outperforms SoTA methods. It is also used to E2E design a snapshot 3D microscope. This new design significantly outperforms a U-net based system (in simulation), though it is not compared with lenslet based systems. The authors wrote and plan to release parallelized software which allows one to simulate wave propagation using 8 GPUs.
strengths_and_weaknesses: # Strengths

Using a large receptive field in when dealing with optical systems with large PSFs is well motivated and the proposed design seems to work better than Unets. 

Proposed method produces SOTA results on diffusercam data

Paper is well-written and easy to read

Proposed methodology could lead to interesting new microscope designs

The multi-GPU simulator could be useful for a number of research groups

# Weaknesses

The paper doesn't adequately discuss or compare against methods (particularly in the 3D experiments) methods that first approximately inverse the optical system before applying a NNs [3,A]. Learned Wiener filters [A] seem quite similar to the proposed technique.

No experimental 3D microscopy results (with the proposed optics).

No comparisons with classical or lenslet based approaches to snapshot 3D imaging.



[A]Yanny, Kyrollos, et al. "Deep learning for fast spatially varying deconvolution." Optica 9.1 (2022): 96-99.


questions: How do E2E designed masks compare to lenselet based masks?

Could the authors speak to why an approximate inverse (i.e., A^dagger y or Wiener deconv) before a U-net doesn't adequately model non-local effects.
limitations: Addressed
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes an end-to-end (E2E) optimized computational imaging method to reconstruct 3D volumes (or 2D images). The method uses simulated optical encoders which encode the input into a 2D encoded phase image which is then decoded by a Fourier transform-based neural network. The method is then evaluated on 3D snapshot microscopy data where the input is a 3D volume, and DiffuserCam-based lensless photography, where the input is the intermediate 2D phase image itself. The method outperforms existing methods and will be openly be made (which the reviewer acknowledges very positively).

The author’s claims are:
1.	A multi-GPU fast implementation of the simulated optical encoder where the parameterized phase mask can be optimized E2E
2.	Fourier Transform-based decoder converts the intermediate input into Fourier space and does convolution (actually element-wise multiplications) that achieves faster computation of the final output. This helps the network to reduce the locality bias and learn from a wider global context.
3.	They collected extra data to show the merit of their method in a region-of-interest (ROI) specific 3D volume imaging case.
4.	SOTA results on the following tasks – 3D snapshot microscopy and DiffuserCam-based lensless photography
5.	This is the first time that a direct E2E optimization of the spatial light modulator (SLM) based optical encoders have been proposed. They back this claim using the simulation result and a single case of implementation on a programmable microscope.

strengths_and_weaknesses: Strengths:
1.	The paper addresses an important area in optics and clearly defines the existing limitations of exploiting the latest advances in end-to-end machine learning methods and optimizing the optical encoding system. 
2.	The arguments, concepts, assumptions, and methods are lucidly explained by the authors in the paper.
3.	The paper builds on the advances in multi-GPU-based parallelism and efficient deployment of neural networks for a problem that is otherwise not feasible due to limitations in computing. 
4.	The author’s assumption that the locality bias by having a small filter size in existing methods cannot effectively capture then information from the image formation process, which is non-local, sounds plausible. To address this problem, the authors propose a method using Fourier transform and using larger filter sizes.
5.	Relevant papers have been cited and tricky parts have been described verbose in the appendix.
6.	The overview figure clearly depicts the proposed method.
7.	The use of Fourier domain convolutions reduces the computation time and increases inference speed.
8.	The loss function has been designed to give more importance to the high frequencies of the input which are more important than low frequencies which are relatively less important. 
9.	The results on DiffuserCam dataset corroborate the working of the decoder in capturing the high frequencies better than existing methods.
10.	The authors show that their method can indeed be implemented on a programmable microscope as a prototype.
11.	The results on the 3D microscopy dataset show that:
a.	The E2E optimization outperforms non-E2E methods in reconstruction quality with fewer parameters.
b.	The proposed method can be optimized for ROI-specific optimizations, that achieve better results than existing methods.

Weaknesses:
1.	The application of this method seems to be in a very limited field which is a differentiable simulation of optical encoders.
2.	The authors could have shown the result on 1~2 more datasets.
3.	UNets have been there for a while. Are they indeed the best baseline method to compare the presented method against?
4.	There is no theory or citations to back the claim made in line 136.
5.	An entire multi-GPU setup is required for the optimizations in the proposed method, which makes it not very accessible for many potential users.

questions: * A brief explanation of the non-local optical encoder would have been helpful.
* Are there other baselines you could compare your approach agains?
limitations: I believe so.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper has two main contributions: (1) development of a parallelized differentiable wave optical encoder simulator and (2) a Fourier-space neural network which enables non-local optical decoding and outperforms previous deep learning-based image reconstruction strategies. Taken together, these two contributions enable end-to-end optimization of non-local optical encoding/decoding systems. The paper demonstrates both of these techniques on a new dataset of 3D snapshot zebrafish larvae microscopy images and demonstrates the reconstruction strategy on a previously published dataset for DiffuserCam lensless photography. The paper also demonstrates that the learned optical encoders can be implemented in practice on a prototype programmable microscope.
strengths_and_weaknesses: [Note for AC: I don’t have direct experience with 3D snapshot microscopy or differentiable optical simulation — while I read through those details and they seem solid to me, the other reviewers may be more qualified to comment on that component of this work. My experience is more closely related to Fourier-space neural network architectures part of this work.]

# Strengths
- This paper is extremely clearly written — it is quite easy to follow, even as someone relatively unfamiliar with 3D snapshot microscopy.
- The experimental results are compelling and show clear improvements from the FourierNet architecture, both qualitatively and quantitatively.
- The appendix contains extensive, easy-to-follow characterization of the computation time and memory requirements of all methods studied, as well as detailed pseudocode and implementation details for the various algorithms described in the paper.

# Weaknesses
 I cannot find any major technical weaknesses with this paper and have only minor comments/questions/clarification requests here.
- There are currently two differences between FourierNets and standard ConvNets: The global-width convolution filters, implemented in Fourier space, and the different pooling strategy which crops in Fourier space, instead of, say, max or average pooling. If time/resources are available, a small ablation which compares FourierNets to FourierNets with image-space max/average pooling would further strengthen the argument that it is the global filter extent which is leading to performance gains. However, I don’t think this is strictly necessary for acceptance, because both of the implemented differences rely on the Fourier implementation of the networks used in this paper.
- Is the U-Net in all experiments designed to have a receptive field that covers the entire image? If not, this comparison should be made to show that, even if the U-net has such a wide receptive field, the local operations at each layer are ill-suited to tasks with a wide point spread function. If so, it may be worth it to clarify this explicitly in the text.
- Why are results reported with MSE and MSE+LPIPS loss for FourierNets in Table 1, but only for MSE+LPIPS for the other methods? Do the other methods do worse with MSE loss?
- Lines 219-223 describe a two-step training process — was this used for generating the metrics in Table 2 just when training FourierNets as microscopes? Or also when using the UNets?
- Is the loss function in Equation (4) used to train all methods in Table 2, including the U-Nets? I assume so, since no other loss functions are mentioned, but if not, ablations are needed to show that the architecture, and not the loss function, are contributing to the improvement observed here.
- There is limited technical novelty _from an ML perspective_ in this work (neural networks operating via elementwise operations in the Fourier space have previously been developed, as cited in the manuscript). That said, I do think that the use of FourierNets for end-to-end optimization of the types of imaging systems in this paper is novel and a solid computational imaging advance.
- Very minor corrections: Figure 2 caption’s third sentence should read “Bottom row.” And I believe Line 235 should reference Table 2.

questions: The “weaknesses” section is structured as a list of questions I would like answered/clarified. To summarize:
- Do FourierNets with U-Net style pooling still outperform U-Nets?
- Is the U-Net in all experiments designed to have a receptive field that covers the entire image?
- Why are results reported with MSE and MSE+LPIPS loss for FourierNets in Table 1, but only for MSE+LPIPS for the other methods?
- Is the 2-step training procedure used for the U-Nets as well as the FourierNets?
- Is the loss function in Equation (4) used to train all methods in Table 2, including the U-Nets?
limitations: Yes, the authors have clearly addressed the limitations and the societal impact in Section 4. To summarize, the main limitation identified by the authors is that all of the results in this paper are presented on simulated data.
review_title: null

Meta-review: metareview: This paper develops a multi-GPU differentiable simulation of a programmable microscope, wherein a large convolutional kernel used in the first layer is implemented in the Fourier domain.  The work outperforms sota approaches for lensless photography, and it allows the end-to-end design of a snapshot 3D microscope which beats state of the art systems in simulation.  Overall the work provides a compelling demonstration of the power of recent technical advances in ML to lead to demonstrable improvements (in simulation) of computational imaging methods.  
confidence: Certain
Decision: Accept