FourierNets enable the design of highly non-local optical encoders for computational imaging
Authors: Diptodip Deb
Zhenfei Jiao
Ruth R Sims
Alex Bo-Yuan Chen
Michael Broxton
Misha Ahrens
Kaspar Podgorski
Srinivas C Turaga
Venue: NeurIPS 2022 Accept (2022)
Keywords: computational microscopy
computational photography
computer vision
deep learning

Abstract: Differentiable simulations of optical systems can be combined with deep learning-based reconstruction networks to enable high performance computational imaging via end-to-end (E2E) optimization of both the optical encoder and the deep decoder. This has enabled imaging applications such as 3D localization microscopy, depth estimation, and lensless photography via the optimization of local optical encoders. More challenging computational imaging applications, such as 3D snapshot microscopy which compresses 3D volumes into single 2D images, require a highly non-local optical encoder. We show that existing deep network decoders have a locality bias which prevents the optimization of such highly non-local optical encoders. We address this with a decoder based on a shallow neural network architecture using global kernel Fourier convolutional neural networks (FourierNets). We show that FourierNets surpass existing deep network based decoders at reconstructing photographs captured by the highly non-local DiffuserCam optical encoder. Further, we show that FourierNets enable E2E optimization of highly non-local optical encoders for 3D snapshot microscopy. By combining FourierNets with a large-scale multi-GPU differentiable optical simulation, we are able to optimize non-local optical encoders 170$\times$ to 7372$\times$ larger than prior state of the art, and demonstrate the potential for ROI-type specific optical encoding with a programmable microscope.

FourierNets enable the design of highly non-local optical encoders for computational imaging
Diptodip Deb'
Zhenfei Jiao
Ruth R. Sims!
Alex Chen 4d4
Michael Broxton
Misha B. Ahrens
Kaspar Podgorski
Srinivas C. Turaga
HHMI Janelia Research Campus, 2 Wuhha National Laboratory for Optoelectronics, HUST Institut de la Vision, Sorbonne Université, CNRS, Harvard University, Stanford University, {debd, s simsr, ahrensm, podgorskik, turagas} anelia hhmi.org iagghust edu. abchen@g harvard. b broxton@al lumni stanford edu equal contribution
Abstract
Differentiable simulations of optical systems can be combined with deep learning- based reconstruction networks to enable high performance computational imaging via end-to-end (E2E) optimization of both the optical encoder and the deep decoder. This has enabled imaging applications such as 3D localization microscopy, depth estimation, and lensless photography via the optimization of local optical encoders. More challenging computational imaging applications, such as snapshot mi- croscopy which compresses 3D volumes into single 2D images, require a highly non-local optical encoder. locality bias which prevents the optimization of such highly non-local optical encoders. We address this with decoder based on a shallow neural network architecture using global kernel Fourier convolutional neural networks (Fourier- Nets). We show that FourierNets surpass existing deep network based decoders at reconstructing photographs captured by the highly non-local DiffuserCam optical encoder. Further, we show that FourierNets enable E2E optimization of highly non-local optical encoders for 3D snapshot microscopy. By combining Fourier- Nets with a large-scale multi-GPU differentiable optical simulation, we are able optimize non-local optical encoders 170x to 7372x larger than prior state of the art, and demonstrate the potential for ROI-type specific optical encoding with a programmable microscope.
show that existing deep network decoders have
Introduction
Modern computational optics relies on the end-to-end (E2E) optimization of the optical system (optical encoder) and the computational image reconstruction algorithm (computational decoder). This has been enabled by the development of differentiable physics-based simulations of optical encoders, paired with deep network based decoders. This E2E approach has been successfully applied to the estimation depth from defocus [1, 2, 3, 4], lensless photography [5], and particle localization microscopy [6, 7]. Despite this wide success, only the space of local optical encoders has been explored in this fashion due to the computational expense of physics-basee simulation of highly non-local optical encoders [3]. This is particularly the case for 3D snapshot microscopy which is well-known to require highly non-local optical encoders, and for which no E2E solutions have yet been demonstrated.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

We show that there are actually two issues blocking the E2E optimization of highly non-local optical encoders, First, the computational expense of physics- based simulation of non-local optical encoders is significantly greater, since they are orders of magnitude larger than local encoders. And second, the locality bias inherent to deep network architectures currently used for computational imaging prevents decoding from non-local solutions. In this paper, we primarily focus developing computational framework for engincering highly non-local optical encoders by solving both issues. We solve the first issue by developing large-scale multi-GPU differentiable optical simulation, and the second by developing FourierNet: a new decoder network architecture based on Fourier convolutions.
In 3D snapshot microscopy, all existing methods [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 5, 19] employ highly non-local optical encoder to optically transform 3D volume into a single 2D camera image, which is computationally decoded to reconstruct the 3D volume. Fast volumetric imaging invaluable across biology, including for imaging neural activity throughout the whole brain of an animal [20, 21, 22]. Here, we focus on developing a 3DD snapshot microscope for imaging neuronal activity across the whole brain of the larval zebrafish (Danio rerio) at camera rates exceeding 100 Hz. This is two orders of magnitude faster than the fastest conventional microscopes
light sheet imaging of whole brain neural activity can only achieve 0.5 Hz - 2 Hz volume rates [20]. This improvement of temporal resolution is essential for imaging with fast calcium indicators [23] and voltage indicators [24].
Newly developed programmable microscopes with up to 10 free parameters, e.g. pixels on spatial light modulator (SLM), enable the implementation of a rich space of optical encodings and present the possibility of direct optimization of snapshot microscope parameters specifically for particular ROI types and imaging tasks. While our paper primarily focuses on addressing the computational challenges of E2E optimization of highly non-local optical encoders, we also demonstrate a proof-of- concept that an SLM-based programmable microscope can implement such an engineered non-local optical encoder, paving the way to implementing multiple ROI-type and task specific optical encoders in a single physical microscope. In contrast, 3D snapshot imaging has classically been performed using fixed optical encoders based on microlens arrays [8. 9, 10. 11, 12, 13, 14, 15), pseudorandom diffusers/masks [16, 17], and designed or optimized diffusers/masks [18, 5, 19], with some of these methods using deep learning for reconstruction [11, 15].
Problem statement We define an optical encoder as Ma parameterized by dp and computational decoder as Ro parameterized by O. Optical encoders can usually be modeled as linear transfer functions implemented by the optics. In special cases, such as for the optical systems explored in this paper, they can be represented as a linear convolution filter called "point spread function" with coefficients computed by a wave optics simulation, dependent on the physical parameters & of the optical system (Appendix A. We wish to develop reconstruction network architecture for decoding non-local encodings in 2D images c produced by Mo and also to enable the end-to-end optimization of db to produce non-local encodings. We'll consider two applications to investigate these optimization problems: (1) lensless photography (Figure 2). where we optimize only the reconstruction network Ro to reconstruct images of natural scenes from camera images captured by the DiffuserCam [25], and (2) 3D snapshot microscopy using an SLM-bassed programmable mieroscope (Figure IA), where we perform E2E optimization of both the reconstruction network R. and the optical encoding M. based on the SLM parameters in order to image 3D volumes v and reconstruct 3D volumes V. While we focus on solving the computational challenges involved in this E2E optimization of large highly non-local optical encoder, we also demonstrate proof-of-concey implementation of our optimized optical encoder using an SLM-based programmable microscope.
1.1 Our contributions
1. We developed a large-scale, parallel, multi-GPU differentiable wave optics simulation of a programmable microscope, based 4f optical model with a phase mask (+)) implemented using a spatial light modulator (SLM), described further in Appendix A. SLMs (dp) can have over 10 optimizable parameters, which we can feasibly simulate and optimize to produce PSFs with 170x to 7372x more unique voxels than previous attempts at deep learning PSF optimization using singlc GPUs [6. 3] (Appendix A.5).
2. We collected a large dataset of high resolution 3D confocal volumes of zebrafish larvae for the purpose of ROl-type specific end-to-end optimization of optical encoders.

---

3. We introduce an efficient FourierNet reconstruction network architecture for decoding from non-local optical encoders using very large global convolutions implemented via Fourier convolutions.
4. We show that our networks outperform the state-of-the-arr deep decoders for DiffuserCam based lensless photography [25] and for 3D snapshot microscopy.
5. Our method enables, for the first time, direct end-to-end optimization of highly non-local optical encoders in the space of spatial light modulator (SLM) pixels with over 10 parame- ters. In simulation, we demonstrate the potential for significant improvements in imaging resulting from ROI-type specific optimization of optical encoders.
1.2 Prior work
Neural network architectures for computational imaging have all used convolution layers with small filters. End-to-end optimization of optical encoders have largely been performed with UNet- based architectures [1, 2. 3. 4, 7], with one method using ResNettbbased architecture [6]. Such optimization has always led to local optical encodings. End-to-end optimization has never been attempted for large-field of view 3D snapshot microscopy due to the difficulty of simulating and reconstructing from non-local encoders. However, small filter convolutional deep networks have been used non-end-to-end manner to reconstruct volumes from 3D snapshot microscopes designed using microlens arrays [15, 26, 27]. One recent hybrid approach combines UNet with a differentiable approximate inverse method (Wiener filter) to handle non-local spatially varying (non-convolutional) optical encoders [27]. For photography and MRI, another hybrid approach of deep learning combined with unrolling iterations of traditional deconvolution algorithms provides the benefits of fast amortized optimization and higher quality reconstructions due to learning of structural priors [28, 29, 25, 26]. We note that these hybrid approaches typically require measurement of the optical encoder of the system, whereas our method does not and can learn to produce high quality reconstructions using only pairs of ground truth and system images.
In our work, we demonstrate that convolution layers with large filters implemented efficiently in the Fourier domain enable the end-to-end learning of highly non-local optical encoders for 3D) snapshot microscopy. Large convolution filters have been shown to be helpful for other computer vision applications such as semantic segmentation and salient object detection [30, 31], and the Fourier domain parameterization of small filters has been described previously [32].
pioneering strategy in 3D snapshot microscopy is light field microscopy [8], which employs a microlens array at the microscope's 's image plane to create subimages encoding both the amplitude and phase of light [8, 33]. A variety of microlens microlens-array-ao a-raaaraaaaaaaa based light field microscopes have been used to perform whole brain imaging [8, 11, 14, 22, 13, 12. 9, 34, 19]. [19] optimizes the placement of microlenses, but not in an end-to-end manner. Despite variation in design, microlens- based microscopes have, to various degrees, four main limitations that can be improved: 1) blocking or scattering light between microlenses, causing light inefficiency, 2) not making use of all pixels on the camera to encode a 3D volume, leading to inefficient compression and suboptimal reconstructions v, 3) aliasing at some planes with generally nonuniform axial resolution, and 4) a fixed optical encoding scheme.
An alternative to using microlenses is to implement coded detection strategy using a phase mask or diffuser to spread light broadly across the camera sensor [16, 18, 35, 5, 10, 17, 19]. The designed phase masks can be implemented either by manufacturing a custom optical element or using a programmable SI.M [2. 3, 4, 5, 19, 6]. Using a programmable element allows different microscope parameters to be used for different sample and ROI types.
Methods
We show our network architecture and an overview of autoeneoder training both the microscope parameters dp and reconstruction network parameters 0 in Figure IA. The programmable microscope simulated by a differentiable implementation of wave-optics model of light propagation. We have selected a programmable microscope design based on pupil-plan phase modulation with programmable spatial light modulator, for which imaging is well-approximated by a computationally- efficient convolution [36]. A detailed description of our simulation is provided in Appendix A. For lensless photography, there is no optical simulation because the images have been collected on a real camera.
3

---

A 3D snapshot microscopy
programmable microscope (optical encoder)
reconstruction network (computational decoder)
image
sample
M +p
R
3D reconstruction
V
phase mask dp Fourier convolution
B FourierNet R
F
F
W
conVs
conv
Figure I: Overview of our problem setup and our proposed network architectures. Top row (A) shows the problem of 3D snapshot microscopy, where we computationally reconstruct 3D volume from 2D image. Bottom row (B) shows our proposed FourierNet architecture, which includes a Fourier convolution layer that enables efficient computation of global features.
2.1 FourierNet for decoding non-local optical encoders
Because images created by optical encoders can potentially encode signals from the incoming light field to any location in the camera image in non-local manner, it is essential that reconstruction network have global context. Existing multi-scale architectures such as the UNet [37] can achieve global context, but at the expense of many computation layers with small convolution kernels which we speculate to have a local information bias that is inappropriate for computational optics. In this paper, we introduce relatively shallow architectures for computational imaging, which rely on convolution layers with very large filters, implemented efficiently in the Fourier domain [32].
FourierNet We propose simple three layer convolutional network architecture with very large global convolutions at the very first layer, followed by two standard local convolutional layers (Figure convolution with kernel size equal the input image. Such
IB). We define global convolution a convolution achieves global context in a single step but is computationally expensive. Global convolutions are implemented more efficiently in the Fourier domain, yielding a speed up of two orders of magnitude over direct convolution. Due to the use of Fourier convolutions to enable global context, we call our architecture FourierNet. In contrast to typical UNet which can contain many tens of convolution layers, the FourierNet is only three layers deep, which requires backpropagation through fewer layers compared
typical UNet with the same receptive field.
Fourier domain convolutions It is well-known that large kernel convolutions can be implemented more efficiently in the Fourier domain [38, 39, 40]. A naive implementation of global convolution requires ( operations, where N is the number of pixels in both the image and the kernel. An alternative global convolution implementation is to Fourier transform (F) the input and convolution kernel w. perform element-wii -oine multiplication in Fourier space, and finally inverse Fourier transform (F-1), requiring only ONN log N) operations [39, 40J. Following [32], we store and optimize the weights W in Fourier space. This over paramttrrizaiionnn costs x the memory of an equivalent real valued large filter but saves the computational cost of Fourier transforming the real-valued weights (Appendix A.6). Thus Fourier convolution is defined:
Re(F (W O F [(}})
(I)
For image and kernel sizes of 256 256 pixels, our implementation leads to nearly 500x speedup: standard PyTorch convolution takes 2860 ms, while Fourier convolution takes 5.92 ms on a TITAN X. We also show how we naturally extend our Fourier convolutions and FourierNet multiscale version using cropping in the Fourier domain in Appendix A.2.
2.2 Physies-based autoencoder for simultaneous engineering of microscope encoder and reconstruction network decoder
We describe the imaging process the following transformation from the 3D intensity volume v to the 2D image formed on the camera C:
He = Ma (v)
c = max (14, + -,,
(3)

---

where M& denotes the microscope parameterized by a 2D phase mask. cb This phase mask db describes the 3D-to-2D encoding of this microscope model completely. A Poisson distribution with mean rate He describes the physics of photon detection at the camera, but sampling from this distribution not differentiable. We approximate the noise distribution with a rectified Gaussian. We include details on M in Appendix A.1 [36]. Jointly training reconstruction networks and microscope parameters involves image simulation, reconstruction, then gradient backpropagation to update the reconstruction network and microscope parameters, Our parallelization strategy enables optimization of phase masks with millions of parameters in a feasible amount of time and memory per GPU. This also enables voxels than previous attempts (Appendix A.4). Details on parallelization, planewise reconstruction networks, and planewise sparse gradients are provided in Appendix A.4, Figure 12.
produce PSFs with multiple orders of magnitude more unique
We use the normalized mean squared error (NMSE) as the basis for our loss function. Since we care more about the high spatial frequency content of the image which contains mostly cells, and less about the low spatial frequencies which contain mostly background, we implemented two part loss function. LHNMSE measures the NMSE between the high pass filtered volume and high pass filtered reconstruction, This is combined with the unfiltered NMSE LNMSE between the original volume and its reconstruction. Formally, our loss function L(v, V) for all snapshot microscopy reconstruction problems is defined as:
L(v, V) LHNMSE VV, 8) + BLNMS (v, 6) LHIMSE(V. v) E (VV(v) (9))²
(4)
E (v +
_LNMSEVV
(5)
B(H(v)?
E(v²)
where H-) denotes high pass filtering and E(-) denotes the mean over pixels and sample volumes. Both loss terms normalized as shown to reduce variance in L, which can otherwise cause large
magnitude fluctuations based on the brightness variation across training volumes resulting in training instability. For our experiments, set the weight B for the LNMSE term to 0.1.
3 Results
DiffuserCam Lensless Mirflickr Dataset We test reconstruction performance on experimental computational photography data' from [25] (Figure 2). This is dataset constructed by displaying RGB color natural images from the MIRFlickr dataset on a monitor and then capturing diffused images by the DiffuserCam lensless camera. The dataset contains 24,000 pairs of DiffuserCam and ground truth images. The goal of the dataset is to learn to reconstruct the ground truth images from the diffused images. As in [25], we train on 23,000 paired diffused and ground truth images, and test on 999 held-out pairs of images.
Larval Zebrafish Snapshot Microscopy Dataset We show all our results for snapshot microscopy using our simulation of snapshot microscope, and our engineered optical encoders have not been experimentally tested for reconstruction performance on programmable microscope. We simulate snapshot imaging using high resolution confocal imaging volumes of zebrafish. These are volumes of transgenic larval zebrafish whole brains expressing the nuclear-restricted GCaMP6 calcium indicator in all neurons. These images are representative of brain-wide activity imaging. We train on 58 different zebrafish volumes (which we augment heavily) and test on 10 held-out volumes. For all experiments, we downsample the high resolution confocal data to (1.0 jum z, 1.625 um y. 1.625 μmm x). We created 4 datasets from these scanned volumes corresponding to imaging different fields of view or regions of interest (ROIs) for our experiments. Full specifications for these datasets are in Table 6 (Appendix A.4). For Figure 3, we restrict the field of view to (200 um 416 μm y. 416 um x) with a tall cylinder cutout of diameter 193 jum and height 200 μm and image with 256 x 256 pixels on the simulated camera sensor. Figure 4 and Table 3 show our larger experiments with 512 x 512 simulated camera pixels, with a field of view of (250 um z, 832 μm y, 832 μm x).
3.1 FourierNets outperform state-of-the-an for reconstructing natural images captured by DiffuserCam lensless camera
We compare our FourierNet architecture to the best learned method from [25] using unrolled ADMM and denoising UNet, as well to a vanilla UNet from [25]. Architecture details are in Appendix
Publicly available: https: haals::/wllleerre lla..gittbbb htt.th/.ooo/eetsssssseeeennneeee
5

---

DifuseCCan imagee
UNet Le-ADMIAU Fourlertes (ours)
IISE LRPS) INSS LPIPS)
IMSE LAIPS
Figure 2: Comparisons of method (third row) to state-of-the-art learned reconstruction methods on lensless diffused images of natural scenes. Regions labeled indicate missing details, either resolution textures in backgrounds. Regions labeled 2 indicate hallucinated textures. Note that the previous state-of-the-art solution (fourth row) [25] exhibits both issues more often compared to our method.
Table 1: Quality of natural image reconstruction the DiffuserCam Lensless Mirflickr Dataset (mean + s.c.m,, = 999). Superscripts denote loss function: MSE, MSE+LPIPS.
Method
MSE J 3x LPIPS
MS-SSIM
PSNR
Time J (ms)
FourierNet!
0.39 + 0.007
0.20 + 0.00
0.882 + 0.001
24.8 + 0.09
35.54
F
0.54 + 0.010
0.16 + 0.00
0.868 + 0.001
23.4 + 0.09
35.54
Le-ADMM-U2 [25] 0.75 + 0.021
0.19 + 0.00
0.865 + 0.002
22.1 + 0.09
48.59
UNet [25] 1.68 + 0.060
0.24 + 0.00
0.818 + 0.002
19.2 + 0.11
06.97
A.6, A.7. We can see that FourierNet visually outperforms the methods from [25] in Figure 2, and quantitatively Table across all quality metrics. The Le-ADMM-U and UNet results in Table I were reported by [25] using combined MSE LPIPS loss. Unlike [25]. find that training FourierNets with the MSE loss alone provides reconstructions visually similar to the ground truth as shown in Figure 13 (Appendix A.7). Timings in Table are for only the forward pass a single TITAN Xp GPU.
3.2 FourierNets outperform UNets for engineering non-local optical encoders
Table 2: Quality of reconstructed volumes after optimizing microscope parameters to image zebratish on 256 256 pixel camera (mean + s.e.m,, 10)
Microscope
Reconstruction
Lhnmse +
MS-SSIM t
PSNR t
Time (s)
FoDD FoDD
FourierNet3D
0.6093 0.0209
0.955
0.004
34.89
0.88
0.38
UNet3D
0.7298 = 0.0151
0.923 = 0.008
30.16 = 0.94
0.96
Wiener UNet
Wiener + UNet
0.7223 0.0179
0.957 = 0.003
34.49 0.91
0.73
UNer2D
UNet3D
0.7109 + 0.0161
0.913 + 0.009
29.17 + 1.13
0.96
We compare optimizing microscope parameters d with three neural networks: 1) using our FourierNet with 2D convolutions (FourierNet2D),, 2) using a vanilla UNet with 2D convolutions (UNet2D), and 3) using a Wiener deconvolution approximate inverse combined with refining UNet, as

---

FourierNet
UNet
Wiener UNee
x100
Ex10+
xx10
sinulated comeral mmagg
light propaiaatoon
ground truth
1.00
0.01
0.01
0.01
1040
30
637
432
30
645
3,00 5
reconsruccedd volume
max projection
heu Pooeeroo
Figure 3: Comparing simulated camera images (0.01 second expected acquisition time) and corre- sponding reconstructions of volume captured using FourierNet (left) versus UNet (middle) and Wiener + UNet [3] (right) optimized microscopes. Top row shows simulated 256 x 256 pixel camera images; bottom right of camera image shows approximate acquisition time (given reasonable number of simulated photons per pixel, i.e. SNR). Ground truth has no corresponding camera image, because the 3D volume is imaged directly via slow high resolution confocal microscopy (3 second acquisition time). Colored arrows in left column show projection axis for each row. White arrows show individual neurons clearly visible for FourierNet, but not for UNet. Wiener UNet [3] has detail in some planes, but not consistently throughout the whole volume, and also uses fewer camera pixels.
in [3]. Training is a two stage process in which a plane-wise reconstruction network with fewer parameters and 2D convolutions used during optimization of Φ. then once dp fixed a more powerful reconstruction network with 3D convolutions is used, except for the Wiener + UNet [3] method which trained in a single stage as [3]. Table shows the type network used for the first stage to train the optical encoder Ms in the first column, and the type of network used in the second stage for training the reconstruction network only (given fixed dp) in the second column. We find this scheme achieves better reconstruction quality because the reconstruction network does not need constantly adapt to a changing optical encoding (Appendix A.6). We train these microscopes and reconstruction networks on ROIs which are tall cylindrical cutouts of zebrafish with diameter 193 um and height 200 μmm Sample volumes are imaged on a camera with 256 x 256 pixels (Figure 3). FourierNet2D has 2 convolution layers (with 99.8% of its kernel parameters in the initial Fourier convolution layer), while UNet2D has 32 convolution layers (kernel parameters approximately uniformly distributed per layer). UNet2D was designed to have global receptive field. FourierNet2D and UNet2D both have ~ x 10" parameters; FourierNet3D has ~ 6 10" parameters 10 for UNet3D. The Wiener + UNet method [3] has 8 10" parameters. Architecture details are in Appendix A.6, A.8.
Simulated camera images (Figure 3) show that the UNet microscope does not make sufficient use of camera pixels, producing only single view of the volume. We speculate this due local information prior in the small kernels of UNets. Adding Wiener deconvolution an approximate
7

---

Table 3: ROI specific microscope parameter optimization for types of zebrafish volumes (mean PSNR (top), MS-SSIM (bottom) s.e.m., n = 10). Green shows regions of interest.
Microscope parameters optimized for
Type A
Type B
Type C
Tested on
Type A
49.76
1.35
46.03 1.33
42.67 + 1.14
0.998 + 0.000
0.996 + 0.001
0.992 + 0.002
Type
35,56 1.41
37.34 + 0.96
35.38 + 1,16
0.965 = 0.004
0.972 + 0.003
0.967 + 0.003
Type
30.87 1.15
31.48 0.93
33.79 + 0.90
0.912 + 0.007
0.920 + 0.006
0.935 + 0.006
inverse before UNet also does not result in microscope that makes full use of camera pixels, but is better than the UNet alone. The FourierNet microscope uses more camera pixels and performs better than the UNet microscope for reconstruction (Figure 3, quantified in Table 2). Timings in Table are for only the forward pass on a single TTTAN Xp GPU; one training iteration on 8 GPUs takes ~0.4 seconds for FourierNet3D, ~0.8 seconds for Wiener + UNet, and ~0.8 seconds for UNet31) (Appendix A.6). Both reconstruction networks must reconstruct from images that have a compressed encoding of 3D information, but the FourierNet2D is clearly more effective than the UNet2D at optimizing this encoding.
3.3 FourierNets outperform UNets for 3D snapshot microscopy volume reconstruction
We can determine which architecture is better for volume reconstruction by choosing fixed microscope parameters and varying the architecture, except for the Wiener + UNet method [3] which is trained in one stage. In Table 2, we compare results using FourierNet with 3D convolutions (FourierNet3D) and a vanilla UNet with 3D convolutions (UNet3D). UNet3D was also designed to have global receptive field. Architecture details are Appendix A.6, A.8.
Reconstruction results in Table 2 compare normalized MSE LHNMSE between the high pass fil- tered volume and high pass filtered reconstruction, the multiscale structural similarity MS - SSIM between the true volume and its reconstruction, and finally the peak signal-to-noise ratio PSNR. We also visualize reconstruction results for volume in the head of zebrafish in Figure The UNet3D reconstruction networks (using either microscope) fall significantly behind the FourienNet3D reconstruction network in all metrics, despite their global receptive field. The Wiener UNet [3] network achieves similar MS - SSTM and PSNR as the FourierNet but worse LHNMSE due to the inconsistent detail, though the reconstructions are slightly better than the UNet3D for certain regions of the volume.
3.4 Engineered optical encoders can be optimized for a region of interest
To explore the effect of ROI size on optimized op and the resulting reconstruction performance, we optimized cps for three different regions of interest: 1) Type A, ROIs with a short cylinder cutout of 386 um diameter and 25 μm height, 2) Type B, ROIs with tall cylinder cutout of 386 μm diameter and 250 um height, and 3) Type C, ROIs without any cutout of dimension (250 μm Z x 832 μmm y x 832 μm x) (Table 3). All ROI types were imaged with 512 x 512 pixels the simulated camera. We then tested reconstruction performance on all combinations of optimized dp and ROI type, as shown in Table 3 and visualized in Figure for Type B. We include architecture details in Appendix A.6, A.9.
We see in Table 3 that for all types, highest performance is achieved using the phase mask optimized for that particular type. These results show that there potentially large beneñit to optimizing type-specific optical encoders, which can be easily implemented in a programmable microscope.
3.5 Engineered optical encoders can be implemented on programmable microscope
We demonstrate that the optical encoders engineered by simulations can implemented on real hardware using prototype programmable microscope. Our engineered phase masks were displayed
8

---

ground truth (Type B)
optimized for Type
optimized for Type
optimized for Type C
1026
518
582
30
516
Jum
simulated camera image
4x10 2x10
xx10+
100 um
Figure 4: Reconstructed volumes resulting from imaging Type B ROIs by microscopes optimized for Types A, B, and C. Imaging Type B ROIs with microscope parameters optimized for Type ROIS yields the best reconstructions. Top to bottom: xy max projection, xz max projection, simulated 512 x 512 camera image.
on a spatial light modulator (SLM) located in the pupil plane. We observe good qualitative match between the features of the simulated and experimental optical encoders (point spread functions) in Figure 5, technical details in Appendix A.10. There is also a good qualitative match between the simulated camera images generated by both optical encoders. These results demonstrate the potential for programmable microscopy with spatial light modulators.
We also note that our optimized optical encoders result in pencil-like elements, qualitatively similar censlet-based approaches [19, 10, 9]. However as we show our supplement, our optimized phase masks (Figures 11) appear qualitatively different from lenslet-based phase masks, with different regions the pupil contributing to the same pencil. This suggests qualitatively different, and perhaps more light efficient, mechanism for generating high resolution projections of the volume along different axes.
Simulated optical encoder
Measured optical encoder
5x10
1x10
100 um
Figure 5: Simulated versus measured optical encoders (point spread functions) with corresponding
simulated camera images using example from our Type B zebrafish dataset. Zoom in for best viewing.

---

4 Discussion
Summary We have presented the FourierNet architecture as deep network based decoder for computational imaging with highly non-local optical encoders. We demonstrated the superiority of FourierNets for lensless photography on the DiffuserCam dataset, and also for the end-to-end optimization of highly non-local optical encoders (where UNets fail) for 3D snapshot microscopy. FourierNets are many orders of magnitude faster than traditional iterative reconstruction algorithms [25] and generate higher quality reconstructions by learning image priors. Generally, our global Fourier-domain convolution architecture could be applicable to other problems where global integra- tion of features is necessary, though we have focused on computational imaging applications where physics-based optical encoders induce global mixing of information. Our work solves two important computational problems preventing E2E optimization of large highly non-local optical encoders: the computational complexity of simulating large non-local encoders, and the effective decoding from such encoders. Our contributions are primarily computational, however we also demonstrate the potential for implementing 3D snapshot microscopy with an E2E optimized non-local optical encoder using SLM-basek programmable microscope. And in simulation, demonstrate the potential for ROI-specific optimization of optical encoders for 3D snapshot microscopy, which can be efficiently implemented in programmable microscope.
Limitations While we have fully demonstrated that our framework now enables computational E2E optimization of large non-local optical encoders, we do not focus on their hardware implementation. The specific claims regarding the potential benefits of ROI-specific optimization of optical encoders are only evaluated in simulation, and have only been evaluated for whole brain larval zebrafish data.
Reproducibility We train on Quadro RTX 8000 GPUs for the largest experiments, and have described our pre-processing, training, and testing procedures in Appendix A.4, A.6, A.8. We have made our simulation software, training scripts, and our datasets available at https:/// http//ggtthbbbb com/ TuragaLab/ snapshotscope.
Broader Impact End-to-end optimization of optics can improve image quality but also enable multiplexed imaging not currently possible with standard optics. Our Py PyTorch-based optical modeling library well our neural network architectures experiments in neuroscience via whole brain imaging at an order of magnitude greater temporal resolution. Our training procedure does require long optimization periods with many GPUs, which poses a carbon footprint and barrier to usage compared to conventional microscopy.
publicly available and could enable new
Acknowledgement
We would like to thank William Bishop, Roman Vaxenburg, Gert-Jan Both, Janne Lappalainen, Nathan Klapoetke, Richard Xu, Lu Mi, Sridhama Prakhya, and Jinyao Yan for invaluable feedback and discussions. We thank Howard Hughes Medical Institute and a Simons Foundation grant (Simons Collaboration on the Global Brain, 542943SPI, Ahrens) for their funding. We also thank Huazhong University of Science and Technology and the China Scholarship Council for supporting the work of Zhenfei Jiao. We additionally thank the Janelia Visiting Scientist Program for supporting the work of Ruth Sims.
References
[1] J. Chang and G. Wetzstein, "Deep optics for monocular depth estimation and 3d object detection." NKGv:/994.0088
[cs, eess), Apr 2019. arXiv: 1904.08601.
[2] X. Dun, X. Dun, H. Ikoma, G. Wetzstein, Z. Wang, Z. Wang, X. Cheng, X. Cheng, X. Cheng,
Y. Peng, and et al., "I "Learned rotationally symmetric diffractive achromat for full-spectrum computational imaging." Optica, vol. 7, p. 913-922, Aug 2020.
[3] H. Ikoma, C. M. Nguyen, C. A. Metzler, Y. Peng, and G. Wetzstein, "Depth from defocus with learned optics for imaging and occlusion-aware depth estimation," in 2021 I IEEE International Conference on Computational Photography (ICCP), p. 1-12, May 2021.
[4] Y. Wu, V. Boominathan, H. Chen, A. Sankaranarayanan. and A. Veeraraghavan, "Phasecam3d learning phase masks for passive single view depth estimation," 2019 IEEE International
Conference Computational Photography (ICCP), p. 1-12, May 2019.
10

---

[5] Y. Hua, S. Nakamura, M. S. Asif, and A. C. Sankaranarayanan, "Sweepeam depth depth-aware aware lensless imaging using programmable masks," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, p. 1606-1617, Jul 2020.
[6] E. Nehme, D. Freedman, R. Gordon, B. Ferdman, L. E. Weiss, O. Alalouf, T. Naor, R. Orange, T. Michaeli, and Y. Shechtman, "Deepstorm3d: dense 3d localization microscopy and psf design by deep learning," Nature Methods, vol. 17, p. 734-740, Jul 2020.
[7] H. Ikoma, T. Kudo, Y. Peng, M. Broxton, and G. Wetzstein, "Deep learning multi-shot 3d localization microscopy using hybrid optical-electronic computing," Optics Letters, vol. 46,
p. 6023 6026, Dec 2021.
[8] M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, "Light field microscopy," in ACM SIGGRAPH 2006 Papers, SIGGRAPH '06, 924-934, Association for Computing Machinery, Jul 2006.
[9] L. Cong, Z. Wang. Y. Chai, W. Hang. Shang, W. Yang, L. Bai, Du, Wang, and Q. Wen, "Rapid whole brain imaging of neural activity in freely behaving larval zebrafish (danio rerio)," el.ife, vol. 6, p. e28158, Sep 2017.
[10] Linda Liu, G. Kuo, N. Antipa, K. Yanny, and L.. Waller, "Fourier diffuserscope: single-shot 3d fourier light field microscopy with difffueer"" Optics Express, vol. 28, p. 28969, Sep 2020.
[11] N. C. Pégard, H.-Y. Liu, N. Antipa, M. Gerlock, H. Adesnik, and L. Waller, "Compressive light-field microscopy for 3d neural activity recording," " Optica, vol. 3, p. 517-524, May 2016.
[12] R. Prevedel, Y.-G. Yoon, M. Hoffmann, N. Pak, G. Wetzstein, S. Kato, T. Schrödel, R. Raskar, M. Zimmer, E. S. Boyden, and et al., "Simultaneous whole-animal 3d imaging of neuronal activity using light-field microscopy." Nature Methods, vol. 11, p. 727-730, Jul 2014.
[13] O. Skocek, T. Nöbauer, L. Weilguny, F. Martínez Traub, C. N. Xia, M. I. Molodtsov, A. Grama, M. Yamagata, D. Aharoni, D. D. Cox, and et al., "High- volumetric imaging of neuronal activity in freely moving rodents," Nature Methods, vol. 15, p. 42944229455 432,,432,. Jun 2018.
[14] K. Yanny, N. Antipa, R. Ng, L. Waller, and L. Waller, "Miniature 3d fluorescence microscope using random microlenses," " in Biophotonics Congress: Optics in the Life Sciences Congress 2019 (BODA, BRAIN, OMATMM,MM,,MMR (2019), paper 4,3,.4, BT3A.4, Optical Society of America, Apr 2019.
[15] N. Wagner, F. Beuttenmmulllrr N. Norlin, J. Gierten, J. C. Boffi, J. Wittbrodt, M. Weigert, L. Hufnagel, R. Prevedel, and A. Kreshuk, "Deep learning-enhanced light-field imaging with continuous validation," Nature Methods, vol. 18, p. 557-563, May 2021.
[16] N. Antipa, G. Kuo, R. Heckel, B. Mildenhall, E. Bostan, R. Ng, and L. Waller, "Diffusercam: lensless single-exposure 3d imaging," Optica, vol. 5, p. 1-9, Jan 2018.
[17] L. Liu, V. Madhavan, N. Antipa, G. Kuo, S. Kato, and L. Waller, "Single-shot 3d fluorescence microscopy with fourier diffusercam." in Biophotonics Congress: Optics in the Life Sciences Congress 2019 COODA.BRAIN.NTEN OMA, OMP) (2019), paper NS2B.3. p. NS2B.3, Optical Society of America, Apr 2019.
[18) M. S. Asif, A. Ayremlou, A. Veeraraghavan, R. Baraniuk, and A. Sankaranarayanan, "Flatcam: Replacing lenses with masks and computation," " in 2015 IEEE International Conference on Computer Vision Workshop (ICCVW), P. 663-666, Dec 2015.
[19] K. Yanny, N. Antipa, W. Liberti, S. Dehaeck, K. Monakhova, F. L. Liu, K. Shen, R. Ng, and
L. Waller, "Miniscope3d: optimized single-shot miniature 3d fluorescence microscopy." Light: Science & Applications, vol. 9, p. 171, Oct 2020.
[20] M. B. Ahrens, M. B. Orger, D. N. Robson, J. M. Li, and P. Keller, "Whole-brain func- tional imaging at cellular resolution using light-sheet microscopy." Nature Methods, vol. 10,
p. 413-420, May 2013.
[21] Y. Mu, D. Bennett, M. Rubinov, S. Narayan, C.-T. Yang. M. Tanimoto, B. Mensh, L. L. Looger, and M. B. Ahrens, "Glia accumulate evidence that actions are futile and suppress unsuccessful behavior," Cell, vol. 178, pp. 27-43.e19, Jun 2019.
[22] W. Yang and R. Yuste, "In vivo imaging of neural activity." Nature Methods, vol. 14, p. 349-359, Apr 2017.
11

---

[23] Y. Zhang, M. Rózsa, Y. Liang, D. Bushey, Z. Wei, J. Zheng. D. Reep, J. Broussard, A. Tsang. G. Tsegaye, S. Narayan, C. J. Obara, J.-X. Lim, R. Patel, R. Zhang, M. B. Ahrens, G. C. Turner, S. S.-H. Wang, W. I.. Korff, E. R. Schreiter, K. Svoboda, J. P. Hasseman, I. Kolb, and I.. Looger, "Fast and sensitive geamp calcium indicators for imaging neural populations." 2..2021.111 1121111.08.4677990 Nov 2021.
[24] A. S. Abdelfattah, T. Kawashima, A. Singh, O. Novak, H. Liu, Y. Shuai, Y.-.. Huang, L. Cam- pagnola, S. C. Seemann J. Yu, J. Zheng. J. B. Grimm, R. Patel, J. Friedrich, B. D. Mensh, L.. Paninski, J. Macklin, G. J. Murphy, K. Podgorski, B.-J. Lin, T.-W. Chen, G. C. Turner, Z. Liu, M. Koyama, K. Svoboda, M. B. Ahrens, L. D. Lavis, and E. R. Schreiter, "Bright and photostable chemigenetic indicators for extended in vivo voltage imaging," Science, vol. 365, 699-704, Aug 2019.
[25] K. Monakhova, J. Yurtsever, G. Kuo, N. Antipa, K. Yanny, and L. Waller, "Learned reconstruc- tions for practical mask-based lensless imaging," Optics Express, vol. 27, p. 28075-28090. Sep 2019.
[26] Z. Wang. L. Zhu, H. Zhang, G. Li, C. Yi, Li, Y. Yang, Y. Ding, M. Zhen, S. Gao, and et al., "Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning," Nature Methods, p. 1 1-6, Feb 2021.
[27] K. Yanny, K. Monakhova, R. W. Shuai, and L. Waller, "Deep learning for fast spatially varying deconvolution," Optica, vol. 9, p. 96-99, Jan 2022.
[28] S. Diamond, V. Sitzmann, F. Heide, and G. Wetzstein, "Unrolled optimization with deep priors." arXiv e-prints, vol. 1705, p. arXiv:1705.0004. 7054044,,, May 2017.
[29] J. Dong. S. Roth, and B. Schiele, "Deep wiener deconvolution: Wiener meets deep learning for image deblurring," " Advances in Neural Information Processing Systems, vol. 33, p. 1048-1059, 2020.
[30] C. Peng. X. Zhang, G. Yu, G. Luo, and J. Sun. 'Large kernel matters improve semantic segmentation by global convolutional network." " in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). p. 1743-1751, IEEE, Jul 2017.
[31] N. Mu, X. Xu, and X. Zhang, "Salient object detection low contrast images via global convolution and boundary refinement." in 2019 JEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops (CVPRW), P. 743-751, IEEE, Jun 2019.
[32] O. Rippel, J. Snock, and R. P. Adams, "Spectral representations for convolutional neural networks." in Advances in Neural Information Processing Systems (C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, eds.), vol. 28, Curran Associates, Inc., 2015.
[33] E. Adelson and J. Wang, "Single lens stereo with a plenoptic camera," " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, p. 99-106, Feb 1992.
[34] L. Grosenick, M. Broxton, C. K. Kim, C. Liston, B. Poole, Yang, A. Andalman, E. Scharff, N. Cohen, O. Yizhar, and et al., "Identification of cellular-activity dynamics across large tissue volumes in the mammalian brain," bioRxiv, p. 132688, May 2017.
[35] M. Broxton, Volume Reconstruction and Resolution Limits for Three Dimensional Snapshot Microscopy. PhD thesis, Stanford University, Aug 2017.
[36] J. Goodman, Introduction Fourier Optics. Macmillan Learning, 4 ed., 2017.
[37] O. Ronneberger. P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," " in Medical Image Computing and Computer-Assister Intervention MICCAI 2015 (N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, eds.), Lecture Notes in Computer Science, p. 234-241, Springer International Publishing, 2015.
[38] A. V. Oppenheim and A. S. Willsky, Signals and Systems: Pearson New International Edition. Pearson Education Limited, Jul 2013. Google-Books-ID: U19AQQAACAJJ.
[39] M. Mathieu, M. Henaff, and Y. LeCun, "Fast training of convolutional networks through ffts." 132 [cs], Mar 2014. arXiv: 1312.5851.
[40] N. Vasilache, Johnson, M. Mathieu, S. Chintala, S. Piantino, and Y. LeCun, "Fast convolu- tional nets with fbfft: A gpu performance evaluation," 11X [es], Apr 2015. arXiv:
1412.7580.
12

---

[41] S. Ioffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," 0rXiv:/502.03167 [cs], Mar 2015. arXiv: 1502.03167.
[42] D. Ulyanov, A. Vedaldi, and V. Lempitsky, "Instance normalization: The missing ingredient for fast stylization," arXiv:1607.00022 [cx], Nov 2017. arXiv: 1607.08022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See our summary Section (b) Did you describe the limitations of your work? [Yes] We also discuss limitations in Section 4.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We also discuss broader impacts in Section 4.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yss]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either the supplemental material or as URL)? [Yes] See Ap- pendix A.4, A.6, A.8. We have also released our simulation and optimization library (along with instructions download data) at snapshots cope.
com/Turagaaab/
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A.4, A.6, A.8.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See standard error of the mean values in Table 2. (d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See limitations in Section 4 and Appendix A.6.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We have cited UNets as [37] and DLMD as [25].
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or URL? [Yes] We have released our simulation and reconstruction network code at https:
//githi (igithu.... cww.com////oo/oc coom/Tragaaab// seuurggaaabsnnashheee wwwpaa/ssaaphosssoooooooooo
(d) Did you discuss whether and how consent obtained from people whose data you'n using/curating? [Yes] See footnote about DiffuserCam dataset. Zebrafish data is our own.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "lKI-UATzYxn", "reviewer": "Reviewer_FwUp", "summary": "The paper proposes an end-to-end (E2E) optimized computational imaging method to reconstruct 3D volumes (or 2D images). The method uses simulated optical encoders which encode the input into a 2D encoded phase image which is then decoded by a Fourier transform-based neural network. The method is then evaluated on 3D snapshot microscopy data where the input is a 3D volume, and DiffuserCam-based lensless photography, where the input is the intermediate 2D phase image itself. The method outperforms existing methods and will be openly be made (which the reviewer acknowledges very positively).\n\nThe author’s claims are:\n1.\tA multi-GPU fast implementation of the simulated optical encoder where the parameterized phase mask can be optimized E2E\n2.\tFourier Transform-based decoder converts the intermediate input into Fourier space and does convolution (actually element-wise multiplications) that achieves faster computation of the final output. This helps the network to reduce the locality bias and learn from a wider global context.\n3.\tThey collected extra data to show the merit of their method in a region-of-interest (ROI) specific 3D volume imaging case.\n4.\tSOTA results on the following tasks – 3D snapshot microscopy and DiffuserCam-based lensless photography\n5.\tThis is the first time that a direct E2E optimization of the spatial light modulator (SLM) based optical encoders have been proposed. They back this claim using the simulation result and a single case of implementation on a programmable microscope.\n", "questions": "* A brief explanation of the non-local optical encoder would have been helpful.\n* Are there other baselines you could compare your approach agains?", "limitations": "I believe so.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThe paper addresses an important area in optics and clearly defines the existing limitations of exploiting the latest advances in end-to-end machine learning methods and optimizing the optical encoding system. \n2.\tThe arguments, concepts, assumptions, and methods are lucidly explained by the authors in the paper.\n3.\tThe paper builds on the advances in multi-GPU-based parallelism and efficient deployment of neural networks for a problem that is otherwise not feasible due to limitations in computing. \n4.\tThe author’s assumption that the locality bias by having a small filter size in existing methods cannot effectively capture then information from the image formation process, which is non-local, sounds plausible. To address this problem, the authors propose a method using Fourier transform and using larger filter sizes.\n5.\tRelevant papers have been cited and tricky parts have been described verbose in the appendix.\n6.\tThe overview figure clearly depicts the proposed method.\n7.\tThe use of Fourier domain convolutions reduces the computation time and increases inference speed.\n8.\tThe loss function has been designed to give more importance to the high frequencies of the input which are more important than low frequencies which are relatively less important. \n9.\tThe results on DiffuserCam dataset corroborate the working of the decoder in capturing the high frequencies better than existing methods.\n10.\tThe authors show that their method can indeed be implemented on a programmable microscope as a prototype.\n11.\tThe results on the 3D microscopy dataset show that:\na.\tThe E2E optimization outperforms non-E2E methods in reconstruction quality with fewer parameters.\nb.\tThe proposed method can be optimized for ROI-specific optimizations, that achieve better results than existing methods.\n\nWeaknesses:\n1.\tThe application of this method seems to be in a very limited field which is a differentiable simulation of optical encoders.\n2.\tThe authors could have shown the result on 1~2 more datasets.\n3.\tUNets have been there for a while. Are they indeed the best baseline method to compare the presented method against?\n4.\tThere is no theory or citations to back the claim made in line 136.\n5.\tAn entire multi-GPU setup is required for the optimizations in the proposed method, which makes it not very accessible for many potential users.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TaCuEyY263X", "reviewer": "Reviewer_wkcK", "summary": "This paper proposes using a shallow network with a large convolutional kernel for end-to-end (E2E) learning of computational imaging systems. The proposed large convolutional kernel (used only in the first layer) is implemented in the Fourier domain. The proposed network is used to reconstruct experimentally captured diffuserCam data, where it outperforms SoTA methods. It is also used to E2E design a snapshot 3D microscope. This new design significantly outperforms a U-net based system (in simulation), though it is not compared with lenslet based systems. The authors wrote and plan to release parallelized software which allows one to simulate wave propagation using 8 GPUs.", "questions": "How do E2E designed masks compare to lenselet based masks?\n\nCould the authors speak to why an approximate inverse (i.e., A^dagger y or Wiener deconv) before a U-net doesn't adequately model non-local effects.", "limitations": "Addressed", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nUsing a large receptive field in when dealing with optical systems with large PSFs is well motivated and the proposed design seems to work better than Unets. \n\nProposed method produces SOTA results on diffusercam data\n\nPaper is well-written and easy to read\n\nProposed methodology could lead to interesting new microscope designs\n\nThe multi-GPU simulator could be useful for a number of research groups\n\n# Weaknesses\n\nThe paper doesn't adequately discuss or compare against methods (particularly in the 3D experiments) methods that first approximately inverse the optical system before applying a NNs [3,A]. Learned Wiener filters [A] seem quite similar to the proposed technique.\n\nNo experimental 3D microscopy results (with the proposed optics).\n\nNo comparisons with classical or lenslet based approaches to snapshot 3D imaging.\n\n\n\n[A]Yanny, Kyrollos, et al. \"Deep learning for fast spatially varying deconvolution.\" Optica 9.1 (2022): 96-99.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MsnHPN-jswy", "reviewer": "Reviewer_qEpE", "summary": "This paper has two main contributions: (1) development of a parallelized differentiable wave optical encoder simulator and (2) a Fourier-space neural network which enables non-local optical decoding and outperforms previous deep learning-based image reconstruction strategies. Taken together, these two contributions enable end-to-end optimization of non-local optical encoding/decoding systems. The paper demonstrates both of these techniques on a new dataset of 3D snapshot zebrafish larvae microscopy images and demonstrates the reconstruction strategy on a previously published dataset for DiffuserCam lensless photography. The paper also demonstrates that the learned optical encoders can be implemented in practice on a prototype programmable microscope.", "questions": "The “weaknesses” section is structured as a list of questions I would like answered/clarified. To summarize:\n- Do FourierNets with U-Net style pooling still outperform U-Nets?\n- Is the U-Net in all experiments designed to have a receptive field that covers the entire image?\n- Why are results reported with MSE and MSE+LPIPS loss for FourierNets in Table 1, but only for MSE+LPIPS for the other methods?\n- Is the 2-step training procedure used for the U-Nets as well as the FourierNets?\n- Is the loss function in Equation (4) used to train all methods in Table 2, including the U-Nets?", "limitations": "Yes, the authors have clearly addressed the limitations and the societal impact in Section 4. To summarize, the main limitation identified by the authors is that all of the results in this paper are presented on simulated data.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Note for AC: I don’t have direct experience with 3D snapshot microscopy or differentiable optical simulation — while I read through those details and they seem solid to me, the other reviewers may be more qualified to comment on that component of this work. My experience is more closely related to Fourier-space neural network architectures part of this work.]\n\n# Strengths\n- This paper is extremely clearly written — it is quite easy to follow, even as someone relatively unfamiliar with 3D snapshot microscopy.\n- The experimental results are compelling and show clear improvements from the FourierNet architecture, both qualitatively and quantitatively.\n- The appendix contains extensive, easy-to-follow characterization of the computation time and memory requirements of all methods studied, as well as detailed pseudocode and implementation details for the various algorithms described in the paper.\n\n# Weaknesses\n I cannot find any major technical weaknesses with this paper and have only minor comments/questions/clarification requests here.\n- There are currently two differences between FourierNets and standard ConvNets: The global-width convolution filters, implemented in Fourier space, and the different pooling strategy which crops in Fourier space, instead of, say, max or average pooling. If time/resources are available, a small ablation which compares FourierNets to FourierNets with image-space max/average pooling would further strengthen the argument that it is the global filter extent which is leading to performance gains. However, I don’t think this is strictly necessary for acceptance, because both of the implemented differences rely on the Fourier implementation of the networks used in this paper.\n- Is the U-Net in all experiments designed to have a receptive field that covers the entire image? If not, this comparison should be made to show that, even if the U-net has such a wide receptive field, the local operations at each layer are ill-suited to tasks with a wide point spread function. If so, it may be worth it to clarify this explicitly in the text.\n- Why are results reported with MSE and MSE+LPIPS loss for FourierNets in Table 1, but only for MSE+LPIPS for the other methods? Do the other methods do worse with MSE loss?\n- Lines 219-223 describe a two-step training process — was this used for generating the metrics in Table 2 just when training FourierNets as microscopes? Or also when using the UNets?\n- Is the loss function in Equation (4) used to train all methods in Table 2, including the U-Nets? I assume so, since no other loss functions are mentioned, but if not, ablations are needed to show that the architecture, and not the loss function, are contributing to the improvement observed here.\n- There is limited technical novelty _from an ML perspective_ in this work (neural networks operating via elementwise operations in the Fourier space have previously been developed, as cited in the manuscript). That said, I do think that the use of FourierNets for end-to-end optimization of the types of imaging systems in this paper is novel and a solid computational imaging advance.\n- Very minor corrections: Figure 2 caption’s third sentence should read “Bottom row.” And I believe Line 235 should reference Table 2.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept