Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces the use of Fourier convolutional neural networks to model both image encoding and reconstruction processes. These neural networks learn multiplicative weights in the Fourier domain, enabling efficient global convolutions (i.e., convolutions with kernels the size of the entire image), which are particularly important for imaging applications with a wide PSF. These networks compare favorably to UNets for the task of joint 3D snapshot microscope parameter optimization and image reconstruction. These networks also outperform the state-of-the-art architectures for a lensless imaging reconstruction task.
main_review: Strengths
-------------
- The paper is well written and easy to understand.
- The experimental results are promising. 
- I really appreciate the thorough limitations section.


Weaknesses
-----------------
- The paper has limited technical novelty when compared to a previous paper which implemented Fourier-domain networks, and does not currently contain experiments showing that the differences in approaches provide a meaningful improvement (see Detailed Comments).
- The paper is missing some implementation details which affect the reader’s understanding of the Fourier structure of the neural network (see Detailed Comments).

Detailed Comments
--------------------------
- The following paper contains very closely related work and should be cited and discussed (I am not affiliated with this paper):   

    Rippel, Oren, Jasper Snoek, and Ryan P. Adams. "Spectral representations for convolutional neural networks." Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 2. 2015.  

    In particular, that paper proposes to learn multiplicative weights in Fourier space instead of learning image space convolutional kernels. The learned weights are parameterized to correspond to convolutional kernels with limited extent in the image space, which outperform image-space convolutions on a CIFAR classification task (Fig. 5a). This paper does differ from Rippel et al. in that the the learned convolutions have a kernel the size of the full image. It thus seems necessary to show that it is the global extent of the filters in this paper, and not the fact that the optimization is being performed in the space of the multiplicative Fourier weights, is driving the performance improvements. Further, that paper previously introduced the spectral cropping approach described in this paper for doing multi-scale learning in the Fourier space (although this is used for a classification task and not in the context of a U-net architecture). 

- I believe that the input images and output images of the networks are real-valued, and thus have a complex-valued Fourier transform with conjugate symmetry. (I do not have experience with this type of imaging and am going off Eq. 12-17). I see that the learned weights are complex valued.  How is the real valued-ness of the output of each layer maintained? Are the multiplicative weights chosen such that the output in Fourier space has conjugate symmetry? Since the core contribution of this paper is the Fourier network, these details would be useful to include in the main text, along with the fact that the multiplicative weights are complex-valued -- I think this is currently only mentioned in the Appendix.

- Could you provide a more concrete sense of the memory requirements of the FourierNetRGB Architecture as compared to those of Le-ADMM-U and/or a comparison of the number of parameters? I see these for the other Fourier architectures but not FourierNetRGB in the Appendix. If the memory requirements/number of parameters is significantly smaller for Le-ADMM-U, a comparison may be needed with a version of that architecture scaled to match for fair comparison. I understand for the microscopy experiments that the imaging simulation requires significant memory, which is what motivates the multi-GPU implementation. However, it’s hard to know concretely what that means for the lensless imaging comparison, since only reconstruction is being performed in this case, and it would be good to know that the comparison in Table 3 is fair.  

    Relatedly, could you provide the total training times for the architectures in the lensless imaging example? I assume the times in Tables 2 and 3 are the total inference time for reconstructing a single image; it is good to see that FourierNet is faster than Le-ADMM-U^2 there, but I don’t have a sense of whether the 20ms difference is relevant for a practical application. It would be great for readers to know whether there are significant differences in training time when choosing between these methods.
summary_of_the_review: This is a well-written paper with good reconstruction results. In its current form, the paper lacks important comparisons/contextualization with a very closely related work (Rippel et al.), which previously introduced many of the main technical ideas in this in this work. Also, for the lensless imaging task in this paper, more details are needed to understand whether the resources required for this method and the state-of-the-art are comparable. 

Overall, I would be inclined to improve my rating if the paper is extended with (1) a more careful comparison with Rippel et al., demonstrating which specific aspects of this work provide innovation and improvement over previous approaches and (2) more details about the resources required for the lensless imaging experiments, to better understand whether the comparison provided is fair.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces an end-to-end optical encoder and deep learning decoder optimization for 3D snapshot microscopy. The main challenge is that the decoder needs to be able to handle the global PSFs used in the optical encoder in 3D snapshot microscope. This is difficult to be achieved in conventional UNet architectures as the global context requires many layers that is computationally expensive. The authors use a Fourier convolutional network that achieves the global context in a single layer with a much lower computational cost. The method is demonstrated on simulated data of one volume with four different fields of view. The authors also applied the technique to lensless imaging.

main_review: #### Strengths
* The idea of using Fourier transform to achieve large-size PSF is interesting.
* The design of the Fourier network is well explained: There are some features to reduce the computation in the Fourier network (e.g., save the parameters in Fourier space to save a padding operation) and they are explained in detail. The explanation of the multi-scale features in the Fourier domain is also helpful to understand the effectiveness of the proposed architecture.
* The paper has a nice section on limitations.

#### Weaknesses
* Lack of evaluation: While this paper introduces an interesting architecture, the results are not sufficient to demonstrate the effectiveness of the architecture. First, the 3D snapshot microscope is evaluated only on simulated data. As the author mentioned in the limitations paragraph, the real optical encoder will have some artifacts because of aberrations and calibration error, and these effects need to be evaluated. Second, the evaluation is only on one type of volume (i.e., Larval Zebrafish). While the paper shows additional results on lensless imaging, it cannot support 3D snapshot microscopy, which is the title of this paper.
* The result from UNet is unclear to me. Although it is computationally inefficient, UNet should be able to achieve global context if it has enough layers and thereby produce similar quality of results. However, in Figure 2, UNet provides much worse results compared to FourierNet. Why does UNet provide such bad results? Is it because the parameters of UNet are not sufficient to handle the global context?
* Table 3: Why is UNet much faster compared to the FourierNet results? This is the opposite of what is claimed in the paper. Also, why is the FourierUNet result missing in this experiment?
* I understand Lensless imaging results are included to demonstrate the effectiveness of Fourier network, but it is somewhat unnatural to have this new application abruptly considering the title of the paper starts with “3D snapshot microscopy”

#### Minor comments
* Figure 2: What is the 3.00s in the ground truth of the second row?

##### Typos
* “field field” in the 3rd paragraph of A.1
* Table 3: unnecessary line break in the caption
* Table 4: unnecessary line break in the second row.

summary_of_the_review: This paper proposes an interesting architecture for the end-to-end optimization of the optical encoder and deep learning decoder for 3D snapshot microscopy. In particular, the authors introduce the Fourier network to handle the global PSF in the encoder with a lower computational cost compared to conventional UNet architecture. However, the method is evaluated only on simulated data, with a single type of object, and the explanation of conventional UNet result is insufficient to demonstrate the effectiveness of the proposed architecture. So the decision is weak reject.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a neural network architecture that applies a series of convolution kernels on the Fourier domain to focus on global information of the captured images. The authors make use of the Fourier-plane phase modulation and apply this idea to single-shot 3D microscopic imaging using the Fourier-plane phase modulation for 3D microscope and lensless imaging. Also, this is further extended to the end-to-end optimization of the phase modulation and the reconstruction network. An extensive synthetic evaluation is conducted to validate the effectiveness of the proposed method over existing non-Fourier neural networks in the context of computational imaging.
main_review: This paper is well written with a clear description. I like the idea of incorporating Fourier convolutional neural networks to more efficiently propagate gradients to the global structures, which allow for handling large PSFs often desired in high-dimensional imaging. Although the proposed architectures are simple modifications to the existing CNNs, their motivation is clear and demonstrates the advantages in synthetic experiments. In addition, all the evaluations are pretty well designed and executed. 

Nevertheless, I have some questions and concerns about the claimed contributions shown in Section 1.2. 

First, the authors highlight that the parallel differentiable wave optics simulation achieved the optimization of 10^6 phase-modulating pixels. It is not clear whether achieving 10^6 pixels as optimization variables can be only achieved using the parallel simulator. Also, having 10^6 pixels is necessary to achieve the presented performance? In other words, I wonder how the performance of the proposed method would vary as the pixel count changes. Even though this is often regarded as a training detail in most end-to-end optimization papers, validating this would make the main contribution of this paper clear to readers because it could explain the motivation for the main contribution. 

The second concern is, probably as expected by the authors, is the missing real-world experiments. The domain gap between the synthetic and real experiments is generally large for the end-to-end optimization of optics and reconstruction, making it challenging to incorporate many optimization variables for the optics encoder. This means that directly applying the proposed high pixel-count optics optimization to real-world experiments may impose challenges partially conflicted with the core idea of increasing optimization parameters in this work. I do not believe that this question comes as a surprise to the authors, but a more detailed discussion on this potential problem is worth to be given in the paper.
summary_of_the_review: In summary, there are some major issues better be addressed in the paper. However, I don't see them as a critical barrier for blocking this paper from being accepted, given its high-quality execution and potential impact on computational imaging research.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a convolutional neural network architecture for designing phase patterns to display on a spatial light modulator in the imaging path of a microscope, as well as decoding the resulting image into a 3D volume. The authors propose to use Fourier features in the decoding stage to enable features that are more strongly nonlocal than in regular (primal-domain) convolution than what they claim would be possible to achieve using "vanilla" UNet/ResNet-like architectures. They demonstrate their approach for snapshot 3D microscopy as well as the reconstruction of light fields from input captured using lensless system (with randomized optical elements).

main_review: Positive:
* The paper is nicely written, illustrated. Nice results.
* On the method side, the integration of Fourier-domain layers, while not necessarily 100% novel, seems to do its job quite well and could inspire follow-up work in computational imaging.
* The method is validated against multiple scenarios including an existing database and newly captured data, as well as simpler, more obvious architectures..


Questions:
* I would assume that Fourier transforming the signal is not a new concept in the machine learning community. Among the things that come to mind are papers like Rippel et al., Spectral Representations for Convolutional Neural Networks, NIPS 2015 - but also other interpretations like "positional encoding" (Vasvani et al., Gehring et al., both 2017). Efficient large-kernel convolutions by virtue of the convolution theorem comes up as the main application scenario when doing a web search on "Fourier CNN". I would appreciate if the authors could clarify the relation of this work to those.

Doubt:
* Doing the Fourier convolution in various frequency bands (by cropping) seems somewhat redundant, as the downsampling amounts to cropping in Fourier domain and so one and the same frequency shows up in different layers. Could the desired effect (giving more weight to low frequencies) not be achieved just as well by appropriate frequency-dependent weighting or feature depth?
summary_of_the_review: In my opinion, the foremost contributions of this paper are in the application and the pipeline, and not quite so much in machine learning methodology as the main idea (Fourier convolution) has been around for quite some time. The proposed pipeline is thoroughly motivated, described and evaluated. To me, this looks like an acceptable paper.

That said, I'm not an expert on the application (though it does feel like a bit of a niche). My relation to machine learning is that of a casual user rather than a contributor of hardcore methodology. So I'm looking forward to reading the other opinions on the work.
review_title: null

Meta-review: The paper proposes an application of the CNN to the microscopy problem of constructing 3D volumes from 2D captured images. The four reviewers thought the paper was a straightforward application of existing techniques to a new problem, while they were borderline towards accept the overall sentiment was that the technical novelty was very low from an ML perspective, and the ICLR community would only find the application potentially of interest. (Two reviewers changed from borderline reject, while the other two chose not to change their scores following the authors’ request.)
Decision: Reject