Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a method for heterogeneous cryo-EM reconstruction that fits 3D density volumes as a linear combination of voxel arrays where scale factors are inferred with an encoder network in a VAE framework. The model for volumes is represented in Fourier space and trained on projection images via the Fourier slice theorem (i.e. as 2D planar slices of 3D volumes) to reduce the memory requirement for backpropagation. The authors compare their method against cryoDRGN (a VAE-based model with a coordinate MLP decoder) and show slightly higher resolution reconstructions with their voxel-based generative model on a small synthetic dataset. The method yields qualitatively similar results to cryoDRGN on an experimental dataset of the Plasmodium falciparum 80S ribosome.
strengths_and_weaknesses: Strengths:
* Novel formulation of cryo-EM heterogeneous reconstruction, which mixes and matches the generative model from 3DVA (linear combination of voxel arrays) and inference method from cryoDRGN (variational encoder MLP).
* Rendering volumes is fast and interpretable, i.e. a linear combination of K 3D voxel arrays.
* The paper is well written and easy to follow

Weaknesses:

While the method is interesting and could become a useful tool, the evaluation is flawed/limited.

* The authors state that one of the main advantages is the computational cost for training (218 min for their method vs. 1638 min for cryoDRGN), but the training times are reported for a fixed number of training epochs without considering the convergence rate of the different methods. 
* Testing on synthetic data is useful for validating the method since there is ground truth, but the synthetic dataset used here seems a little too simple to be realistic (only 10k images of 10 discrete states at a very high signal-to-noise ratio of 2/3). 
* The authors mention a novel backpropagation method, but there is very little detail on what this method is. Is this just using the Fourier slice theorem to render 2D slices instead of 3D volumes (e.g. in cryoDRGN and in the voxel-based method described in Ullrich et all UAI 2019). Could the authors elaborate?

A minor comment:
* I don't follow some aspects of the motivation (Section 4.1). The authors point out that the cryo-EM signal is high-frequency in the Fourier domain and coordinate MLPs like cryoDRGN/NERF poorly capture high frequency signals unless a high-frequency positional encoding is provided in the input. But cryoDRGN already uses a positional encoding for this very reason, so what's the issue? Later in the results section, much of the results are instead focused on the training/render time of a voxel-based representation. The authors also discuss at length many advantages of a real space formulation, which further confuses the motivation of the work (a Fourier space representation). 

questions: * How does this method compare to 3DVA on the tested datasets? Can the authors discuss the advantages/disadvantages of this method compared to 3DVA?
* How is the model initialized and how much does this impact the convergence/accuracy of the reconstruction?
* On the synthetic dataset, the reconstruction accuracy of just a single volume is quantified. It would be good to verify that the accuracy is consistent across all the different states.
* There are some architectural and training details that are missing, e.g. the size of the network for encoding images and decoding scale factors. Training details like the batch sizes are also missing, which has a huge impact on training times, especially when comparing methods by the number of epochs trained.

limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes to replace the coordinate-based Fourier space cryo-EM reconstruction approach in CryoDRGN with a volumetric Fourier space representation. The idea is that this volumetric representation allows the model to capture high-frequency variation typical of Fourier space data better than coordinate-based networks, which implicitly tend to captures smoother structures. While the volumetric representation increases memory requirements especially during backpropagation, the paper introduces a sparse back propagation strategy which takes advantage of the sparsity of the projection operation in the cryo-EM forward model. The paper demonstrates similar or better performance on reconstructing two datasets of cryo-EM images while requiring significantly shorter training and inference times, because forward passes do not require individual inference runs for each coordinate.
strengths_and_weaknesses: # Strengths
- The paper is quite clearly written and provides enough background for a reader outside of cryo-EM to easily understand the problem setup and categories of competing approaches, as well as some of their tradeoffs.
- The reported savings in computational resources from the new method are significant, intuitively make sense (in that it is no longer necessary to run individual inference passes for each coordinate), and would be very useful for users of cryo-EM reconstruction tools.
- The sparse backpropagation scheme is clever and makes use of the volumetric representation effectively.

# Weaknesses
- **It is not clear that the reconstruction quality of the proposed method is significantly higher than that of cryoDRGN.** This may be because I am not an expert in reading these kinds of images, and I am willing to change my mind on this. But as the manuscript currently stands, I can see differences between Figure 2B and 2D, but I’m not sure how to tell which is correct or why the proposed method reconstructions are better. Perhaps it would be helpful to point these out explicitly with arrows or more detailed text descriptions. Similarly, I am not sure what the salient takeaways are from the visual results in Figure 3. The FSC plot in 2E does show slightly higher FSC correlation than the cryoDRGN method, but this difference appears to decrease at high resolution, where I presume these methods would be most useful.
- **It is not clear that the proposed method makes it easier for the networks to capture “local, high-frequency changes.”** While I agree that the coordinate-based approaches may suffer from this issue, it is not clear to me that the proposed approach is significantly better — in particular, I imagine the tri-linear interpolation (lines 204-205) might correspondingly struggle to handle very high frequency changes in the Fourier-space data. Perhaps the tri-linear interpolation does provide good results if the high-frequency changes are greater than ~1 pixel in width, but if so, I would expect the coordinate methods to also capture this reasonably well. This is further supported by my previous point that it is not clear that the proposed method results in significantly higher reconstruction quality, so at a minimum, some more discussion of this issue is needed.
- (Minor) It may be helpful to report the real memory usage of both CryoDRGN and the proposed method so that readers can understand this tradeoff a bit more concretely. Relatedly, I see the parameter counts for cryoDRGN in Appendix C.2, but do not see these for the proposed method -- please provide these to compare.  This will give a rough sense for the expressive capacity of the models relative to each other (and, depending on the numbers, may make the point about capacity in lines 339-348 more concrete).
questions: - What aspects of the visual results lead to the claim that the reconstruction quality is improved with the proposed method (lines 256-257)?
- How is the tri-linear interpolation able to handle the very high frequency changes in Fourier-space data which is ostensibly problematic for the coordinate-based approach?

[My suggestion may change based on the responses to the above two questions, but as it currently stands, my suggestion is to remove claims about improved performance while focusing on the computational savings, which are more thoroughly justified. Or, if results using spatial priors capitalizing on the volumetric representation significantly improve the results, that could strengthen these claims -- but I don't think these experiments are necessary for publication of the paper as is.]
limitations: The authors describe a key limitation of their method in the last paragraph of Section 6 and address societal impacts in the "Broader Impacts" section. I believe that the proposed method has higher memory requirements than cryoDRGN, which I would encourage the authors to discuss more explicitly in the Discussion section as well.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes an approach for learning a volumetric Fourier reconstruction in a VAE framework for heterogeneous reconstruction of SPA cryoEM. The authors bring attention to the fact that the previous related work of coordinate-based reconstructions can be improved by learning Fourier representations on a regular cubic grid. The new approach is more efficient and reconstructs with better FSC than current SOTA but at one-seventh of the computational cost. 
strengths_and_weaknesses: Strengths:
1. Faster and more efficient method makes cryoEM more accessible, allowing to iterate faster.
2. VAE framework, coupled with the previous point, allows interactive analysis of latent space (i.e. heterogeneouity), in contrast with discrete classification in the common cryoEM approaches.   
3. The method is also better suited for cryoEM modality than previous coordinate-based networks.
4. The method is validated on both simulated and experimental datasets and shows better FSC than the closest related work.

Weaknesses:
1. No comparisons with current defacto standard cryoEM software packages like RELION and cryoSPARC. For practical usage in cryoEM, it would be necessary to compare the proposed method with current standards. 
2. The method expects projection angle, shift and CTF parameters to be provided. The authors however mention that it should be doable to estimate them and leave it for the future work. 
questions: 1. AFAIK, the paper or supplemental material makes no mention of the memory usage and the number of parameters. Moreover, you mention that the number of volumes for structural basis and volume size has major impact on memory footprint. Could you please expand on memory usage and at least approximately how it would scale?
2. Did you encounter any cases where your method fails or which you expect to fail? For example, do you believe the method would still work with low SNR datasets or datasets with many classes?
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: The paper presents a new cryoEM reconstruction algorithm for data with multiple structural states for the same protein. In contrast to previous approaches which use a coordinate-based implicit representation of reconstructed density, this paper reconstructs an explicit volumetric grid and the optimization is formulated in the Fourier domain. Good accuracy and faster convergence is demonstrated, in comparison to prior methods.

While all reviewers agreed that the method constituted an important contribution to the field, they also pointed to shortcomings in the evaluation of the method. In response, the authors added analysis of the convergence based on FSC-based resolution estimates, and offered to add a comparison to the 3DVA which uses a similar representation to the proposed method.

Based on the reviews and author responses, I recommend acceptance of this paper. I urge the authors to follow through on "writing a more expansive comparison to 3DVA, which will be included in the camera-ready version of the manuscript". 
confidence: Certain
Decision: Accept