Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This manuscript introduces a new method for the reconstruction of continuous variability in cryogenic electron-microscopy (cryo-EM) based on amortized inference and neural network autoencoders. While amortized inference and autoencoders have previously been used for different aspects of this problem, the proposed method provides a more comprehensive approach. Specifically, previous methods have relied on a consensus model for pose estimation or performed expensive pose estimation as part of an alternating refinement process where the main focus has been on estimating the latent states of the molecule. The proposed method performs the pose estimation at the same time as the latent state estimation, achieving significant speedups with respect to the state of the art. The method is validated on numerous synthetic datasets and one experimental dataset.

strengths_and_weaknesses: The main components of the proposed method have been proposed earlier: amortized inference for pose estimation has been introduced by Miolane et al. [15] and for latent state estimation by Zhong et al. [33]. Joint estimation of both pose estimation has been performed by Rosenbaum et al. [24], but only for synthetic data. These previous results are accurately described in the manuscript. As such, the original work in this manuscript consists of combining these existing components into a functioning whole, which is no easy feat. It also does this while achieving significantly reduced running times compared to the state of the art (albeit with a slight loss of accuracy).

The method is promising and the authors provide numerical results to support their claims. That being said, these numerical results do not completely validate the method as they currently stand. First, the synthetic data generated has a very high signal-to-noise ratio (SNR) by cryo-EM standards. This can be seen most clearly by comparing the sample projections in Figure S3, where two synthetic projection images are compared to an experimental projection image. The noise level of the former is much lower compared to the latter. In the text, the authors simply mention that the variance of the noise applied is one without providing a measure of the variance of the clean images. Judging by eye, I would guess that the SNR for the synthetic images is close to one, while realistic cryo-EM data typically has SNRs ranging from one hundredth to one tenth. It is important to note that despite these high SNRs, the proposed method performs worse than the state-of-the-art, with rotation errors on the order of 2× and translation errors typically on the order of 10×.

The results on the experimental data is also concerning. In the supplementary material, the authors mention that “particle images are shifted by their published poses, since the particles in this dataset are significantly out of center”. In other words, the pose estimation is not fully validated on experimental data (specifically, while the rotational component of pose estimation is validated, the translational component is not). While it may well be the case that the particles in this dataset are significantly out of center, this is a common occurrence in many cryo-EM datasets, and it is something that a robust method should be able to handle. Given the fact that an important contribution of the proposed method is its validation on experimental data, this discrepancy significantly detracts from that contribution. Furthermore, the fact that only one experimental dataset is tested also reduces the scope of validity (compared to the cryoDRGN and Relion multi-body refinement papers, which present results on three or more experimental datasets).

The overall presentation of the work is clear, with most concepts being well defined. Given the importance put on amortized inference, it would be good to specify early on what is meant in a formal manner to reduce any possible confusion about the term. In a similar vein, a definition of the Hartley transform (and its relative advantages over the Fourier transform) would be a good addition.o

As stated above, the significance of this paper is the joint amortized inference of pose and latent space applied to experimental data. As such, it provides an important methodological milestone for the processing of cryo-EM data. However, the numerical results do not completely back up these claims. Additionally, the authors also do not make clear what it is about their setup that makes the method successful. In other words, what has been missing from previous attempts at joint inference (such as the work of Rosenbaum et al. [24]) that is addressed by the current work?

questions: – Please provide a formal definition for amortized inference early on (Section 1 or 2) for context.
– What does “disentanglement” mean in the context of cryoVAEGAN (line 108)?
– Another approach for joint estimation of pose and latent state (similar to 3DVA of cryoSPARC) is presented by the hypervolume model of Lederman et al. (2020). It would be good to compare this approach with the proposed model.
– What is calligraphic F in line 134?
– Section 3.1 describes an image formation model entirely in the continuous domain. Please discuss issues of discretization (since we are dealing with discrete data, after all). How is this taken into account?
– Where does D come from in line 156?
– How and why is the rotation “represented in the 6-dimensional space S² × S²”?
– “defoci” on line 196 should be “defocus” (plural is on “parameters”).
– What is meant by major and minor on line 224? (This is explained in the supplement, but needs to be briefly described in the main text.)
– What is meant by “images are masked with a circle of radius 32 pixels for pose search” on line 228?
– Similarly, what is meant by “pose search is done every 5 epochs”? Given that there is no search per se, does this mean that the inferred pose parameters are held fixed (i.e., not recomputed using the corresponding MLP) for 5 epochs in a row before the corresponding weights are updated?
– Why split the data into training and testing? Is this merely for validation purposes or part of the overall workflow? If it is part of the overall workflow, how is a user supposed to perform this split and how is the final reconstruction obtained?
– On line 243, the error is the “ratio of misclassified images” to what quantity?
– In Table 1, why is resolution not reported for the test sets?
– What is meant by “conditioning” on line 245?
– What is K on line 267?
– The Spearman correlation on line 271 is calculated between what variables?
– Why is only PCA of dimension 1 considered in Section 4.2.

limitations: As stated above, the high SNRs of the synthetic data, limited experimental validation, and lack of validation for translational pose estimation in the experimental data detract from the overall claims of the manuscript and should either be resolved or acknowledged in the text.

The authors discuss potential negative societal impact of their work (and the field in general) quite well.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper describes a method, CryoFIRE, to recapitulate the 3D structure of a biomolecule from cryo-EM images using coordinate-based neural model that enables continuous representation. This is inspired by recent advancements of CryoDRGN/CryoDRGN2 for ab initio 3D molecular reconstruction. Different from existing methods, the authors propose to use an encoder to jointly estimate the particle poses and conformation state (a latent vector) from a set of 2D projections. The predicted poses and latent vector are then fed into a feedforward neural network to aggregate each particle image into an implicit representation of the scattering potential volume. The encoder and decoder are trained in an end-to-end fashion using L2 distance. This method achieves comparable results on both simulated and experimental datasets to SOTA. In addition, the paper shows that such autoencoder-based method can significantly speed up the reconstruction time compared to CryoDRGN2 when using a large number of 2D projections.
strengths_and_weaknesses: Strengths:

The paper presents a novel approach for single particle cryo-EM that quantitatively and qualitatively works well. In addition, the paper is overall well written, and the method is well evaluated on experimental dataset. 

Weaknesses:

Some claimed contributions and technical details might need to be clarified and discussed.


questions: Here are my detailed review comments.

1), One of the claimed contributions of this work is confusing, and some clarifications will be helpful. In specific, this work claims that “the first instance of amortized inference for ab initio heterogeneous reconstruction of experimental cryo-EM datasets.” Seems that CryoFIRE uses the same publicly available dataset as cryoDRGN2, and directly takes the results from cryoDRGN2.

2), The discussions around the differences and contributions over existing CryoAI is missing. Both methods use an encoder to capture the high-level poses of particle images and symmetrized loss.  In addition, CryoAI proposes to use two MLPs to represent the complex signals as phase and magnitude separately, while this work seems not adopt this strategy. What results in this different?

3), The rational and advantages of using the symmetrized loss during training in Sec 3.5 is not very instructive at its current state. A deeper understanding and intuition with some ablation studies will be helpful. In addition, how is the symmetrized loss generalizable? Would other training losses such as the statistical distances used in CryoDRGN benefit from this in-plane constrain?

4), While explicitly estimating the poses and amortized variations for parametrizing 3D structure is an interesting direction, recent work in deep learning such as CryoGAN [R1], for single particle cryo-EM has attempted to recover the 3D structure without explicitly recovering the poses using adversarial networks. It may be worth to discuss the relations and differences in the revision. 

5), Consider explaining CryoDRGN2 in detail given that is the state-of-the-art technique and to which all the CryoFIRE results are compared.

6), What would runtime comparisons be for cryoDGRN, for an unsupervised heteregeneous reconstruction?

7), Consider adding more description of the neural network architecture that is the main contribution of this work (more than what is reported on page 5 and page S1 in the supplement).

[R1] Gupta et al., "CryoGAN: A New Reconstruction Paradigm for Single-Particle Cryo-EM Via Deep Adversarial Learning," in IEEE TCI, 2021.

Minor comments:

Figure 1 (c) is not very informative; it is hard to be linked to Figure 2.

limitations: The limitations of current method are missing. Considering single particle cryo-EM is an important but challenging problem, some discussions will be helpful (future work).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a novel method to infer both pose and conformational states of protein structures from raw cryo-EM images in a way that is amortized over the size of the dataset. The authors compared the proposed method with cryoDRGN2 on several synthetic and experimental datasets. They demonstrated that their method requires less time compared to cryoDRGN2 by sacrificing little accuracy. 

strengths_and_weaknesses: The paper is well-written and overall easy to follow. The motivation and importance of the problem are well discussed. The paper provides a thorough overview of the literature. The proposed method is novel to the best of my knowledge.

However, there are several issues with the experiments, baselines, and results. The authors mentioned that Rosenbaum et. al.’s method was the only one to perform amortization for both pose and conformation estimation. In such a case, that particular method should have been kept as a baseline, not cryoDRGN2 since it amortizes for conformation estimation only. Even the comparison with cryoDRGN2 is not very impressive. It is quite usual that methods with an exhaustive search for pose would take more time with more accuracy. If the proposed method, cryoFIRE could achieve better or similar accuracy with less time then it would have been much more impressive. But cryoFIRE performs much worse than cryoDRGN2 even in large datasets.

No comparison has been showed for conformation estimation. Is cryoFIRE as good as cryoDRGN2 for conformation estimation as well? Can it recover some conformations that can not be recovered by cryoDRGN2 or 3DVA or E2GMM?

The paper mentions Rosembaum et al’s method only worked on the simulated dataset, but it did not discuss why it can not be applied to real datasets. What makes it inapplicable for real datasets but applicable for realistically simulated datasets? Or is it the case that the authors are the first to apply the method to the experimental dataset, this is clearly not a big contribution. 

questions: 1. How good is cryoFIRE in conformation estimation compared to other methods?
2. Why Rosembaum et al.’s method can not be applicable to a real dataset?
3. I would suggest taking a look at methods like SpatialVAE [1] and Harmony [2] that disentangles the rotation and translation from cryo-EM images. How are those methods related to yours? The setup of the discriminative model (Section 3.3) looks very similar. 

[1] Bepler, Tristan, et al. "Explicitly disentangling image content from translation and rotation with spatial-VAE." Advances in Neural Information Processing Systems 32 (2019).

[2] Uddin, Mostofa Rafid, et al. "Harmony: A Generic Unsupervised Approach for Disentangling Semantic Content From Parameterized Transformations." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

limitations: The authors discussed the limitations and possible negative social impact of the work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes an ab initio heterogeneous reconstruction method for SPA cryoEM, named cryoFIRE. There are multiple advantages to the proposed approach and most of them come from the ability to jointly estimate both poses and conformations, allowing to skip computationally expensive step of pose search. The authors show the effectiveness of the method on multiple datasets (including one experimental) and compare the approach with SOTA of cryoEM NN-based heterogeneous reconstruction (cryoDRGN/2). The authors promise to provide an open-source implementation of cryoFire upon publication. 
strengths_and_weaknesses: Strengths:
1. The paper is written cleanly and I especially enjoyed the succinct formal description of cryoEM image formation.
2. The method is novel and presents the first instance of amortised inference for ab initio cryoEM heterogeneous reconstruction.
3. Joint estimation of poses and conformations brings one order of magnitude speedup, scaling much better with the ever-increasing amount of data produced by cryoEM.

Weaknesses:
1. As the authors themselves point, parameterisation calculated with cryoFIRE is not uniquely defined ("a rotation of the molecule can be equivalently represented by R_i or by a change of conformation state z_i"). The proposed workaround in the paper is to use training schedule and have "pose-only" phases at the start of training and then every N epochs. Additional work is required to be able to decouple rotation from conformation without training schedules.
2. cryoFIRE is directly compared with cryoDRGN and cryoDRGN2 (Table 1), as well as cryoSPARC (Table S2). I don't think that this represents software frameworks currently used by cryoEM groups.
questions: 1. The datasets used for the experiments have quite high SNR (especially synthetic ones), at least judging by the samples in the figure S3. Did you explore whether there is a limit of SNR at which the method does not work?
2. Even though Table 1 and Table S2 show that cryoFIRE is slightly bit less precise than cryoDRGN2 and cryoSPARC, the speedup in training and especially testing is more than worth the tradeoff. However, what do you think is the reason for decreased precision? What is the number of parameters in the models and have you experimented with changing number of parameters/layers?

limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: The paper studies the problem of simultaneously estimating poses and conformation of a biomolecule, cryo-EM images. It presents a pipeline which integrates reconstruction and conformation estimation, leading to significant time savings compared to methods that alternate between accurately estimating poses and conformations. Conventional pose estimation is expensive because it involves searching over the space of rigid body motions, by repeatedly rendering images from from various viewpoints. The paper proposes an autoencoder-like structure, which generates conformation and pose estimates for each image, which are used by a decoder to produce an estimate of the conformation. 

Reviewers generally evaluated the paper positively, noting that it achieves an order of magnitude speedup compared to a conventional baseline. Reviewers note that although elements of the paper have previously appeared (the use of autoencoder-like architectures for cryo-EM reconstruction, the use of amortized inference over conformations and poses), the combination employed here is novel. Reviewers generally appreciated the paper’s experimental results, while raising some concerns about the SNR and baselines for comparisons. Finally, reviewers noted that the paper provided a clear exposition of both cryo-EM and the proposed techniques. Overall, the paper exhibits a well-chosen combination of learning techniques, which lead to performance improvements for a problem of significant scientific interest. 
confidence: Certain
Decision: Accept