Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors tackle the problem of image orientation estimation in single particle cryo-electron microscopy. This imaging technique generates a dataset of ~10^{5-7} 2D projection images of a 3D protein with unknown pose (elements of SO(3)xR2). Once poses are inferred, the 3D structure may be reconstructed with standard tomographic projection techniques. To estimate the distances, the authors propose a Siamese convolutional network architecture, which takes as input a pair of images, and attempts to predict the distance between their orientations (parameterized as quaternions). Once this distance function is learned (from a training set of posed images on a single protein), gradient-based optimization is used to infer each image’s pose. The authors show results on two small, synthetic datasets of 5000 images.
main_review: I appreciate the detailed study performed by the authors, however I do not think this paper is suitable for publication at NeurIPS as it focuses on a particular application without introducing any technical novelty, and the task is somewhat contrived for the application domain. 

The task of ab initio reconstruction for homogeneous proteins is well-studied/solved. The authors should benchmark against a state of the art algorithm. 

Fundamentally, the method relies on a training set of previously posed images, so the task they are addressing is not a realistic setting. As the authors discuss, if this approach were extended to be trained across multiple datasets and could predict orientations for a new (real) dataset zero-shot, it would be very high impact; however I am not convinced from the current results that extending this approach is promising. The paper lacks theoretical grounding, which the authors admit, and I am not convinced that there are any generalization properties for functions on distances between projection images (to unseen proteins).



limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper uses deep learning to recover the orientations of molecules measured in the context of single particle cyroEM. The proposed method first estimates the pairwise distances between the angles of molecules based on the their projections. This is accomplished by embedding the projections into a feature space that was trained such that the cosine similarity between two features is equal to the distance between the orientations of the two molecules. Next, the authors solved an optimization problem (3) to recover the orientations, up to global ambiguities, from the pairwise distances.

main_review: 
# Originality
Recovering cryoEM orientations or structure with DL is not new, but the previous works I'm familiar with have mostly focused on solving the task with dimensionality reduction. I haven't seen a paper tackle the problem by learning to estimate the pairwise distances between angles.

# Quality
The paper did a good job of validating each proposed contribution in isolation. Simulations results were overall thorough, though there was no real experimental data.

# Clarity
Well-written

# Significance
It's not evident the proposed method will work with real data and the method was trained on the same molecules whose orientations it was going to reconstruct.

It's also unclear if the proposed methods improvements will actually improve the final quality of cryoEM reconstructions. This tasks performed in this paper are essentially preprocessing steps that recover an initialization that is fed into another algorithm (e.g., expectation maximization and RELION) that would reconstruct the structure. It's unclear if the improvements in the init help the end result.

# Related work
Recent work in DL for single particle cryoEM has attempted to recover the 3D structure without explicitly recovering the angles using adversarial networks [A]. It may be worth mentioning.
[A] Gupta, Harshit, et al. "Cryogan: A new reconstruction paradigm for single-particle cryo-em via deep adversarial learning." BioRxiv (2020).

# Questions
It's my understanding that some molecules will preferentially align in certain directions, so that the view angles in cyroEM are not uniformly distributed. This is a major issue for embedding-based methods. How would non-uniformly distributed angles effect the proposed method?


limitations_and_societal_impact: Authors are transparent about the limitations of current method. Single particle cryoEM is an important problem.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, “Learning to recover orientations from projections in single-particle cryo-EM,” the authors present an approach to recovering unknown projection orientations based on learning to predict the distance between projection angles given observed projections. Given these predicted projection angle distances, it is possible to recover the orientations themselves by solving for the orientations with distances that best match the predicted distances. The distances themselves are predicted using a Siamese neural network that is trained on projection pairs with known orientation distances. The authors demonstrate this approach on two synthetic datasets.
main_review: Although the method has some clear limitations and is only demonstrated on synthetic datasets, this paper presents a compelling and well described method and proof-of-concept demonstration. The paper is well written, and the problem and method are well motivated with relevant background. Specific comments follow below.
1.	The big question: how will this work in practice on real cryoEM datasets where orientations are not known? Even training this on real cryoEM datasets is potentially problematic because the orientations of the particles were estimated to produce the reconstruction. Will a network be able to generalize over particles from different structures?
2.	What happens when the particles are not derived from single rigid structure (i.e., there is conformational heterogeneity)?
3.	When minimizing (4) to solve for the orientations from the distances, how are the orientations initialized? Similarly to t-SNE and other distance-based embedding methods, it seems like the objective is non-convex and so orientations may be mis-estimated due to poor initialization. How quickly does the minimization converge?
4.	It would be useful to see a comparison with orientations estimated using a common lines approach (e.g., http://spr.math.princeton.edu/orientation). It would also be useful to see the orientations estimated by an iterative reconstruction procedure as baselines to better understand how well the orientations in these datasets can be estimated by other approaches.
Overall, I think there are some nice ideas here that are well motivated and described. The main weaknesses are the question about how this will apply to real cryoEM data and the lack of strong baselines for orientation estimation.
limitations_and_societal_impact: Limitations and societal impact are sufficiently addressed.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work presents a method for estimating the viewing directions of projection images in single-particle cryo-electron microscopy (cryo-EM). The method consists of two steps, first the distance between the viewing angles are estimated using a neural network, then these distance estimates are combined to yield a viewing angle estimate for each projection image. The method is trained and evaluated on synthetic projection images, where it is shown to perform well in producing reconstruction distance estimates are combined to yield a viewing angle estimate for each projection image. The method is trained and evaluated on synthetic projection images, where it is shown to perform well in producing reconstructions of moderate resolution.

main_review: This paper presents an interesting method for ab initio reconstruction in single-particle cryo-EM, a difficult problem that stands to benefit from data-driven approaches due to the high noise content of the data, which is otherwise quite structured. The proposed method is promising, but numerical validation in the present work is lacking, especially with respect to comparisons with existing methods or experimental data. Instead, the performance of the method is evaluated in a synthetic-data setting without comparison to other methods and with a network that is trained on the same molecule. It is therefore difficult to judge the wider applicability of the method, despite its inherent appeal. If this type of validation could be provided (even in partial form), I would recommend this paper be accepted as part of the proceedings, but otherwise I ask that it not be included. More detailed comments follow.

As stated above, the problematic component of the work concerns the numerical validation. The motivation and description of the method is quite sound, but could be better explained in parts. For example, the authors switch back and forth between Euler angles θ_i and quaternions q_i (they are simply introduced without explanation on line 63), which can make for confusing reading. The authors also state that the mapping from S³ to SO(3) is a double cover, but do not explain whether this is a problem or not for the problem setting. On lines 120–121, the authors also discuss size invariance, but it is not clear why this is necessary. When are projections expected to be of different sizes? This should not be the case in the numerical experiments discussed since the training data and testing data arise from the same protein structure. A similarly confusing statement is on line 127, where the authors claim that “a space of n_f = 4 dimensions does not have room for G_w to represent other factors of variation”. What does this mean? Why are we constrained to have n_f = 4?

Constructing viewing angle estimates from a set of distance estimates also deserves some closer discussion in Section 2.3. Indeed, constructing a global set of angle estimates from smaller local distance estimates can lead to inconsistencies, as pointed out by Zhao and Singer, 2014 (Section 2.1). Why do we expect that constructing small subsets of the embedding at a time will result in a globally consistent embedding?

For the evaluation method described in Section 2.4, it is a bit problematic to use a non-deterministic error measure since it induces its own “error” for each run. It would be better if this could be replaced with a deterministic error. One way to achieve this is to replace the L¹-type norm used on the rotations and instead use a Frobenius norm on the rotation matrices. In this case, the optimal rotational alignment between the sets can be calculated by an SVD. That being said, perhaps the authors have other reasons for choosing the error measure that make it more appropriate than the simpler Frobenius norm. If that is the case, it should be motivated in the text.

For the numerical validation, it is striking that only uniform distributions of viewing angles are tested. First, these are relatively uncommon in experimental data, with a preferred orientation being the norm. Second, while many algorithms tend to work well for uniform distributions, that is not necessarily the case with non-uniform distributions. It would therefore be a good test of the approach to evaluate its performance in this regime.

A big problem with the evaluation is the choice of datasets. The authors acknowledge this in the discussion, but all the same, it would be good to include the results of some preliminary experiments here. For example, the networks are trained on one of two molecules. What happens if one network is used to predict the distances for the other dataset? Does performance suffer greatly or is there a degree of robustness here. What if the network is trained on both datasets? The universality of the approach will be crucial when extending the method to experimental data, so it is important to know what can be expected here, even at a preliminary level. Other questions related to training mismatch concern noise levels. To what extent can we train on one noise level and test on another?

When it comes to the noise levels, it is hard to extract a useful context. For example, the authors give the noise variance of the data, but without knowing the variance of the images, it is not obvious if this is a high or low noise level. Later in the text, we see that σ² = 16 corresponds to an SNR of –12 dB (which presumably means an SNR of 10 ^ (-1.2), but we are not given the formula). This is a low SNR, to be sure, but not exceptionally low for experimental cryo-EM data. Judging from the images shown in Figure 11 (which presumably have σ² = 16), these are indeed of moderate SNR for cryo-EM.

Another piece of the validation section that is missing is comparison to other methods. In the introduction, a host of alternative methods for ab initio structure determination are listed, but none are applied to the synthetic data used to evaluate the proposed method. It would be simple enough to run the synthetic data through the ab initio pipeline of Cryosparc, Relion, or Aspire to provide a simple baseline for comparison. Right now, the viewing angle accuracies and reconstruction resolutions are hard to place since there is no basis for comparison. This is especially important since it would provide an indication of the benefit derived from a data-driven approach, where specific features in the images are used to estimate the distances. Due to the training–testing regime, it is expected that the proposed method would perform better in this setting (since it possesses a much stronger prior compared to the baseline methods), but this needs to be validated numerically.

Some minor comments:

– Constant references to figures outside of the main text makes for tedious reading. I would try to include these in the main text (perhaps by cutting other figures) or remove references entirely. If this cannot be resolved, I suggest labeling these figures so that it is clear that they refer to supplementary material (Figure S1, etc.).

– On line 100, it should be clarified what is meant by the “magnitude” of a rotation.

– On line 102, what does it mean the “design” the estimator, and why is it intricate or impossible? What do invariants have to do with this?

– On line 120, a CNN provides equivariance (covariance) not necessarily invariance to translation (unless a pooling layer is included).

– On line 143, “our loss function” is referred to but has yet to be introduced.

– On lines 172, 244, 248, 252, and 268, spaces are missing between the number and the unit (Å).

– On lines 180, 191, and the caption of Table 1, there is an extra space after the comma in the numbers larger than one thousand.

– In footnote 6, sampling the Euler angles from uniform distributions does not yield a uniform distribution on SO(3). The colatitude (θ₂) has to be reweighted.

– On line 244, “an” should be removed.

– On lines 228–229, what does it mean that “the observed overfitting indicates that more training data should further decrease the sensitivity of the SNN to noise”?

– On line 240, “robuster” should be “more robust”.

– On line 253, why does higher resolution of the ground truth imply better reconstruction resolution?

– Several article and journal titles are improperly capitalized in the bibliography, including for references [11], [12], [18], [20], [21], [28], [30], [31], [32], [40], [41], and [43].

limitations_and_societal_impact: The authors discuss the limitations of their model and their numerical validation at the end of the manuscript. When it comes to the validation, I think more can be done (as stated in the main part of the review). This being an application of data-driven methods to a problem in structural biology, I do not see any potential negative societal impacts that would emerge directly from this work. The authors are, however, helpful in providing an estimated GHG budget for training their model.
review_title: null

Meta-review: The Cryo-EM problem is of significant importance and ML techniques for it are in scope for the NeurIPS conference.  As the submission did not provide a novel machine learning algorithm, the paper was evaluated primarily on its potential impact in application.  There was a consensus among the reviewers that the paper suffers from methodological concerns, including a lack of comparison to appropriate baselines given the same setup, the generalizability of the method toward proteins not seen in training data, and unclear performance improvement as part of a full cryo-EM pipeline.  Consequently, it is difficult to know how significant the result is in application.  The authors are encouraged to address these concerns in a future submission.  

Decision: Reject