Learning to recover orientations from projections in single-particle cryo-EM
Authors: Jelena Banjac
Laurène Donati
Michaël Defferrard
Venue: NeurIPS 2021 Submitted (2021)
Keywords: metric learning
pose estimation
cryo-EM

Abstract: A major challenge in single-particle cryo-electron microscopy (cryo-EM) is that the orientations adopted by the 3D particles prior to imaging are unknown; yet, this knowledge is essential for high-resolution reconstruction. We present a method to recover these orientations directly from the acquired set of 2D projections. Our approach consists of two steps: (i) the estimation of distances between pairs of projections, and (ii) the recovery of the orientation of each projection from these distances. In step (i), pairwise distances are estimated by a Siamese neural network trained on synthetic cryo-EM projections from resolved bio-structures. In step (ii), orientations are recovered by minimizing the difference between the distances estimated from the projections and the distances induced by the recovered orientations. We evaluated the method on synthetic cryo-EM datasets. Current results demonstrate that orientations can be accurately recovered from projections that are shifted and corrupted with a high level of noise. The accuracy of the recovery depends on the accuracy of the distance estimator. While not yet deployed in a real experimental setup, the proposed method offers a novel learning-based take on orientation recovery in SPA. Our code is available at https://github.com/anonymous/protein-reconstruction.

Learning to recover orientations from projections in single-particle cryo-EM
Anonymous Author(s)
Affiliation
Address
email
Abstract
A major challenge in single-particle cryo-electron microscopy (cryo-EM) is that the orientations adopted by the 3D particles prior to imaging are unknown; yet, this knowledge is essential for high-resolution reconstruction. We present method to recover these orientations directly from the acquired set of 2D projections. Our approach consists of two steps: (i) the estimation of distances between pairs of projections, and (ii) the recovery of the orientation of each projection from these distances. In step (i), pairwise distances are estimated by Siamese neural network trained on synthetic cryo-EM projections from resolved bio-structures. In step (ii), orientations are recovered by minimizing the difference between the distances estimated from the projections and the distances induced by the recovered orientations. We evaluated the method synthetic cryo-EM datasets. Current results demonstrate that orientations can be accurately recovered from projections that shifted and corrupted with high level of noise. The accuracy of the recovery depends on the accuracy of the distance estimator. While not yet deployed in real experimental setup, the proposed method offers novel learning- based take on orientation recovery in SPA. Our code is available at https: //github. com/
6
10
12
14
16
17
Introduction
Single-particle cryo-electron microscopy (cryo-EM) has revolutionized the field of structural biology over the last decades [1, 2, 3]. The use of electron beams to image ice-embedded samples has 21 permitted the recovery of 3D bio-structures at unprecedented resolution. This "resolution revolution" has had tremendous impact in biomedical research, providing invaluable insights into the biological 23 processes that underlie many current diseases.
19
24 In single-particle cryo-EM, every 3D particle adopts random orientation 0; in the ice layer before
25 being imaged. Hence, the projection geometry associated each acquired 2D projection (Figure 26 is unknown. Yet, this knowledge is essential for the tomographic reconstruction of bio-structures [4]. consider that cryo-EM measurement (i.e., projection) P: E "p acquired through
27
pj Pe,
where x E Re is the unknown 3D density map [5] (Coulomb potential). The operator Po, RR 29 R"P is the projection along the orientation 0, (i.e., the x-ray transform). The operator S. Rp + R"> is a shift of the projection by = (t,, ) The convolution operator C.p F + 1 models 31 the microscope point-spread function (PSF) with parameters &p = (d1, d2, Qast)) where dj is the defocus-major, d2 is the defocus-minor, and Qast is the angle of astigmatism [6, 7]. Finally, E R 39 represents additive noise. Figure illustrates the effect of projection, shift, and noise. The challenge then to reconstruct from set of projections {p.).- acquired along unknown orientations.
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

(Pr.))
(P).
dg ①
Protein
Projecton
d,49. m
dd 9
(pL:
Figure Geometry of the imaging model de- fined in (I). The 3D density x in the coordinate system (21, 22, 233) is imaged along the orienta- tion O to produce the 2D projection p in the co- ordinate system (y1, 1/22 of the microscope's de- tector plane. The orientation 0 = (0g. 02 01) is decomposed as the direction (02,01. E [0, n x [0, 2T[ (paramcterizing the sphere $²) and the in-plane rotation 03- E [0, 2T| (parameterizing the circle 8)). In work, represent the orientation 0 as a unit quaternion q.
Figure Single-particle cryo-EM produces P projections (with P in the order of 10") from unknown orientations: {(P,, 4s 1 Observing that distances between orientations constrain the latter, we aim to recover the orientations {q} from {dg (gi, qj)}, where (g 15) is the dis- tance (angle) between orientations 4 and 95. Ob- serving that the similarity between projections depends on their relative orientation, we aim to estimate the distance dg(9s, 9j) from the projec- tions (P:.p))
35 popular approach alternatively refine the 3D structure and estimated orientations [8, 9, 10, 11, 36 12, 131. Yet, the outcome of these iterativerrefnnmmen procedures is often predicated on the quality of the initial reconstruction, or, equivalently, on the initial estimation of the orientations [14, 15].
37
38 Several methods have been designed to produce a first rough ab initio structure for the refinement
39 procedure [16]. Moment-matching techniques [17, 18, 19, 20] reconstruct an initial structure such that the first few moments of the distribution of its theoretical measurements match the ones of its experimental projections; however, they typically remain sensitive to error in data and can require relatively high computational complexity. Based on the central-slice theorem, common-lines methods [21, 8, 22, 23, 24. 25, 26] aim at uniquely determining the orientations of each projection by identifying the common-ling lines between triplets of projections- real challenge given the massive amount of noise. Alternatively, the marginalized maximum likelihood (ML) formulation of the reconstruction problem clasccassically used for the perrativerefinemenn procedures themselves. can be minimized using stochastic gradient descent [27]. This permits avoid the need for an initial volume estimate, at the possible cost of greater convergence instability.
40
41
42
43
44
45
46
47
48
More recently, the recovery of geometrical information from unknown view tomography of 2D point sources has been proposed [28], but the extension to 3D cryo-EM tomography is not straightforward. Finally, [29] proposed recover the in-plane rotations by learning embed projections in
50
51
52
appropriate latent space. but only after directions had been estimated through three rounds of 2D classification in RELION.
53
54 Despite the aforementioned advances, providing a robust initial volume remains a challenge due to the high-dimensionality and ill-posedness of the underlying optimization problem. On the other hand, the remarkable ability convolutional neural networks to capture relevant representations of images has had profound influence in imaging [30]. In this work, present leariin-basse approach to 58 recover the unknown orientations directly from the acquired set of projections-withee the need for an intermediate reconstruction procedure an initial volume estimate.
56
59
Method
60
61
Our approach relies on two observations (Figure 2), yielding two steps (Figure 3). First, the more similar two projections (p:, P,). the more likely they originated from two particles that adopted close

---

Inputs:: Aoquired Projection Decaset
Step 1: Estimate Distances
Intermediate Representation
(p..))
(P-
Pp
Siamese Neural Network (troined synthetie disiey
10-
Step
Orinnttions
Internediate Recresentation
Outputs: Recovered Orientarions
4,10.
$
Up
(..PP) d,99 q21²
19.
Our method consists of two steps. First, we estimate distances between pairs of projections. Second, we recover the orientation of each projection from these distances.
Figure 3:
63 orientations (gi: qj) in the ice prior to imaging:' this observation guides a number of applications in
64 the field [2]. Hence, we aim to estimate distances between orientations dg ((g 9i) from the projections
as dz (P:: Pj), which we discuss in $2.2. Second, an orientation g is constrained by the distances between itself and the other orientations {d(q, 9,)}. Hence, we aim to recover orientations {9k) such
65
68
that the induced distances {dy (Te, } are close the estimated distances (p Pý)}. which 68 discuss in $2.3. All in all, from set of projections } we aim to recover their orientations {Φk} such that dg (0: ()) (P., pÿ) Ra (g (());))), with equality if d, and {Øe} are perfectly estimated.
67
70 Our approach is similar to [31]. While the authors reconstruct 2D images from 1D projections, they
71 rely on the same two-step approach: they (i) estimate distances as de (P: P2) Ip: py l2 then (ii) recover the orientations by spectrally embedding that distance graph. The Euclidean distance is however not robust to perturbations: for example, two projections that only differ by shift S. of one pixel would be considered far apart while their orientations the same. They noted that issue and we observed it too (Appendix E). To circumvent this, we propose learn dp from examples ($2.2).
73
75
2.1
Representation of orientations with quaternions
77
The orientation of 3D particle with respect to the microscope's detector plane is a rotation relative to reference orientation (Figure 1). The group of all 3D rotations under composition is identified with (3(3), the group of 3 x 3 orthogonal matrices with determinant under matrix multiplication. A rotation matrix Rg E (3(3) can be decomposed as a product of (3) = independent rotations, for
79
80
81
(0g, 02 01) E (0, 2 x (0, π x [0, 2x[ are the (extrinsic and proper) Euler angles in the ZY Z convention common parameterization in cryo-EM) [32].
example as R. Rog Ro, Ro, where
83
While Euler angles are a concise representation of orientation numbers for degrees of freedom), they suffer from topological constraint- is no covering map from the 3-torus to SO(3) which manifests itself in the gimbal lock, the loss of one degree freedom when 02 0. This makes their optimization by gradient descent ($2.3) problematic. On the other hand, optimizing rotation matrices (made of numbers) would require computationally costly constraints (orthogonality and determinant 1) to reduce the degrees of freedom to 3. Moreover, the distance between orientations cannot directly computed from Fuler angles and costly (30 multiplications) to compute from rotation matrices [33]. We solve both problems by representing orientations with unit quaternions.
84
85
86
87
89
90
91 Quaternions q Hi are an extension of complex numbers of the form q = a bi + cj + dk where 82 a, b, d E IR. Unit quaternions E S3, where S {q E Hi lg| 1} is the 3-sphere (with 93 the additional group structure inherited from quaternion multiplication), concisely and elegantly sin 2) + sin 2/2)]=
94 represent rotation of angle 0 about axis (æ1, Z2, 23) as q
95 I3 (0/ They parameterize rotation matrices as
aa2
6² c²
df
2bc 2ad
2bd + 2ac
2bc
2ad
a 1/2 + c d2
2nd 2ab
2bd 2ac
2cd + 2ab
a bo ²² d2
Up to protein symmetries, which we discuss later. "The algebra H is similar to the algebra of complex numbers C, with the exception of multiplication being non-commutative.
3

---

Training Dalasel
Stamese Neural Network as dp
Pr
& (.,)
Fealuree Desenne d/pp.
Sister CNN #1
Functon
d/P..p)
INVono Skarco Hosgens
Loss
Pj
T (P))))
[p op qp
Sister CNN #2
d.
9
4,19 9)
Figure 4: Distance learning. We are looking for distance d, between projections that is accurate estimator the distance dg between their orientations. We propose to parameterize dp as a Siamese neural network (SNN), trained on synthetic dataset of projections with associated orientation.
Note that $ (3(3) is two-to-one mapping (a double cover) and -q represent the same 57 orientation. Unlike Euler angles, @ is isomorphic to the universal cover of SO(3). Hence, distance between two orientations, i.e., the length of the geodesic between them SO(3), is
: S
[0, m|:
(, (3) 2 arccos (Kgs 1)),
99 where (: is the inner product, and the absolute value | ensures that (g 13) ) dg(gs. -0- -;). The distance dg (4: qj) corresponds the magnitude of the rotation R. such that Rg, R. R, [33].
100
101
Distance learning
We aim to estimate function &, such that (p., Pj) so dg(4.. 9j). While we could principle design dp, that would be intricate-if not impossible- partly because the invariants are difficult to specify. We instead opt to learn dy. capitalizing on (i) the powerful function approximation capabilities of neural networks, and (ii) the possibility to generate realistic datasets supported by the availability numerous 3D atomic models' and our ability to model the cryo-EM imaging procedure.
102
104
105
106
107
From a training dataset
we learn the projection distance
å
arg min LDE, where LDE
[dd (4(
(3)
dp
108
is the loss and dy is defined in (2). The dp is parameterized the Siamese neural network (SNN) [34]
(p. (p) d, (p.)
109 where Go, a convolutional neural network with weights w that is trained to extract the most relevant features TR" from projection p.. SNNs, also termed "twin networks". are commonly used in 111 the field of deep metric learning to learn similarity functions [35]. We set the feature space distance
112 d as the cosine distance to facilitate the learning of dp that respects the elliptic geometry of $3 (Appendix F). Figure illustrates the proposed learning paradigm.
113
114 As evaluating a sum over p2 pairs is computationally intractable for cryo-EM datasets with typically sample the sum and minimize (3) with stochastic gradient
P in the order of 10" projections, descent (SGD) over small batches of pairs. The weights to are updated by back-propagation.
116
The architecture of G., is described in Appendix G. When designing the architecture, constrain the functional space from which the trained G. is drawn and express our prior expert knowledge. For example, we realize shift invariance, i.e., guarantee that shift St does not change our estimated distances and orientations, with a fully convolutional architecture. Size invariance, i.e., taking projections of varying sizes Np while yielding representation f of fixed size n is realized by final average pooling layer. As we do not (yet) know how to realize an invariance to noise or PSE we resort to data augmentation, i.e., training perturbed projections. In $3.4, show that built-in invariance (shift) is far preferable to one learned through augmentation (noise). Finally, as projections
118
120
122
124
https://wwe ebi. ac. uk/ htk/pdbe/emdl

---

125
are made by integrating through the 3D volume, projections from opposed directions are mirrors of each other.* That is another kind of physical knowledge that should ideally be built into our method.
126
127
One could hope to train Gue to directly map projections to orientations as T = f, = Gue (p:). While that would avoid the orientation recovery step, space of ny 4 dimensions does not have room for I to represent the other factors of variation in p, such as different noise levels, PSFs, or proteins.
129
130
We tested that hypothesis in Appendix F.
2.3
Orientation recovery
132
The task of recovering points based on their relative distances has been extensively studied. Many methods aim mapping high-dimensional data onto lower-dimensional space while preserving distances. primarily for dimensionality reduction and data visualization. Well-known examples
133
134
135
include MDS [36], Isomap [37], LLE [38], Laplacian eigenmaps [39], t-SNE [40], and UMAP [41].
136
The embedding of distance matrices in Euclidean space (given by their eigenvectors) is especially dese-des. In particular, the framework of Euclidean distance matrices (HDMs) [42] provides theoretical guarantees on the recovery of points from distances.
137
We however aim to embed the orientations q in S ($2.1), setting for which we are unaware of any theoretical characterization (e.g., on the shape of the loss function or its behavior when distances are missing or noisy). The fact that $ is locally Euclidean does however offer some hope. Indeed, despite the convexiy and the lack of theoretical guarantees, we are able to appropriately minimize our loss function, as we experimentally demonstrate in Appendix D.
139
140
141
142
143
We recover the orientations of a set of projections (px} ) through
144
arg min LOR:
where
LoR =
Σ 1.
(p..p)) -d (qi ())
(4)
k==
[nu EB9)
145 is the loss and de is the estimator trained in (3). Note that the sole difference with (3) is that the minimization is performed over the orientations q rather than the distance dy. Here again, we sample the sum in practice and minimize (4) with mini-batch SGD. Sampling the sum amounts to building a sparse (instead of complete) distance graph before embedding, common strategy.
148
147
148
2.4 Evaluation
150
While not a part of the method per se, we must evaluate the quality of the recovered orientations. Unfortunately, we cannot directly take the difference between the recovered orientations {9x ==
151
152
and the true orientations {qx as orientations are rotations up an arbitrary reference orientation.
153
Any global rotation or reflection of the recovered orientations is as valid as any other, i.e., dg g) =
154
d Tq; Tq,) VT O(4), where (44 the group of 4 orthogonal matrices. Hence, we align
155
the sets of orientations and compute the mean orientation recovery error as
P
EOR
min = TEO(1)
1
Σ
dd (q. |.
(5)
P
We implement T product of (1) independent rotations and optional reflection:
T
m 0
Teg m m 0g [0,2
(6)
ISicjE4
157 where To,, E SO(4) is rotation by angle O-y on the (ri, (ty) plane.
158 In practice, we again minimize (5) with mini-batch SGD. Because O(4) is disconnected, we optimize
159
the 6 angles separately for m = 1 (proper rotations) and m = -1 (improper rotations). Figure 15
160 shows an alignment to EOR = 0 after a perfect recovery.
'Tht fact prevents the resolution of chirality, i.e., cannot distinguish protein from its mirrored form.
5

---

161 3
Experiments
162
We first evaluated whether orientation recovery through (4) was feasible assuming perfect distances, and how it was affected by errors in the distances ($3.2). We then learned to estimate the distances through (3), and evaluated the accuracy this procedure ($3.3) and its robustness to perturbations of the projections ($3.4). Finally, we the whole machinery synthetic dataset to assess how well orientations could be recovered from estimated distances ($3.5).
164
186
187
3.1
Experimental conditions
188
Density maps. We considered two proteins (Figure 10): the 3-galactosidase, a protein with a dihedral (D2) symmetry, and the lambda excision HJ intermediate (HJI), an asymmetric protein with local cyclic (C1) symmetry. Their deposited PDB atomic models are 5a1a [43] and 5j0n [44], respectively. From these atomic models, we generated the density maps in Chimera [45] by fitting the models with 1Å map for 5ala and a 3.67A map for 5j0n; this gave us volume of 110 x 155 x 199 voxels for 5a1a a and one of 69 x 57 75 voxels for 5jOn.
189
170
172
173
Protein symmetries. Symmetries are problematic when learning distances: two projections can be identical while not originating from the same orientation, which breaks an axiom of distance functions (identity of indiscemibles). Figure 16b illustrates this problem. To capture only identical projections of 5a1a, we restricted directions to (02, 01) E [0, æ x [0, z (a quarter of the sphere, illustrated in Figure 12a) for that protein. This treatment of symmetries is incomplete" but sufficient for proof-of-concep..
175
of four
177
2
178
179
Projections. Using the ASTRA projector [46], we generated P 5, 000 synthetic projections of 275 x 275 pixels (downsampled to 116 x 116) for 5ala and 116 x 116 pixels for 5jOn, taken from uniformly sampled orientations." We then perturbed the measurements with different levels of additive Gaussian noise [47, 48] and off-centering shifts. Figure 11 displays some samples.
181
183
184
For each protein, we split the projections into training, validation, and test subsets, and created disjoint pairs of projections from each (Table 1). The training and validation sets were used to train and evaluate the SNN, while the test set was used to evaluate orientation recovery given a trained SNN. Sampling orientations (mostly) uniformly induces a distribution of distances that is skewed towards larger distances (shown in Figure 12b). As this would skew LDE and bias dy. we further sampled 1% of the training and validation pairs to make the distribution of distances uniform for
Datasets.
185
186
189
d to be uniformly accurate over the whole [0, T] range of distances (see Appendix B for further illustrations). While 1, 650 projections were enough to perfectly reconstruct the density maps (as shown in Figures 9e and 9j), our method is not limited by the number of projections as optimization done per batch. Optimization settings are described in Appendix C.
190
191
192
193
194
3.2
Sensitivity of orientation recovery to errors in distance estimation
195
We first evaluated the feasibility of orientation recovery assuming that the exact distances were known. The method successfully recovers the orientation of every projection in this case (see Appendix D).
196
To evaluate the robustness of (4), we perturbed the distances prior recovery with error sampled from a Gaussian distribution with mean 0 and variances o² [0.0, 0.8). Figure shows that the recovery error EOR is a monotonic function of the error in distances: from EOR 0 with exact distances to EOR ss 0.2 radians (== 11.5%) for o² = 0.8.
198
200
201
These results demonstrate that the performance of orientation recovery (4) depends on the quality of the estimated distances, which advocates for proper and extensive training of the SNN. Moreover, we observe that FOR is reliable proxy for FOR, allowing to assess recovery performance in the absence of ground-truth orientations (i.e., when recovering the orientations of real projections).
202
204
'The remaining issue is that of four distances arbitrarily chosen per pair of projections.
'Orientations used in $3.2 (Figure 5) and $3.4 (Figure 7) were actually obtained by uniformly sampling the Euler angles O constrained to (0g, O2. 01) E [0, 2n x [0, ][[ [0, 2 for 5jOn. Our conclusions would be identical if orientations were uniformly sampled from SO(3) (3) instead.

---

0R:
Thikingedd
Truiking
Table 1: Split of P 5, 000 projections in train- ing, validation, and test subsets.
0.4
0.4
0.215
0,272
0:159
Dataset
P
p2
Used pairs
%%
92
Epach
320
Epoch
120
Training
2,512 (50%)
6,310,144
63,101
Validation
838 (17%)
702,244
7,022
(a) Loss converged on 5j0n (left) and 5a1a (right).
Test
1,650 (33%)
2,722,500
2,722,500
projettion par
prgeciic par
0.16-
0.16
0.50
0.5
Kop 0.211
1ce 0.378
0.08
0.08
d,lq. q))
dg1q, 92)
0.00
0.00
3.00
0.0
622
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.6
(b) Relationship between dp and dg on 1, 000 pairs from the test sets 5j0n (left) and Sa1a (right).
Figure 5: Orientation recovery from perturbed distances jj (left) and 5a1a (right).
Figure 6: Distance learning.
3.3
Learning to estimate distances
206
We evaluated the ability of the SNN to learn approximate the orientation distance dg. For comparison, we evaluated a baseline, the Euclidean distance dp(p:. P,,) = =l|P., /|Pp..,/l2. in Appendix E.
207
Figure 6a shows the convergence of LDE, reached in about 50 epochs. Figure 6b shows the relationship between the distance de estimated from projections and the true distance dy. The outliers for 5a1a are explained by our incomplete treatment of its symmetry. While our learned distance function a much better estimator than the Euclidean distance- compare Figure 6b with Figure 16 16-they share one characteristic: both plateau and underestimate the largest distances. We did attenuate the phenomenon by sampling training distances uniformly (see $3.1), and the issue is much less severe than with the Euclidean distance. An alternative could be to only rely on smaller distances for recovery. That would however require the addition a spreading term (4) to prevent the recovered orientations to collapse.
209
210
211
212
213
214
215
217
These results confirm that SNN is able to estimate differences in orientations from projections alone, even though much has yet to be gained from improving upon the rather primitive SNN architecture we are currently using. The use of additional training data should help further diminish overfitting.
218
219
3.4
Sensitivity of distance learning to perturbations in the projections
221
We first demonstrated that the learning of distances is insensible to off-centering shifts (Figure 7a), which is expected given that shift invariance is built in our SNN (see $2.2).
223
As cannot-or do not yet know how to to-build noise invariance in the SNN architecture, we trained the SNN on noisy projections and evaluated whether could learn to treat noise as an irrelevant information. Figure 7b shows EOR z 0.16 radians (+% 99) for noiseless projections and EOR 0.42 radians (+= 249) for a more realistic noise variance of o² = 16 (with signal-to-noise ratio of -12 dB). Whereas a naive distance function (e.g., an Euclidean distance) would be extremely sensitive noise,
224
225
226
227
228
the SNN mostly learned to discard it. Moreover, the observed overfitting indicates that more training data should further decrease the sensitivity of the SNN to noise.
229
230
Note that we did not evaluate sensitivity to the PSF at this stage but expect a similar behavior.
Here again ($3.2), we observed that (i) the estimation of more accurate distances (a smaller IDE) leads to the recovery more accurate orientations (a smaller ToR and FOR), and that (ii) an higher recovery loss LOR induces an higher error EOR.
232
7

---

valdeooue
10
15
20
Projection shift limit [px]
Projection noise variance of
(a) Learning from shifted projections {St, P).x), with shifts bi and lig sampled from triangular distribution with mean O and of increasing limits.
(b) Learning from noisy projections (Po, n). with white noise n ~ (10, TT) I) of increasing vari- ance o².
Figure 7: Sensitivity of distance learning to perturbations the projections of 5jOn. The box plots show the distance learning loss LDE (the distribution is taken over epochs). Boxes show the orientation recovery loss LOR and error EOR-
Rou.r 5se 3.1993
-0278 Ext
Apcival
0.3254720
03137
Foncuw 0:358
00:55
0:333
3:312
LOx
LdR
Lox
0
6320
12030
6103
12005
1003
CO03
$000 Steps
16000
Steps
Steps
Steps
(a) jjOn, noiseless.
(h) 5jOn, noisy.
(c) 5ala, noiseless.
(d) Sala, noisy.
The green and orange boxes show DE (3) on the training and validation sets. The blue curve shows the evolution of the recovery loss until convergence, with the minimum LoR (4) highlighted. The red histogram shows the errors in the recovered orientations (dy(g, TT)), with the mean EOR (5) highlighted.
Figure 8: Distance learning and orientation recovery from estimated distances.
234
3.5
Orientation recovery and reconstruction density maps
235
As proof-of-concept, attempted solve the full inverse problem posed by (1), reconstruct the density maps X from sets of projections {p:} and their orientations { recovered through the proposed method. It is worth noting that, this stage of development, we only trained the SNN on projections originating from the protein we were attempting to reconstruct. In addition, reconstruction was performed with direct reconstruction algorithm (ASTRA's GPU implementation of the CGLS algorithm) rather than with robuster iterative method. This is specific experimental case that only partially shines light on the applicability of the method in real situations; this discussed in $4.
236
237
238
239
240
241
242
Figure 8a shows the recovery of orientations from distances that were estimated from noiseless projections of 5jOn. A mean error of EOR z 0.20 radians (== 11") in the recovered orientations led to reconstruction with resolution of 12.2Å at Fourier shell coefficient (FSC) of 0.5, shown in Figure 9c. As predicted by our other experiments, corrupting the projections with noise (o2 = 16) negatively impacts the quality of the recovered orientations (Figure 8b); the obtained mean error then FOR 2 0.25 radians ( 149). Unsurprisingly, this leads to a reconstruction with a lower resolution of 15.2Å, shown in Figure 9d. (Note that reconstruction was here obtained from the noiseless projections, the goal being to evaluate only the impact of orientation mis mistestimation.)
245
247
248
250
Finally, Figures 8c,d show the recovery of orientations from noiseless and noisy projections of 5a1a. A mean error of FOR R 0.13 radians ( 7") in both cases led to reconstructions with resolutions of 8.0 and 9.6À shown in Figures 9h,i. Distance estimation, orientation recovery, and reconstruction performed better on 5a1a than 5jOn because its ground-truth density is of higher resolution.
252
254
These results tend to indicate that reasonable first structure can be reconstructed from projections whose orientations have been recovered through our method.
255
Discussion
256
257
In this work, we explored the use of distance learning between pairs of 2D cryo-EM projections from 3D protein structure to infer the unknown orientation at which each projection imaged

---

0.5
0.143 @ 7.64
0.007
0,082
(18
19.11
Resolution [4
(a) (b) (c)
(d)
(c)
196
0.343
4.24
(30.0)
53.11
(5.31
22.51
G2.01
Resolution [A
(f)
(g)
(h)
(
())
Figure 9: Density maps & reconstructed from (a.f) ground-truth orientations, (b.g) random orientations, (c,h) orientations recovered from noiseless projections, and (d,i) orientations recovered from noisy projections. The Fourier shell correlation (FSC) curves (e.j) indicate the resolutions of the densities (w.r.t. ground-truth densities, shown in Figures 10b,d).
259
from. Our two-step method relies on the estimation of pairwise distances between unseen projections, followed by the recovery of the orientations from these distances.
260
261
The method has been evaluated on synthetic datasets for two different proteins. The results provide key insights the viability of the proposed scheme. First, they demonstrate that SNN can learn a distance function between projections that estimates the difference in their orientation ($3.3) and that is invariant shifts and robust increasing levels of noise ($3.4) important condition in cryo-EM. Second, they demonstrate that an accurate estimation of distances leads to an accurate recovery of orientations ($3.2, $3.4). Finally, our method was able to recover orientations with an error of 0.12 to 0.25 radians (7 to 149)-icading to an initial volume with a resolution of 8 to 15A ($3.5). In summary, the more accurate the estimated distances, the more precise the recovered orientations, and, ultimately, the higher-resolution the reconstructed volume.
282
263
285
286
287
288
269
270
While the method is not yet ready to be deployed in practice, we believe that a series of developments could make it relevant for single-particle cryo-EM reconstruction." As previously discussed, the results underline the importance of learning an accurate distance estimator. performance of the SNN could be improved. First, the architecture of the twin convolutional neural networks should be expanded and tuned. Second, training could be improved, perhaps by providing more supervision by separately predicting the differences in direction (02,0) and in-plane angle 0g-
272
this regard, the
274
276
Importantly, the SNN would be better trained on more diverse cryo-EM dataset. Indeed, its success as faithful estimator eventually relies on capacity to generate synthetic training dataset whose data distribution is diverse enough to cover that of unseen projection datasets. Such realistic cryo-EM projections could be generated by relying on more expressive formulation of the cryo-Eb EM physics and taking advantage of the thousands of atomic models available in the PDB. In particular, a necessary extension will be to include the effects of the PSF and to evaluate its impact.
278
280
282
final phase of tests before deploying the method on real cryo-EM measurements will be to extensively test the method on "unseen proteins", i.e., proteins whose simulated projections have never been seen by the SNN. In this regard, an interesting aspect of our method that the twin networks within the SNN intrinsically predict the relationship between projections, allowing the SNN as a whole to abstract the particular volume. Learning should benefit from the profound structural similarity shared by proteins proteins-after after all, they are all derived from the same 21 building blocks.
284
286
288
Training our 4.5M parameter model (see Appendices and C) has the following negative environ- mental impact: it consumes 13 kWh of energy, which produces 6.36 lbs of CO2 on average [49].
NNte that the present project will not be further continued by its authors due to other professional occupations. Hence, we strongly encourage anyone interested to build on these ideas and, hopefully, make it practical tool.

---

References
290
[1] J. Dubochet, M. Adrian, J.-J. Chang, J.-C. Homo, J. Lepault, A. W. McDowall, and P. Schultz, "Cryo-electron microscopy of vitrified specimens." Quarterly Reviews of Biophysics, vol. 21, no. 2. pp. 129-228, 1988.
292
294
[2] J. Frank, Tiree-dimensoonal electron microscopy of macromolecular assemblies: Visualization of biological molecules in their native state. Oxford University Press, 2006.
295
296
[3] Nature, "Method of the year 2015," Nature Methods, vol. 13, no. 1, 2016.
297
[4] F. Natterer, The mathematics of computerized tomography. Society for Industrial and Applicd
Mathematics, jan 2001.
289
[5] F. DiMaio, D. A. Kondrashov, E. Bitto, A. Soni, C. A. Bingman, G. N. Phillips, and
300
J. W. Shavlik, "Creating protein models from electron-density maps using particle-fitterng methods," Bioinformatics, vol. 23, no. 21, pp. 2851-2858, Nov. 2007. [Online]. Available:
301
302
303
[6] M. Vulović, R. B. Ravelli, I.. J. van Vliet, A. J. Koster, I. Lazić, U. Lücken, H. Rullgård, O Öktem, and B. Rieger. "Image formation modeling in cryo-electron mieroscopy," Journal of Structural Biology, vol. 183, no. 1, pp. 19-32, Jul. 2013.
304
305
306 [7] H. Rullgård, L.-G. Öfverstedt, S. Masich, B. Dancholt, and O. Öktem, "Simulation of transmis- sion electron microscope images of biological specimens," Journal of Microscopy, vol. 243, no. 3. pp. 234-256, 2011.
308
309
[8] A. Penczek. R. A. Grassucci, and J. Frank, "The ribosome at improved resolution: New techniques for merging and orientation refinement in 3D cryo-electron microscopy of biological particles," Utrraiccoocoppy, vol. 53, no. 3, pp. 251-270, 1994.
310
311
312
[9]
T. Baker and R. Cheng. "A model-based approach for determining orientations of biological macromolccules imaged by cryoclectron microscopy." Journal of Structural Biology, vol. I 116, no. pp. 120-130, 1996. [Online]. Available: S $1047847796900209
313
315
316
[10]
A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," Journal of the Royal Statistical Society. Series B (Methodological), vol. 39, no. 1. pp. 1-38, 1977.
317
318
319
[1I] F. J. Sigworth, "A maximum-likelihood approach to single-particle image refinement," Journal of structural biology, vol. 122, no. 3. pp. 328-339, 1998.
320
[12] S. Scheres, "A bayesian view on cryo-EM structure determination." Journal of molecular biology, vol. 415, no. 2. pp. 406-418, 2012.
322
323
[13] M. Zehni, L. Donati, E. Soubies, Z. J. Zhao, and M. Unser, "Joint Angular Refinement and Reconstruction for Single-Particle Cryo-EM," IEEE Transactions on Image Processing, 2020.
324
325
[14] C. O. S. Sorzano, R. Marabini, A. Pascual-Montano, H. Scheres, and J. M. Carazo, "Opti- mization problems in electron microscopy of single particles," Annals of Operations Research, vol. 148, no. 1, pp. 133-165, 2006.
326
327
328
115] R. Henderson, A. Sali, M. I.. Baker, B. Carragher, B. Devkota, K. H. Downing, E. H. Egelman, 7. Feng, J. Frank, N. Grigorieff, W. Jiang, S. J. Ludtke, O. Medalia, P. A. Penczek, P. B. Rosenthal, M. G. Rossmann, M. F. Schmid, G. F. Schröder, A. C. Steven, D. L. Stokes, J. D. Westbrook, W. Wriggers, H. Yang, J. Young, H. M. Berman, W. Chiu, G. J. Kleywegt, and C. L. Lawson, "Outcome of the first electron microscopy validation task force meeting," Structure, vol. 20, no. 2, pp. 205-214, 2012.
329
330
331
332
333
334
[16] A. Singer and F. J. Sigworth. "Computational methods for Lingle-partccee electron cryomi- croscopy," Annual Review of Biomedical Data Science, vol. 3, 2020.
335
336
[17] Z. Kam, "The reconstruction of structure from electron micrographs of randomly oriented
337
particles," in Electron Microscopy at Molecular Dimensions. Springer, 1980, pp. 270-277.
338
[18] D. B. Salzman, "A method of general moments for orienting 2d projections of unknown 3d objects." " Computer vision, graphics, and image processing, vol. 50, no. 2. pp. 129-156, 1990. [19] A. Goncharov, "Integral geometry and three-dimensional reconstruction of randomly oriented identical particles from their electron microphotos," Acta Applicandae Mathematica, vol. 11,
339
341
no. 3, pp. 199-211, 1988.
10

---

343
[20] N. Sharon, J. Kileel. Y. Khoo, B. Landa, and A. Singer, "Method of moments for 3-d single particle ab initio modeling with non-uniform distribution of viewing angles," Inverse Problems, 2019.
344
345
348
[21] M. Van Heel, "Angular reconstitution: a posteriori assignment of projection directions for 3d
347
reconstruction," Uttramicroscopy. vol. 21, no. 2. pp. 111-123, 1987.
348
[22] S. P. Mallick, S. Agarwal, D. J. Kriegman, S. J. Belongie, B. Carragher, and C. S. Potter, "Structure and view estimation for tomographic reconstruction: A bayesian approach." " in 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 006). vol. 2. IEEE, 2006, pp. 2253-2260.
349
350
351
352
[23] A. Singer, R. R. Coifmann F. J. Sigworth, D. W. Chester, and Y. Shkolnisky. "Detecting consistent common lines in cryo-EM by voting," " Journal of structural biology, vol. 169, no. pp. 312-322, 2010.
353
354
355
[24] L. Wang, A. Singer, and Z. Wen, "Orientation determination of cryo-EM images using least unsquared deviations," " SIAM journal on imaging sciences, vol. 6, no. 4, pp. 2450-2483, 2013.
356
[25] I. Greenberg and Y. Shkolnisky, "Common lines modeling for reference free ab-initio recon- struction in cryo-EM," Journal of structural biology, vol. 200, no. 2, pp. 106-117 2017
358
359
[26] G. Pragier and Y. Shkolnisky, "A common lines approach for ab initio modeling of cyclically symmetric molecules," Inverse Problems, vol. 35, no. 12, p. 124005, 2019.
[27] A. Punjani, J. L. Rubinstein, D. J. Fleet, and M. A. Brubaker, "cryoSPARC: Algorithms for rapid unsupervised cryo-EM structure determination," Nature Methods, vol. 14, no. 3, p. 290, 2017.
362
[28] M. Zehni, S. Huang, I. Dokmanić, and Z. Zhao, "Distance retrieval from unknown view
365
tomography of 2d point sources," Electronic Imaging, vol. 2019, no. 13, pp. 134-1, 2019.
366
[29] N. Miolane, F. Poitevin, Y.-T. Li, and S. Holmes, "Estimation of orientation and camera parameters from cryo-electron microscopy images with variational autoencoders and generative adversarial networks," " 2019.
368
389
[30] Y. LeCun, Bengio, and G. Hinton, "Deep learning," nature, vol. 521, no. 7553, pp. 436 6664444,
370
2015.
371
[3]] R. R. Coifman, Y. Shkolnisky, J. Sigwort,, and A. Singer. "Graph laplacian tomography from
372
unknown random projections." " IEEE Transactions on Image Processing, vol. 17, no. 10, pp. 1891-1899, 2008.
379
374
[32] C. Sorzano, R. Marabini, J. Vargas, J. Otón, J. Cuenca-Alba, A. Quintana, J. de la Trea-Trevín, and J. Carazo, "Interchanging geometry conventions in 3dem: mathematical context for the development of standards." " in Computational Methods for Phree-DDmensiooal Microscopy Reconstruction. New York, NY: Springer New York, 2014, pp. 7-42. [Online]. Available:
375
378
377
378
https:/ddoi.or/11.100000 1007//978- 1-6614-9521-5_2
379
[33] D. Q Huynh, "Metrics for 3D rotations: Comparison and analysis." Journal of Mathematical Imaging and Vision, vol. 35, no. 2. pp. 154-164, 2009.
380
381
[34] S. Chopra, R. Hadsell, and Y. LeCun, "Learning similarity metric discriminatively, with application to face verification," " in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), vol. 1. IEBE, 2005, pp. 539-546.
382
383
384
[35] D. Yi, Z. Lei, S. Liao, and S. Z. Li, "Deep metric learning for person re-identification." in 2014 22nd International Conference on Pattern Recognition. IEEF, 2014, pp. 34-39.
385
386
[36] M. A. Cox and T. F. Cox, "Multidimensional scaling," in Handbook of data visualization.
387
Springer, 2008, pp.315 315-347.
[37] J. B. Tenenbaum, d. Silva, and J. C. Langford, "A global geometric framework for nonlinear dimensionality reduction," " Science, vol. 290, no. 5500, pp. Available:
389
2000. [Online].
[38] S. T. Roweis and L. K. Saul, "Nonlinear dimensionality reduction by locally linear embedding." Science, vol. 290, no. 5500, pp. 2323-2326, 2000.
392
393
[39] M. Belkin and Niyogi, "Laplacian eigenmaps for dimensionality reduction and data represen- tation," " Neural computation, vol. 15, no. 6. pp. 1373-1396, 2003.
11

---

395 [40] L. Van der Maaten and G. Hinton, "Visualizing data using t-sne," Journal of machine learning research, vol. 9, no. 11, 2008.
396
397
[41] L. McInnes, Healy, and J. Melville, "Umap: Uniform manifold approximation and projection for dimension reduction," arXiv preprint arXiv: 1802..1802.08: 1802.03426, 2018.
398
389
[42] I. Dokmanic, R. Parhizkar, J. Ranieri, and M. Vetterli, "Fuclidean distance matrices: essential theory, algorithms, and applications." IEEE Signal Processing Magazine, vol. 32, 6, pp. 2015.
400
402
[43] A. Bartesaghi, A. Merk, S. Banerjee, D. Matthies, X.
Wu, J. Milne, and S. Subramaniam, "2.2 å resolution cryo-EM structure of [--gllactosidase in complex with cell-permeant inhibitor," Science, vol. 348, no. 6239, pp. 1147-1151, 2015.
404
405
[44] G. Laxmikanthan, C. Xu, A. Brilot, D. Warren, L. Steele, N. Seah, W. Tong. N. Grigorieff, A. Landy, and G. D. Van Duyne, "Structure of holliday junction complex reveals mechanisms governing a highly regulated dna transaction," " Elife, vol. 5, p. e e14313, 2016.
407
408
[45]
E. F. Pettersen, T. D. Goddard, C. C. Huang, G. S. Couch, D. M. Greenblatt, E. C. Meng, and T. E. Ferrin, "Uesf chimera-a visualization system for exploratory research and analysis," Journal of Computational Chemistry, vol. 25, no. 13, pp. 1605-1612, 2004.
410
411
[46] W. van Aarle, W. J. Palenstijn, J. De Beenhouwer, T. Altantzis, S. Bals, K. Batenburg, and
J. Sijbers, "The ASTRA toolbox: A platform for advanced algorithm development in electron tomography," Ultramccroccopy, vol. 157, pp. 35-47, 2015.
413
414
[47] C. Sorzano, L. De La Fraga, R. Clackdoyle, and Carazo, "Normalizing projection images: A study of image normalizing procedures for single particle three-dimensional electron mi- croscopy." Uttraiccosccppy, vol. 101, no. 2-4, pp. 129-138, 2004.
416
417
[48] H. Shigematsu and F. Sigworth, "Noise models and cryo-EM drift correction with direct- electron camera," Ultramicroscopy, vol. 131, pp. 61-69, 2013.
418
419
[49] E. Strubell, A. Ganesh, and A. McCallum, "Energy and policy considerations for modern deep learning research," Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 09, pp. 13 693-13 696, Apr. 2020. [Online]. Available: https://ojs.uuai.org//mn httg/inee...pp/AAAA/aatttccceeee
420
422
423
[50]
Tieleman and G. Hinton, "Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude," COURSERA: Neural networks for machine learning, vol. 4, no. 2, pp.
424
425
26-31, 2012.
[51] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv:1412.6980, 2014.
427
428
[52] H. B. McMahan, D. Golovin, S. Chikkerur, D. Liu, M. Wattenberg, A. M. Hrafnkelsson, T. Boulos, J. Kubica, G. Holt, D. Sculley, M. Young, D. Ebner, J. Grady, L. Nie, T. Phillips, and E. Davydov, "Ad click prediction: a view from the trenches," in Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining KDD '13. ACM Press, 2013, p. 1222. [Online]. Available: http://sc.ooooggiioooootooooeees
430
431
432
433
Checklist
434
1. For all authors...
438
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
437
contributions and scope? [Yes]
438
(b) Did you describe the limitations of your work? [Yes] See $4.
439
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We didn't t identify any potential risk for improving protein imaging. Moreover, our work only addresses small step in a huge pipeline. We however mentioned the environmental impact of training our model (see $4).
410
442
443
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
12

---

448
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof's of all theoretical results? [N/A]
448
3. If you ran experiments..
449
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL.? [Yes] We include a URL in the Abstract to a git repository that includes code, data, and instructions to reproduce our results. Moreover, notebooks and an interactive website are provided to further play with the method.
450
451
452
453
454
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See $3.1 (including Table 1) for the preparation of data and how they were split. See Appendix C for the hyperparameters.
455
456
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [Yes] When there was variance, e.g. on Figure 7 and Figure 17b.
458
459
460
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix C.
462
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
463
(a) If your work uses existing assets, did you cite the creators? [Yes] We used proteins from the publicly available Protein Data Bank (PDB) and cited the ones we used, see
465
$3.1. We also used and cited the ASTRA toolbox in the same section.
466
(b) Did you mention the license of the assets? [Yes] The license of our code is mentioned in the README. md and included in a LICENSE. txt file in our git repository. PDB data are free of all copyright restrictions and made fully and freely available for both non-commercial and commercial use.
488
469
470
(c) Did you include any new assets either the supplemental material or as URL? [Yes] As URL in the Abstract.
(d) Did you discuss whether and how consent was obtained from people whose data you'ne rre'r using/curating? [N/A] Our data are proteins.
473
474
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] Our data are proteins.
475
476
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
478
479
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
480
481
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
482
13


Reviews:
[{"review_id": "uMgJOYKP_k5", "reviewer": "Reviewer_Z4BJ", "summary": "In this paper, “Learning to recover orientations from projections in single-particle cryo-EM,” the authors present an approach to recovering unknown projection orientations based on learning to predict the distance between projection angles given observed projections. Given these predicted projection angle distances, it is possible to recover the orientations themselves by solving for the orientations with distances that best match the predicted distances. The distances themselves are predicted using a Siamese neural network that is trained on projection pairs with known orientation distances. The authors demonstrate this approach on two synthetic datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although the method has some clear limitations and is only demonstrated on synthetic datasets, this paper presents a compelling and well described method and proof-of-concept demonstration. The paper is well written, and the problem and method are well motivated with relevant background. Specific comments follow below.\n1.\tThe big question: how will this work in practice on real cryoEM datasets where orientations are not known? Even training this on real cryoEM datasets is potentially problematic because the orientations of the particles were estimated to produce the reconstruction. Will a network be able to generalize over particles from different structures?\n2.\tWhat happens when the particles are not derived from single rigid structure (i.e., there is conformational heterogeneity)?\n3.\tWhen minimizing (4) to solve for the orientations from the distances, how are the orientations initialized? Similarly to t-SNE and other distance-based embedding methods, it seems like the objective is non-convex and so orientations may be mis-estimated due to poor initialization. How quickly does the minimization converge?\n4.\tIt would be useful to see a comparison with orientations estimated using a common lines approach (e.g., http://spr.math.princeton.edu/orientation). It would also be useful to see the orientations estimated by an iterative reconstruction procedure as baselines to better understand how well the orientations in these datasets can be estimated by other approaches.\nOverall, I think there are some nice ideas here that are well motivated and described. The main weaknesses are the question about how this will apply to real cryoEM data and the lack of strong baselines for orientation estimation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact are sufficiently addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "ssW9JQNeeNw", "reviewer": "Reviewer_zuH5", "summary": "The authors tackle the problem of image orientation estimation in single particle cryo-electron microscopy. This imaging technique generates a dataset of ~10^{5-7} 2D projection images of a 3D protein with unknown pose (elements of SO(3)xR2). Once poses are inferred, the 3D structure may be reconstructed with standard tomographic projection techniques. To estimate the distances, the authors propose a Siamese convolutional network architecture, which takes as input a pair of images, and attempts to predict the distance between their orientations (parameterized as quaternions). Once this distance function is learned (from a training set of posed images on a single protein), gradient-based optimization is used to infer each image’s pose. The authors show results on two small, synthetic datasets of 5000 images.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I appreciate the detailed study performed by the authors, however I do not think this paper is suitable for publication at NeurIPS as it focuses on a particular application without introducing any technical novelty, and the task is somewhat contrived for the application domain. \n\nThe task of ab initio reconstruction for homogeneous proteins is well-studied/solved. The authors should benchmark against a state of the art algorithm. \n\nFundamentally, the method relies on a training set of previously posed images, so the task they are addressing is not a realistic setting. As the authors discuss, if this approach were extended to be trained across multiple datasets and could predict orientations for a new (real) dataset zero-shot, it would be very high impact; however I am not convinced from the current results that extending this approach is promising. The paper lacks theoretical grounding, which the authors admit, and I am not convinced that there are any generalization properties for functions on distances between projection images (to unseen proteins).\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "aR72g63J6UY", "reviewer": "Reviewer_9yDH", "summary": "This paper uses deep learning to recover the orientations of molecules measured in the context of single particle cyroEM. The proposed method first estimates the pairwise distances between the angles of molecules based on the their projections. This is accomplished by embedding the projections into a feature space that was trained such that the cosine similarity between two features is equal to the distance between the orientations of the two molecules. Next, the authors solved an optimization problem (3) to recover the orientations, up to global ambiguities, from the pairwise distances.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n# Originality\nRecovering cryoEM orientations or structure with DL is not new, but the previous works I'm familiar with have mostly focused on solving the task with dimensionality reduction. I haven't seen a paper tackle the problem by learning to estimate the pairwise distances between angles.\n\n# Quality\nThe paper did a good job of validating each proposed contribution in isolation. Simulations results were overall thorough, though there was no real experimental data.\n\n# Clarity\nWell-written\n\n# Significance\nIt's not evident the proposed method will work with real data and the method was trained on the same molecules whose orientations it was going to reconstruct.\n\nIt's also unclear if the proposed methods improvements will actually improve the final quality of cryoEM reconstructions. This tasks performed in this paper are essentially preprocessing steps that recover an initialization that is fed into another algorithm (e.g., expectation maximization and RELION) that would reconstruct the structure. It's unclear if the improvements in the init help the end result.\n\n# Related work\nRecent work in DL for single particle cryoEM has attempted to recover the 3D structure without explicitly recovering the angles using adversarial networks [A]. It may be worth mentioning.\n[A] Gupta, Harshit, et al. \"Cryogan: A new reconstruction paradigm for single-particle cryo-em via deep adversarial learning.\" BioRxiv (2020).\n\n# Questions\nIt's my understanding that some molecules will preferentially align in certain directions, so that the view angles in cyroEM are not uniformly distributed. This is a major issue for embedding-based methods. How would non-uniformly distributed angles effect the proposed method?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors are transparent about the limitations of current method. Single particle cryoEM is an important problem.\n", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "FhZdxG2jZYX", "reviewer": "Reviewer_RNam", "summary": "This work presents a method for estimating the viewing directions of projection images in single-particle cryo-electron microscopy (cryo-EM). The method consists of two steps, first the distance between the viewing angles are estimated using a neural network, then these distance estimates are combined to yield a viewing angle estimate for each projection image. The method is trained and evaluated on synthetic projection images, where it is shown to perform well in producing reconstruction distance estimates are combined to yield a viewing angle estimate for each projection image. The method is trained and evaluated on synthetic projection images, where it is shown to perform well in producing reconstructions of moderate resolution.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents an interesting method for ab initio reconstruction in single-particle cryo-EM, a difficult problem that stands to benefit from data-driven approaches due to the high noise content of the data, which is otherwise quite structured. The proposed method is promising, but numerical validation in the present work is lacking, especially with respect to comparisons with existing methods or experimental data. Instead, the performance of the method is evaluated in a synthetic-data setting without comparison to other methods and with a network that is trained on the same molecule. It is therefore difficult to judge the wider applicability of the method, despite its inherent appeal. If this type of validation could be provided (even in partial form), I would recommend this paper be accepted as part of the proceedings, but otherwise I ask that it not be included. More detailed comments follow.\n\nAs stated above, the problematic component of the work concerns the numerical validation. The motivation and description of the method is quite sound, but could be better explained in parts. For example, the authors switch back and forth between Euler angles θ_i and quaternions q_i (they are simply introduced without explanation on line 63), which can make for confusing reading. The authors also state that the mapping from S³ to SO(3) is a double cover, but do not explain whether this is a problem or not for the problem setting. On lines 120–121, the authors also discuss size invariance, but it is not clear why this is necessary. When are projections expected to be of different sizes? This should not be the case in the numerical experiments discussed since the training data and testing data arise from the same protein structure. A similarly confusing statement is on line 127, where the authors claim that “a space of n_f = 4 dimensions does not have room for G_w to represent other factors of variation”. What does this mean? Why are we constrained to have n_f = 4?\n\nConstructing viewing angle estimates from a set of distance estimates also deserves some closer discussion in Section 2.3. Indeed, constructing a global set of angle estimates from smaller local distance estimates can lead to inconsistencies, as pointed out by Zhao and Singer, 2014 (Section 2.1). Why do we expect that constructing small subsets of the embedding at a time will result in a globally consistent embedding?\n\nFor the evaluation method described in Section 2.4, it is a bit problematic to use a non-deterministic error measure since it induces its own “error” for each run. It would be better if this could be replaced with a deterministic error. One way to achieve this is to replace the L¹-type norm used on the rotations and instead use a Frobenius norm on the rotation matrices. In this case, the optimal rotational alignment between the sets can be calculated by an SVD. That being said, perhaps the authors have other reasons for choosing the error measure that make it more appropriate than the simpler Frobenius norm. If that is the case, it should be motivated in the text.\n\nFor the numerical validation, it is striking that only uniform distributions of viewing angles are tested. First, these are relatively uncommon in experimental data, with a preferred orientation being the norm. Second, while many algorithms tend to work well for uniform distributions, that is not necessarily the case with non-uniform distributions. It would therefore be a good test of the approach to evaluate its performance in this regime.\n\nA big problem with the evaluation is the choice of datasets. The authors acknowledge this in the discussion, but all the same, it would be good to include the results of some preliminary experiments here. For example, the networks are trained on one of two molecules. What happens if one network is used to predict the distances for the other dataset? Does performance suffer greatly or is there a degree of robustness here. What if the network is trained on both datasets? The universality of the approach will be crucial when extending the method to experimental data, so it is important to know what can be expected here, even at a preliminary level. Other questions related to training mismatch concern noise levels. To what extent can we train on one noise level and test on another?\n\nWhen it comes to the noise levels, it is hard to extract a useful context. For example, the authors give the noise variance of the data, but without knowing the variance of the images, it is not obvious if this is a high or low noise level. Later in the text, we see that σ² = 16 corresponds to an SNR of –12 dB (which presumably means an SNR of 10 ^ (-1.2), but we are not given the formula). This is a low SNR, to be sure, but not exceptionally low for experimental cryo-EM data. Judging from the images shown in Figure 11 (which presumably have σ² = 16), these are indeed of moderate SNR for cryo-EM.\n\nAnother piece of the validation section that is missing is comparison to other methods. In the introduction, a host of alternative methods for ab initio structure determination are listed, but none are applied to the synthetic data used to evaluate the proposed method. It would be simple enough to run the synthetic data through the ab initio pipeline of Cryosparc, Relion, or Aspire to provide a simple baseline for comparison. Right now, the viewing angle accuracies and reconstruction resolutions are hard to place since there is no basis for comparison. This is especially important since it would provide an indication of the benefit derived from a data-driven approach, where specific features in the images are used to estimate the distances. Due to the training–testing regime, it is expected that the proposed method would perform better in this setting (since it possesses a much stronger prior compared to the baseline methods), but this needs to be validated numerically.\n\nSome minor comments:\n\n– Constant references to figures outside of the main text makes for tedious reading. I would try to include these in the main text (perhaps by cutting other figures) or remove references entirely. If this cannot be resolved, I suggest labeling these figures so that it is clear that they refer to supplementary material (Figure S1, etc.).\n\n– On line 100, it should be clarified what is meant by the “magnitude” of a rotation.\n\n– On line 102, what does it mean the “design” the estimator, and why is it intricate or impossible? What do invariants have to do with this?\n\n– On line 120, a CNN provides equivariance (covariance) not necessarily invariance to translation (unless a pooling layer is included).\n\n– On line 143, “our loss function” is referred to but has yet to be introduced.\n\n– On lines 172, 244, 248, 252, and 268, spaces are missing between the number and the unit (Å).\n\n– On lines 180, 191, and the caption of Table 1, there is an extra space after the comma in the numbers larger than one thousand.\n\n– In footnote 6, sampling the Euler angles from uniform distributions does not yield a uniform distribution on SO(3). The colatitude (θ₂) has to be reweighted.\n\n– On line 244, “an” should be removed.\n\n– On lines 228–229, what does it mean that “the observed overfitting indicates that more training data should further decrease the sensitivity of the SNN to noise”?\n\n– On line 240, “robuster” should be “more robust”.\n\n– On line 253, why does higher resolution of the ground truth imply better reconstruction resolution?\n\n– Several article and journal titles are improperly capitalized in the bibliography, including for references [11], [12], [18], [20], [21], [28], [30], [31], [32], [40], [41], and [43].\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the limitations of their model and their numerical validation at the end of the manuscript. When it comes to the validation, I think more can be done (as stated in the main part of the review). This being an application of data-driven methods to a problem in structural biology, I do not see any potential negative societal impacts that would emerge directly from this work. The authors are, however, helpful in providing an estimated GHG budget for training their model.\n", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues are raised by the manuscript.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The Cryo-EM problem is of significant importance and ML techniques for it are in scope for the NeurIPS conference.  As the submission did not provide a novel machine learning algorithm, the paper was evaluated primarily on its potential impact in application.  There was a consensus among the reviewers that the paper suffers from methodological concerns, including a lack of comparison to appropriate baselines given the same setup, the generalizability of the method toward proteins not seen in training data, and unclear performance improvement as part of a full cryo-EM pipeline.  Consequently, it is difficult to know how significant the result is in application.  The authors are encouraged to address these concerns in a future submission.  

Decision: Reject