Learning to recover orientations from projections in single-particle cryo-EM

A major challenge in single-particle cryo-electron microscopy ( cryo-EM ) is that1 the orientations adopted by the 3D particles prior to imaging are unknown ; yet , this2 knowledge is essential for high-resolution reconstruction . We present a method3 to recover these orientations directly from the acquired set of 2D projections.4 Our approach consists of two steps : ( i ) the estimation of distances between pairs5 of projections , and ( ii ) the recovery of the orientation of each projection from6 these distances . In step ( i ) , pairwise distances are estimated by a Siamese neural7 network trained on synthetic cryo-EM projections from resolved bio-structures.8 In step ( ii ) , orientations are recovered by minimizing the difference between9 the distances estimated from the projections and the distances induced by the10 recovered orientations . We evaluated the method on synthetic cryo-EM datasets.11 Current results demonstrate that orientations can be accurately recovered from12 projections that are shifted and corrupted with a high level of noise . The accuracy13 of the recovery depends on the accuracy of the distance estimator . While not14 yet deployed in a real experimental setup , the proposed method offers a novel15 learning-based take on orientation recovery in SPA . Our code is available at https:16 //github.com/anonymous/protein-reconstruction.17 1 Introduction18 Single-particle cryo-electron microscopy ( cryo-EM ) has revolutionized the field of structural biology19 over the last decades [ 1 , 2 , 3 ] . The use of electron beams to image ice-embedded samples has20 permitted the recovery of 3D bio-structures at unprecedented resolution . This “ resolution revolution ” 21 has had a tremendous impact in biomedical research , providing invaluable insights into the biological22 processes that underlie many current diseases.23 In single-particle cryo-EM , every 3D particle adopts a random orientation θi in the ice layer before24 being imaged . Hence , the projection geometry associated to each acquired 2D projection ( Figure 1 ) 25 is unknown . Yet , this knowledge is essential for the tomographic reconstruction of bio-structures [ 4 ] .26 We consider that a cryo-EM measurement ( i.e. , a projection ) pi ∈ Rnp is acquired through27 pi = CϕStiPθix+ n , ( 1 ) where x ∈ Rnx is the unknown 3D density map [ 5 ] ( Coulomb potential ) . The operator Pθi : Rnx →28 Rnp is the projection along the orientation θi ( i.e. , the x-ray transform ) . The operator Sti : Rnp →29 Rnp is a shift of the projection by ti = ( ti1 , ti2 ) . The convolution operator Cϕ : Rnp → Rnp models30 the microscope point-spread function ( PSF ) with parameters ϕ = ( d1 , d2 , αast ) , where d1 is the31 defocus-major , d2 is the defocus-minor , and αast is the angle of astigmatism [ 6 , 7 ] . Finally , n ∈ Rnp32 represents additive noise . Figure 11 illustrates the effect of projection , shift , and noise . The challenge33 is then to reconstruct x from a set of projections { pi } Pi=1 acquired along unknown orientations.34 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . A popular approach is to alternatively refine the 3D structure and estimated orientations [ 8 , 9 , 10 , 11,35 12 , 13 ] . Yet , the outcome of these iterative-refinement procedures is often predicated on the quality36 of the initial reconstruction , or , equivalently , on the initial estimation of the orientations [ 14 , 15 ] .37 Several methods have been designed to produce a first rough ab initio structure for the refinement38 procedure [ 16 ] . Moment-matching techniques [ 17 , 18 , 19 , 20 ] reconstruct an initial structure such39 that the first few moments of the distribution of its theoretical measurements match the ones of40 its experimental projections ; however , they typically remain sensitive to error in data and can41 require relatively high computational complexity . Based on the central-slice theorem , common-lines42 methods [ 21 , 8 , 22 , 23 , 24 , 25 , 26 ] aim at uniquely determining the orientations of each projection by43 identifying the common-lines between triplets of projections—a real challenge given the massive44 amount of noise . Alternatively , the marginalized maximum likelihood ( ML ) formulation of the45 reconstruction problem [ 11 ] —classically used for the iterative-refinement procedures themselves—46 can be minimized using stochastic gradient descent [ 27 ] . This permits to avoid the need for an initial47 volume estimate , at the possible cost of greater convergence instability.48 More recently , the recovery of geometrical information from unknown view tomography of 2D point49 sources has been proposed [ 28 ] , but the extension to 3D cryo-EM tomography is not straightforward.50 Finally , [ 29 ] proposed to recover the in-plane rotations by learning to embed projections in an51 appropriate latent space , but only after directions had been estimated through three rounds of 2D52 classification in RELION.53 Despite the aforementioned advances , providing a robust initial volume remains a challenge due to54 the high-dimensionality and ill-posedness of the underlying optimization problem . On the other hand,55 the remarkable ability of convolutional neural networks to capture relevant representations of images56 has had a profound influence in imaging [ 30 ] . In this work , we present a learning-based approach to57 recover the unknown orientations directly from the acquired set of projections—without the need for58 an intermediate reconstruction procedure or an initial volume estimate.59 2 Method60 Our approach relies on two observations ( Figure 2 ) , yielding two steps ( Figure 3 ) . First , the more61 similar two projections ( pi , pj ) , the more likely they originated from two particles that adopted close62 orientations ( qi , qj ) in the ice prior to imaging ; 1 this observation guides a number of applications in63 the field [ 2 ] . Hence , we aim to estimate distances between orientations dq ( qi , qj ) from the projections64 as d̂p ( pi , pj ) , which we discuss in §2.2 . Second , an orientation q is constrained by the distances65 between itself and the other orientations { d ( q , qj ) } . Hence , we aim to recover orientations { q̂k } such66 that the induced distances { dq ( q̂i , q̂j ) } are close to the estimated distances { d̂p ( pi , pj ) } , which we67 discuss in §2.3 . All in all , from a set of projections { pk } , we aim to recover their orientations { q̂k } 68 such that dq ( q̂i , q̂j ) ≈ d̂p ( pi , pj ) ≈ dq ( qi , qj ) , with equality if d̂p and { q̂k } are perfectly estimated.69 Our approach is similar to [ 31 ] . While the authors reconstruct 2D images from 1D projections , they70 rely on the same two-step approach : they ( i ) estimate distances as d̂p ( pi , pj ) = ‖pi − pj‖2 then71 ( ii ) recover the orientations by spectrally embedding that distance graph . The Euclidean distance is72 however not robust to perturbations : for example , two projections that only differ by a shift St of one73 pixel would be considered far apart while their orientations are the same . They noted that issue and74 we observed it too ( Appendix E ) . To circumvent this , we propose to learn d̂p from examples ( §2.2 ) .75 2.1 Representation of orientations with quaternions76 The orientation of a 3D particle with respect to the microscope ’ s detector plane is a rotation relative to77 a reference orientation ( Figure 1 ) . The group of all 3D rotations under composition is identified with78 SO ( 3 ) , the group of 3× 3 orthogonal matrices with determinant 1 under matrix multiplication . A79 rotation matrix Rθ ∈ SO ( 3 ) can be decomposed as a product of ( 3 2 ) = 3 independent rotations , for80 example as Rθ = Rθ3Rθ2Rθ1 , where θ = ( θ3 , θ2 , θ1 ) ∈ [ 0 , 2π [ × [ 0 , π ] × [ 0 , 2π [ are the ( extrinsic81 and proper ) Euler angles in the ZY Z convention ( a common parameterization in cryo-EM ) [ 32 ] .82 While Euler angles are a concise representation of orientation ( 3 numbers for 3 degrees of freedom ) ,83 they suffer from a topological constraint—there is no covering map from the 3-torus to SO ( 3 ) —84 which manifests itself in the gimbal lock , the loss of one degree of freedom when θ2 = 0 . This makes85 their optimization by gradient descent ( §2.3 ) problematic . On the other hand , optimizing rotation86 matrices ( made of 9 numbers ) would require computationally costly constraints ( orthogonality and87 determinant 1 ) to reduce the degrees of freedom to 3 . Moreover , the distance between orientations88 can not be directly computed from Euler angles and is costly ( 30 multiplications ) to compute from89 rotation matrices [ 33 ] . We solve both problems by representing orientations with unit quaternions.90 Quaternions q ∈ H are an extension of complex numbers2 of the form q = a+ bi+ cj + dk where91 a , b , c , d ∈ R. Unit quaternions q ∈ S3 , where S3 = { q ∈ H : |q| = 1 } is the 3-sphere ( with92 the additional group structure inherited from quaternion multiplication ) , concisely and elegantly93 represent a rotation of angle θ about axis ( x1 , x2 , x3 ) as q = cos ( θ/2 ) +x1 sin ( θ/2 ) i+x2 sin ( θ/2 ) j+94 x3 sin ( θ/2 ) k. They parameterize rotation matrices as95 Rq = a2 + b2 − c2 − d2 2bc− 2ad 2bd+ 2ac2bc+ 2ad a2 − b2 + c2 − d2 2cd− 2ab 2bd− 2ac 2cd+ 2ab a2 − b2 − c2 + d2 . 1Up to protein symmetries , which we discuss later . 2The algebra H is similar to the algebra of complex numbers C , with the exception of multiplication being non-commutative . Note that S3 → SO ( 3 ) is a two-to-one mapping ( a double cover ) as q and −q represent the same96 orientation . Unlike Euler angles , S3 is isomorphic to the universal cover of SO ( 3 ) . Hence , the97 distance between two orientations , i.e. , the length of the geodesic between them on SO ( 3 ) , is98 dq : S3 × S3 → [ 0 , π ] , dq ( qi , qj ) = 2 arccos ( |〈qi , qj〉| ) , ( 2 ) where 〈· , ·〉 is the inner product , and the absolute value |·| ensures that dq ( qi , qj ) = dq ( qi , −qj ) . The99 distance dq ( qi , qj ) corresponds to the magnitude of the rotation R∗ such that Rqi = R∗Rqj [ 33 ] .100 2.2 Distance learning101 We aim to estimate a function d̂p such that d̂p ( pi , pj ) ≈ dq ( qi , qj ) . While we could in principle102 design d̂p , that would be intricate—if not impossible—partly because the invariants are difficult103 to specify . We instead opt to learn d̂p , capitalizing on ( i ) the powerful function approximation104 capabilities of neural networks , and ( ii ) the possibility to generate realistic datasets supported by the105 availability of numerous 3D atomic models3 and our ability to model the cryo-EM imaging procedure.106 From a training dataset { pi , qi } Pi=1 , we learn the projection distance107 d̂p = argmin dp LDE , where LDE = ∑ i , j ∣∣dp ( pi , pj ) − dq ( qi , qj ) ∣∣2 ( 3 ) is the loss and dq is defined in ( 2 ) . The dp is parameterized as the Siamese neural network ( SNN ) [ 34 ] 108 dp ( pi , pj ) = df ( Gw ( pi ) , Gw ( pj ) ) , where Gw is a convolutional neural network with weights w that is trained to extract the most relevant109 features fi ∈ Rnf from a projection pi . SNNs , also termed “ twin networks ” , are commonly used in110 the field of deep metric learning to learn similarity functions [ 35 ] . We set the feature space distance111 df as the cosine distance to facilitate the learning of a d̂p that respects the elliptic geometry of S3112 ( Appendix F ) . Figure 4 illustrates the proposed learning paradigm.113 As evaluating a sum over P 2 pairs is computationally intractable for cryo-EM datasets with typically114 P in the order of 105 projections , we sample the sum and minimize ( 3 ) with stochastic gradient115 descent ( SGD ) over small batches of pairs . The weights w are updated by back-propagation.116 The architecture of Gw is described in Appendix G. When designing the architecture , we constrain117 the functional space from which the trained Gw is drawn and express our prior expert knowledge . For118 example , we realize shift invariance , i.e. , a guarantee that a shift St does not change our estimated119 distances and orientations , with a fully convolutional architecture . Size invariance , i.e. , taking120 projections p of varying sizes np while yielding a representation f of a fixed size nf , is realized by a121 final average pooling layer . As we do not ( yet ) know how to realize an invariance to noise or PSF , we122 resort to data augmentation , i.e. , training on perturbed projections . In §3.4 , we show that a built-in123 invariance ( shift ) is far preferable to one learned through augmentation ( noise ) . Finally , as projections124 3https : //www.ebi.ac.uk/pdbe/emdb are made by integrating through the 3D volume , projections from opposed directions are mirrors of125 each other.4 That is another kind of physical knowledge that should ideally be built into our method.126 One could hope to train Gw to directly map projections to orientations as q̂i = fi = Gw ( pi ) . While127 that would avoid the orientation recovery step , a space of nf = 4 dimensions does not have room for128 Gw to represent the other factors of variation in p , such as different noise levels , PSFs , or proteins.129 We tested that hypothesis in Appendix F.130 2.3 Orientation recovery131 The task of recovering points based on their relative distances has been extensively studied . Many132 methods aim at mapping high-dimensional data onto a lower-dimensional space while preserving133 distances , primarily for dimensionality reduction and data visualization . Well-known examples134 include MDS [ 36 ] , Isomap [ 37 ] , LLE [ 38 ] , Laplacian eigenmaps [ 39 ] , t-SNE [ 40 ] , and UMAP [ 41 ] .135 The embedding of distance matrices in Euclidean space ( given by their eigenvectors ) is especially136 well-described . In particular , the framework of Euclidean distance matrices ( EDMs ) [ 42 ] provides137 theoretical guarantees on the recovery of points from distances.138 We however aim to embed the orientations q in S3 ( §2.1 ) , a setting for which we are unaware of any139 theoretical characterization ( e.g. , on the shape of the loss function or its behavior when distances are140 missing or noisy ) . The fact that S3 is locally Euclidean does however offer some hope . Indeed , despite141 the non-convexity and the lack of theoretical guarantees , we are able to appropriately minimize our142 loss function , as we experimentally demonstrate in Appendix D.143 We recover the orientations of a set of projections { pk } P k=1 through144 { q̂k } P k=1 = argmin { qk∈S3 } LOR , where LOR = ∑ i , j ∣∣∣d̂p ( pi , pj ) − dq ( qi , qj ) ∣∣∣2 ( 4 ) is the loss and d̂p is the estimator trained in ( 3 ) . Note that the sole difference with ( 3 ) is that the145 minimization is performed over the orientations q rather than the distance dp . Here again , we sample146 the sum in practice and minimize ( 4 ) with mini-batch SGD . Sampling the sum amounts to building a147 sparse ( instead of complete ) distance graph before embedding , a common strategy.148 2.4 Evaluation149 While not a part of the method per se , we must evaluate the quality of the recovered orientations.150 Unfortunately , we can not directly take the difference between the recovered orientations { q̂k } Pk=1151 and the true orientations { qk } Pk=1 as orientations are rotations up to an arbitrary reference orientation.152 Any global rotation or reflection of the recovered orientations is as valid as any other , i.e. , dq ( qi , qj ) =153 dq ( Tqi , Tqj ) ∀T ∈ O ( 4 ) , where O ( 4 ) is the group of 4× 4 orthogonal matrices . Hence , we align154 the sets of orientations and compute the mean orientation recovery error as155 EOR = min T∈O ( 4 ) 1 P P∑ i=1 ∣∣dq ( qi , Tq̂i ) ∣∣ . ( 5 ) We implement T as a product of ( 4 2 ) = 6 independent rotations and an optional reflection:156 T = [ m 0 0 I ] ∏ 1≤i < j≤4 Tθij , m ∈ { −1 , 1 } , θij ∈ [ 0 , 2π [ , ( 6 ) where Tθij ∈ SO ( 4 ) is a rotation by angle θij on the ( xi , xj ) plane.157 In practice , we again minimize ( 5 ) with mini-batch SGD . Because O ( 4 ) is disconnected , we optimize158 the 6 angles separately for m = 1 ( proper rotations ) and m = −1 ( improper rotations ) . Figure 15159 shows an alignment to EOR = 0 after a perfect recovery.160 4That fact prevents the resolution of chirality , i.e. , we can not distinguish a protein from its mirrored form . 3 Experiments161 We first evaluated whether orientation recovery through ( 4 ) was feasible assuming perfect distances,162 and how it was affected by errors in the distances ( §3.2 ) . We then learned to estimate the distances163 through ( 3 ) , and evaluated the accuracy of this procedure ( §3.3 ) and its robustness to perturbations of164 the projections ( §3.4 ) . Finally , we ran the whole machinery on a synthetic dataset to assess how well165 orientations could be recovered from estimated distances ( §3.5 ) .166 3.1 Experimental conditions167 Density maps . We considered two proteins ( Figure 10 ) : the β-galactosidase , a protein with a168 dihedral ( D2 ) symmetry , and the lambda excision HJ intermediate ( HJI ) , an asymmetric protein169 with local cyclic ( C1 ) symmetry . Their deposited PDB atomic models are 5a1a [ 43 ] and 5j0n [ 44 ] ,170 respectively . From these atomic models , we generated the density maps in Chimera [ 45 ] by fitting the171 models with a 1Å map for 5a1a and a 3.67Å map for 5j0n ; this gave us a volume of 110×155×199172 voxels for 5a1a and one of 69× 57× 75 voxels for 5j0n.173 Protein symmetries . Symmetries are problematic when learning distances : two projections can174 be identical while not originating from the same orientation , which breaks an axiom of distance175 functions ( identity of indiscernibles ) . Figure 16b illustrates this problem . To capture only one of four176 identical projections of 5a1a , we restricted directions to ( θ2 , θ1 ) ∈ [ 0 , π [ × [ 0 , π2 [ ( a quarter of the177 sphere , illustrated in Figure 12a ) for that protein . This treatment of symmetries is incomplete5 but178 sufficient for a proof-of-concept.179 Projections . Using the ASTRA projector [ 46 ] , we generated P = 5 , 000 synthetic projections180 of 275 × 275 pixels ( downsampled to 116 × 116 ) for 5a1a and 116 × 116 pixels for 5j0n , taken181 from uniformly sampled orientations.6 We then perturbed the measurements with different levels of182 additive Gaussian noise [ 47 , 48 ] and off-centering shifts . Figure 11 displays some samples.183 Datasets . For each protein , we split the projections into training , validation , and test subsets , and184 created disjoint pairs of projections from each ( Table 1 ) . The training and validation sets were used to185 train and evaluate the SNN , while the test set was used to evaluate orientation recovery given a trained186 SNN . Sampling orientations ( mostly ) uniformly induces a distribution of distances that is skewed187 towards larger distances ( shown in Figure 12b ) . As this would skew LDE and bias d̂p , we further188 sampled 1 % of the training and validation pairs to make the distribution of distances uniform—for189 d̂p to be uniformly accurate over the whole [ 0 , π ] range of distances ( see Appendix B for further190 illustrations ) . While 1 , 650 projections were enough to perfectly reconstruct the density maps ( as191 shown in Figures 9e and 9j ) , our method is not limited by the number of projections as optimization192 is done per batch . Optimization settings are described in Appendix C.193 3.2 Sensitivity of orientation recovery to errors in distance estimation194 We first evaluated the feasibility of orientation recovery assuming that the exact distances were known.195 The method successfully recovers the orientation of every projection in this case ( see Appendix D ) .196 To evaluate the robustness of ( 4 ) , we perturbed the distances prior to recovery with an error sampled197 from a Gaussian distribution with mean 0 and variances σ2 ∈ [ 0.0 , 0.8 ] . Figure 5 shows that the198 recovery error EOR is a monotonic function of the error in distances : from EOR = 0 with exact199 distances to EOR ≈ 0.2 radians ( ≈ 11.5° ) for σ2 = 0.8.200 These results demonstrate that the performance of orientation recovery ( 4 ) depends on the quality of201 the estimated distances , which advocates for a proper and extensive training of the SNN . Moreover,202 we observe that LOR is a reliable proxy for EOR , allowing us to assess recovery performance in the203 absence of ground-truth orientations ( i.e. , when recovering the orientations of real projections ) .204 5The remaining issue is that one of four distances is arbitrarily chosen per pair of projections . 6Orientations used in §3.2 ( Figure 5 ) and §3.4 ( Figure 7 ) were actually obtained by uniformly sampling the Euler angles θ , constrained to ( θ3 , θ2 , θ1 ) ∈ [ 0 , 2π [ × [ 0 , π2 [ × [ 0 , 2π [ for 5j0n . Our conclusions would be identical if orientations were uniformly sampled from SO ( 3 ) instead . Table 1 : Split of P = 5 , 000 projections in training , validation , and test subsets . Dataset P P 2 Used pairs Training 2,512 ( 50 % ) 6,310,144 63,101 Validation 838 ( 17 % ) 702,244 7,022 Test 1,650 ( 33 % ) 2,722,500 2,722,500 ��� ��� ��� ��� ��� � ���� ���� ���� � � � ���� ���� ���� � � � ��� ��� ��� ��� ��� � ���� ���� ���� � � � ���� ���� ���� � � � Figure 5 : Orientation recovery from perturbed distances on 5j0n ( left ) and 5a1a ( right ) . � �� ��� ����� ��� ��� ��� ��� � � � ����������� ����������� ������������ �������������� � �� ��� ����� ��� ��� ��� ��� � � � ����������� ����������� ������������ �������������� ( a ) Loss converged on 5j0n ( left ) and 5a1a ( right ) . � � � � ������ ��� � � � � � � �� �� � �� ����������� ��������������� � � � � ������ ��� � � � � � � �� �� � �� ����������� ��������������� ( b ) Relationship between d̂p and dq on 1 , 000 pairs from the test sets of 5j0n ( left ) and 5a1a ( right ) . Figure 6 : Distance learning . 3.3 Learning to estimate distances205 We evaluated the ability of the SNN to learn to approximate the orientation distance dq . For206 comparison , we evaluated a baseline , the Euclidean distance d̂p ( pi , pj ) = ‖pi , pj‖2 , in Appendix E.207 Figure 6a shows the convergence ofLDE , reached in about 50 epochs . Figure 6b shows the relationship208 between the distance d̂p estimated from projections and the true distance dq . The outliers for 5a1a209 are explained by our incomplete treatment of its symmetry . While our learned distance function210 is a much better estimator than the Euclidean distance—compare Figure 6b with Figure 16—they211 share one characteristic : both plateau and underestimate the largest distances . We did attenuate212 the phenomenon by sampling training distances uniformly ( see §3.1 ) , and the issue is much less213 severe than with the Euclidean distance . An alternative could be to only rely on smaller distances for214 recovery . That would however require the addition of a spreading term in ( 4 ) to prevent the recovered215 orientations to collapse.216 These results confirm that a SNN is able to estimate differences in orientations from projections alone,217 even though much has yet to be gained from improving upon the rather primitive SNN architecture218 we are currently using . The use of additional training data should help further diminish overfitting.219 3.4 Sensitivity of distance learning to perturbations in the projections220 We first demonstrated that the learning of distances is insensible to off-centering shifts ( Figure 7a ) ,221 which is expected given that shift invariance is built in our SNN ( see §2.2 ) .222 As we can not —or do not yet know how to—build noise invariance in the SNN architecture , we trained223 the SNN on noisy projections and evaluated whether it could learn to treat noise as an irrelevant224 information . Figure 7b shows EOR ≈ 0.16 radians ( ≈ 9° ) for noiseless projections and EOR ≈ 0.42225 radians ( ≈ 24° ) for a more realistic noise variance of σ2 = 16 ( with signal-to-noise ratio of -12 dB ) .226 Whereas a naive distance function ( e.g. , an Euclidean distance ) would be extremely sensitive to noise,227 the SNN mostly learned to discard it . Moreover , the observed overfitting indicates that more training228 data should further decrease the sensitivity of the SNN to noise.229 Note that we did not evaluate sensitivity to the PSF at this stage but expect a similar behavior.230 Here again ( §3.2 ) , we observed that ( i ) the estimation of more accurate distances ( a smaller LDE ) 231 leads to the recovery of more accurate orientations ( a smaller LOR and EOR ) , and that ( ii ) an higher232 recovery loss LOR induces an higher error EOR.233 3.5 Orientation recovery and reconstruction of density maps234 As a proof-of-concept , we attempted to solve the full inverse problem posed by ( 1 ) , i.e. , to reconstruct235 the density maps x̂ from sets of projections { pi } and their orientations { q̂i } recovered through the236 proposed method . It is worth noting that , at this stage of development , we only trained the SNN on237 projections originating from the protein we were attempting to reconstruct . In addition , reconstruction238 was performed with a direct reconstruction algorithm ( ASTRA ’ s GPU implementation of the CGLS239 algorithm ) rather than with a robuster iterative method . This is a specific experimental case that only240 partially shines light on the applicability of the method in real situations ; this is discussed in §4.241 Figure 8a shows the recovery of orientations from distances that were estimated from noiseless242 projections of 5j0n . A mean error of EOR ≈ 0.20 radians ( ≈ 11° ) in the recovered orientations led243 to a reconstruction with a resolution of 12.2Å at a Fourier shell coefficient ( FSC ) of 0.5 , shown in244 Figure 9c . As predicted by our other experiments , corrupting the projections with noise ( σ2 = 16 ) 245 negatively impacts the quality of the recovered orientations ( Figure 8b ) ; the obtained mean error246 is then EOR ≈ 0.25 radians ( ≈ 14° ) . Unsurprisingly , this leads to a reconstruction with a lower247 resolution of 15.2Å , shown in Figure 9d . ( Note that reconstruction was here obtained from the248 noiseless projections , the goal being to evaluate only the impact of orientation mis-estimation . ) 249 Finally , Figures 8c , d show the recovery of orientations from noiseless and noisy projections of 5a1a.250 A mean error of EOR ≈ 0.13 radians ( ≈ 7° ) in both cases led to reconstructions with resolutions of251 8.0Å and 9.6Å , shown in Figures 9h , i . Distance estimation , orientation recovery , and reconstruction252 performed better on 5a1a than 5j0n because its ground-truth density is of higher resolution.253 These results tend to indicate that a reasonable first structure can be reconstructed from projections254 whose orientations have been recovered through our method.255 4 Discussion256 In this work , we explored the use of distance learning between pairs of 2D cryo-EM projections257 from a 3D protein structure to infer the unknown orientation at which each projection was imaged258 from . Our two-step method relies on the estimation of pairwise distances between unseen projections,259 followed by the recovery of the orientations from these distances.260 The method has been evaluated on synthetic datasets for two different proteins . The results provide261 key insights on the viability of the proposed scheme . First , they demonstrate that a SNN can learn262 a distance function between projections that estimates the difference in their orientation ( §3.3 ) and263 that is invariant to shifts and robust to increasing levels of noise ( §3.4 ) —an important condition in264 cryo-EM . Second , they demonstrate that an accurate estimation of distances leads to an accurate265 recovery of orientations ( §3.2 , §3.4 ) . Finally , our method was able to recover orientations with266 an error of 0.12 to 0.25 radians ( 7 to 14° ) —leading to an initial volume with a resolution of 8 to267 15Å ( §3.5 ) . In summary , the more accurate the estimated distances , the more precise the recovered268 orientations , and , ultimately , the higher-resolution the reconstructed volume.269 While the method is not yet ready to be deployed in practice , we believe that a series of developments270 could make it relevant for single-particle cryo-EM reconstruction.7 As previously discussed , the271 results underline the importance of learning an accurate distance estimator . In this regard , the272 performance of the SNN could be improved . First , the architecture of the twin convolutional neural273 networks should be expanded and tuned . Second , training could be improved , perhaps by providing274 more supervision by separately predicting the differences in direction ( θ2 , θ1 ) and in-plane angle θ3.275 Importantly , the SNN would be better trained on a more diverse cryo-EM dataset . Indeed , its success276 as a faithful estimator eventually relies on our capacity to generate a synthetic training dataset whose277 data distribution is diverse enough to cover that of unseen projection datasets . Such realistic cryo-EM278 projections could be generated by relying on a more expressive formulation of the cryo-EM physics279 and taking advantage of the thousands of atomic models available in the PDB . In particular , a280 necessary extension will be to include the effects of the PSF and to evaluate its impact.281 A final phase of tests before deploying the method on real cryo-EM measurements will be to282 extensively test the method on “ unseen proteins ” , i.e. , proteins whose simulated projections have283 never been seen by the SNN . In this regard , an interesting aspect of our method is that the twin284 networks within the SNN intrinsically predict the relationship between projections , allowing the SNN285 as a whole to abstract the particular volume . Learning should benefit from the profound structural286 similarity shared by proteins—after all , they are all derived from the same 21 building blocks.287 Training our 4.5M parameter model ( see Appendices G and C ) has the following negative environ-288 mental impact : it consumes 13 kWh of energy , which produces 6.36 lbs of CO2 on average [ 49 ] .289 7Note that the present project will not be further continued by its authors due to other professional occupations . Hence , we strongly encourage anyone interested to build on these ideas and , hopefully , make it a practical tool . References290 [ 1 ] J. Dubochet , M. Adrian , J.-J . Chang , J.-C. Homo , J. Lepault , A. W. McDowall , and P. Schultz,291 “ Cryo-electron microscopy of vitrified specimens , ” Quarterly Reviews of Biophysics , vol . 21,292 no . 2 , pp . 129–228 , 1988.293 [ 2 ] J. Frank , Three-dimensional electron microscopy of macromolecular assemblies : Visualization294 of biological molecules in their native state . Oxford University Press , 2006.295 [ 3 ] Nature , “ Method of the year 2015 , ” Nature Methods , vol . 13 , no . 1 , 2016.296 [ 4 ] F. Natterer , The mathematics of computerized tomography . Society for Industrial and Applied297 Mathematics , jan 2001.298 [ 5 ] F. DiMaio , D. A. Kondrashov , E. Bitto , A. Soni , C. A. Bingman , G. N. Phillips , and299 J. W. Shavlik , “ Creating protein models from electron-density maps using particle-filtering300 methods , ” Bioinformatics , vol . 23 , no . 21 , pp . 2851–2858 , Nov. 2007 . [ Online ] . Available:301 https : //academic.oup.com/bioinformatics/article/23/21/2851/374177302 [ 6 ] M. Vulović , R. B. Ravelli , L. J. van Vliet , A. J. Koster , I. Lazić , U. Lücken , H. Rullgård,303 O. Öktem , and B. Rieger , “ Image formation modeling in cryo-electron microscopy , ” Journal of304 Structural Biology , vol . 183 , no . 1 , pp . 19–32 , Jul . 2013.305 [ 7 ] H. Rullgård , L.-G. Öfverstedt , S. Masich , B. Daneholt , and O. Öktem , “ Simulation of transmis-306 sion electron microscope images of biological specimens , ” Journal of Microscopy , vol . 243,307 no . 3 , pp . 234–256 , 2011.308 [ 8 ] P. A. Penczek , R. A. Grassucci , and J. Frank , “ The ribosome at improved resolution : New309 techniques for merging and orientation refinement in 3D cryo-electron microscopy of biological310 particles , ” Ultramicroscopy , vol . 53 , no . 3 , pp . 251–270 , 1994.311 [ 9 ] T. Baker and R. Cheng , “ A model-based approach for determining orientations of biological312 macromolecules imaged by cryoelectron microscopy , ” Journal of Structural Biology , vol . 116,313 no . 1 , pp . 120–130 , 1996 . [ Online ] . Available : http : //www.sciencedirect.com/science/article/pii/314 S1047847796900209315 [ 10 ] A. P. Dempster , N. M. Laird , and D. B. Rubin , “ Maximum likelihood from incomplete data via316 the EM algorithm , ” Journal of the Royal Statistical Society . Series B ( Methodological ) , vol . 39,317 no . 1 , pp . 1–38 , 1977.318 [ 11 ] F. J. Sigworth , “ A maximum-likelihood approach to single-particle image refinement , ” Journal319 of structural biology , vol . 122 , no . 3 , pp . 328–339 , 1998.320 [ 12 ] S. Scheres , “ A bayesian view on cryo-EM structure determination , ” Journal of molecular321 biology , vol . 415 , no . 2 , pp . 406–418 , 2012.322 [ 13 ] M. Zehni , L. Donati , E. Soubies , Z. J. Zhao , and M. Unser , “ Joint Angular Refinement and323 Reconstruction for Single-Particle Cryo-EM , ” IEEE Transactions on Image Processing , 2020.324 [ 14 ] C. O. S. Sorzano , R. Marabini , A. Pascual-Montano , S. H. Scheres , and J. M. Carazo , “ Opti-325 mization problems in electron microscopy of single particles , ” Annals of Operations Research,326 vol . 148 , no . 1 , pp . 133–165 , 2006.327 [ 15 ] R. Henderson , A. Sali , M. L. Baker , B. Carragher , B. Devkota , K. H. Downing , E. H. Egelman,328 Z. Feng , J. Frank , N. Grigorieff , W. Jiang , S. J. Ludtke , O. Medalia , P. A. Penczek , P. B.329 Rosenthal , M. G. Rossmann , M. F. Schmid , G. F. Schröder , A. C. Steven , D. L. Stokes , J. D.330 Westbrook , W. Wriggers , H. Yang , J . Young , H. M. Berman , W. Chiu , G. J. Kleywegt , and C. L.331 Lawson , “ Outcome of the first electron microscopy validation task force meeting , ” Structure,332 vol . 20 , no . 2 , pp . 205–214 , 2012.333 [ 16 ] A . Singer and F. J. Sigworth , “ Computational methods for single-particle electron cryomi-334 croscopy , ” Annual Review of Biomedical Data Science , vol . 3 , 2020.335 [ 17 ] Z. Kam , “ The reconstruction of structure from electron micrographs of randomly oriented336 particles , ” in Electron Microscopy at Molecular Dimensions . Springer , 1980 , pp . 270–277.337 [ 18 ] D. B. Salzman , “ A method of general moments for orienting 2d projections of unknown 3d338 objects , ” Computer vision , graphics , and image processing , vol . 50 , no . 2 , pp . 129–156 , 1990.339 [ 19 ] A. Goncharov , “ Integral geometry and three-dimensional reconstruction of randomly oriented340 identical particles from their electron microphotos , ” Acta Applicandae Mathematica , vol . 11,341 no . 3 , pp . 199–211 , 1988.342 [ 20 ] N. Sharon , J. Kileel , Y. Khoo , B. Landa , and A . Singer , “ Method of moments for 3-d single343 particle ab initio modeling with non-uniform distribution of viewing angles , ” Inverse Problems,344 2019.345 [ 21 ] M. Van Heel , “ Angular reconstitution : a posteriori assignment of projection directions for 3d346 reconstruction , ” Ultramicroscopy , vol . 21 , no . 2 , pp . 111–123 , 1987.347 [ 22 ] S. P. Mallick , S. Agarwal , D. J. Kriegman , S. J. Belongie , B. Carragher , and C. S. Potter,348 “ Structure and view estimation for tomographic reconstruction : A bayesian approach , ” in 2006349 IEEE Computer Society Conference on Computer Vision and Pattern Recognition ( CVPR ’ 06 ) ,350 vol . 2 . IEEE , 2006 , pp . 2253–2260.351 [ 23 ] A . Singer , R. R. Coifman , F. J. Sigworth , D. W. Chester , and Y. Shkolnisky , “ Detecting consistent352 common lines in cryo-EM by voting , ” Journal of structural biology , vol . 169 , no . 3 , pp . 312–322,353 2010.354 [ 24 ] L. Wang , A . Singer , and Z. Wen , “ Orientation determination of cryo-EM images using least355 unsquared deviations , ” SIAM journal on imaging sciences , vol . 6 , no . 4 , pp . 2450–2483 , 2013.356 [ 25 ] I. Greenberg and Y. Shkolnisky , “ Common lines modeling for reference free ab-initio recon-357 struction in cryo-EM , ” Journal of structural biology , vol . 200 , no . 2 , pp . 106–117 , 2017.358 [ 26 ] G. Pragier and Y. Shkolnisky , “ A common lines approach for ab initio modeling of cyclically359 symmetric molecules , ” Inverse Problems , vol . 35 , no . 12 , p. 124005 , 2019.360 [ 27 ] A. Punjani , J. L. Rubinstein , D. J . Fleet , and M. A. Brubaker , “ cryoSPARC : Algorithms for361 rapid unsupervised cryo-EM structure determination , ” Nature Methods , vol . 14 , no . 3 , p. 290,362 2017.363 [ 28 ] M. Zehni , S. Huang , I. Dokmanić , and Z. Zhao , “ Distance retrieval from unknown view364 tomography of 2d point sources , ” Electronic Imaging , vol . 2019 , no . 13 , pp . 134–1 , 2019.365 [ 29 ] N. Miolane , F. Poitevin , Y.-T. Li , and S. Holmes , “ Estimation of orientation and camera366 parameters from cryo-electron microscopy images with variational autoencoders and generative367 adversarial networks , ” 2019.368 [ 30 ] Y. LeCun , Y. Bengio , and G. Hinton , “ Deep learning , ” nature , vol . 521 , no . 7553 , pp . 436–444,369 2015.370 [ 31 ] R. R. Coifman , Y. Shkolnisky , F. J. Sigworth , and A . Singer , “ Graph laplacian tomography from371 unknown random projections , ” IEEE Transactions on Image Processing , vol . 17 , no . 10 , pp.372 1891–1899 , 2008.373 [ 32 ] C. Sorzano , R. Marabini , J. Vargas , J. Otón , J. Cuenca-Alba , A. Quintana , J. de la Rosa-Trevín,374 and J. Carazo , “ Interchanging geometry conventions in 3dem : mathematical context for the375 development of standards , ” in Computational Methods for Three-Dimensional Microscopy376 Reconstruction . New York , NY : Springer New York , 2014 , pp . 7–42 . [ Online ] . Available:377 https : //doi.org/10.1007/978-1-4614-9521-5_2378 [ 33 ] D. Q. Huynh , “ Metrics for 3D rotations : Comparison and analysis , ” Journal of Mathematical379 Imaging and Vision , vol . 35 , no . 2 , pp . 155–164 , 2009.380 [ 34 ] S. Chopra , R. Hadsell , and Y. LeCun , “ Learning a similarity metric discriminatively , with381 application to face verification , ” in 2005 IEEE Computer Society Conference on Computer382 Vision and Pattern Recognition ( CVPR ’ 05 ) , vol . 1 . IEEE , 2005 , pp . 539–546.383 [ 35 ] D. Yi , Z. Lei , S. Liao , and S. Z. Li , “ Deep metric learning for person re-identification , ” in 2014384 22nd International Conference on Pattern Recognition . IEEE , 2014 , pp . 34–39.385 [ 36 ] M. A. Cox and T. F. Cox , “ Multidimensional scaling , ” in Handbook of data visualization.386 Springer , 2008 , pp . 315–347.387 [ 37 ] J . B. Tenenbaum , V. d. Silva , and J. C. Langford , “ A global geometric framework for nonlinear388 dimensionality reduction , ” Science , vol . 290 , no . 5500 , pp . 2319–2323 , 2000 . [ Online ] .389 Available : https : //science.sciencemag.org/content/290/5500/2319390 [ 38 ] S. T. Roweis and L. K. Saul , “ Nonlinear dimensionality reduction by locally linear embedding , ” 391 Science , vol . 290 , no . 5500 , pp . 2323–2326 , 2000.392 [ 39 ] M. Belkin and P. Niyogi , “ Laplacian eigenmaps for dimensionality reduction and data represen-393 tation , ” Neural computation , vol . 15 , no . 6 , pp . 1373–1396 , 2003.394 [ 40 ] L. Van der Maaten and G. Hinton , “ Visualizing data using t-sne , ” Journal of machine learning395 research , vol . 9 , no . 11 , 2008.396 [ 41 ] L. McInnes , J. Healy , and J. Melville , “ Umap : Uniform manifold approximation and projection397 for dimension reduction , ” arXiv preprint arXiv:1802.03426 , 2018.398 [ 42 ] I. Dokmanic , R. Parhizkar , J. Ranieri , and M. Vetterli , “ Euclidean distance matrices : essential399 theory , algorithms , and applications , ” IEEE Signal Processing Magazine , vol . 32 , no . 6 , pp.400 12–30 , 2015.401 [ 43 ] A. Bartesaghi , A. Merk , S. Banerjee , D. Matthies , X. Wu , J. L. Milne , and S. Subramaniam , “ 2.2402 å resolution cryo-EM structure of β-galactosidase in complex with a cell-permeant inhibitor , ” 403 Science , vol . 348 , no . 6239 , pp . 1147–1151 , 2015.404 [ 44 ] G. Laxmikanthan , C. Xu , A. F. Brilot , D. Warren , L. Steele , N. Seah , W. Tong , N. Grigorieff,405 A. Landy , and G. D. Van Duyne , “ Structure of a holliday junction complex reveals mechanisms406 governing a highly regulated dna transaction , ” Elife , vol . 5 , p. e14313 , 2016.407 [ 45 ] E. F. Pettersen , T. D. Goddard , C. C. Huang , G. S. Couch , D. M. Greenblatt , E. C. Meng , and408 T. E. Ferrin , “ Ucsf chimera—a visualization system for exploratory research and analysis , ” 409 Journal of Computational Chemistry , vol . 25 , no . 13 , pp . 1605–1612 , 2004.410 [ 46 ] W. van Aarle , W. J. Palenstijn , J . De Beenhouwer , T. Altantzis , S. Bals , K. J. Batenburg , and411 J. Sijbers , “ The ASTRA toolbox : A platform for advanced algorithm development in electron412 tomography , ” Ultramicroscopy , vol . 157 , pp . 35–47 , 2015.413 [ 47 ] C. Sorzano , L. De La Fraga , R. Clackdoyle , and J. Carazo , “ Normalizing projection images:414 A study of image normalizing procedures for single particle three-dimensional electron mi-415 croscopy , ” Ultramicroscopy , vol . 101 , no . 2-4 , pp . 129–138 , 2004.416 [ 48 ] H. Shigematsu and F. Sigworth , “ Noise models and cryo-EM drift correction with a direct-417 electron camera , ” Ultramicroscopy , vol . 131 , pp . 61–69 , 2013.418 [ 49 ] E. Strubell , A. Ganesh , and A. McCallum , “ Energy and policy considerations for419 modern deep learning research , ” Proceedings of the AAAI Conference on Artificial420 Intelligence , vol . 34 , no . 09 , pp . 13 693–13 696 , Apr . 2020 . [ Online ] . Available:421 https : //ojs.aaai.org/index.php/AAAI/article/view/7123422 [ 50 ] T. Tieleman and G. Hinton , “ Lecture 6.5-rmsprop : Divide the gradient by a running average of423 its recent magnitude , ” COURSERA : Neural networks for machine learning , vol . 4 , no . 2 , pp.424 26–31 , 2012.425 [ 51 ] D. P. Kingma and J. Ba , “ Adam : A method for stochastic optimization , ” arXiv preprint426 arXiv:1412.6980 , 2014.427 [ 52 ] H. B. McMahan , D. Golovin , S. Chikkerur , D. Liu , M. Wattenberg , A. M. Hrafnkelsson,428 T. Boulos , J. Kubica , G. Holt , D. Sculley , M. Young , D. Ebner , J. Grady , L. Nie,429 T. Phillips , and E. Davydov , “ Ad click prediction : a view from the trenches , ” in430 Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery431 and data mining - KDD ’ 13 . ACM Press , 2013 , p. 1222 . [ Online ] . Available:432 http : //dl.acm.org/citation.cfm ? doid=2487575.2488200433 Checklist434 1 . For all authors ... 435 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s436 contributions and scope ? [ Yes ] 437 ( b ) Did you describe the limitations of your work ? [ Yes ] See §4.438 ( c ) Did you discuss any potential negative societal impacts of your work ? [ Yes ] We didn ’ t439 identify any potential risk for improving protein imaging . Moreover , our work only440 addresses a small step in a huge pipeline . We however mentioned the environmental441 impact of training our model ( see §4 ) .442 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to443 them ? [ Yes ] 444 2 . If you are including theoretical results ... 445 ( a ) Did you state the full set of assumptions of all theoretical results ? [ N/A ] 446 ( b ) Did you include complete proofs of all theoretical results ? [ N/A ] 447 3 . If you ran experiments ... 448 ( a ) Did you include the code , data , and instructions needed to reproduce the main exper-449 imental results ( either in the supplemental material or as a URL ) ? [ Yes ] We include450 a URL in the Abstract to a git repository that includes code , data , and instructions to451 reproduce our results . Moreover , notebooks and an interactive website are provided to452 further play with the method.453 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they454 were chosen ) ? [ Yes ] See §3.1 ( including Table 1 ) for the preparation of data and how455 they were split . See Appendix C for the hyperparameters.456 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running ex-457 periments multiple times ) ? [ Yes ] When there was variance , e.g. , on Figure 7 and458 Figure 17b.459 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type460 of GPUs , internal cluster , or cloud provider ) ? [ Yes ] See Appendix C.461 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 462 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] We used proteins463 from the publicly available Protein Data Bank ( PDB ) and cited the ones we used , see464 §3.1 . We also used and cited the ASTRA toolbox in the same section.465 ( b ) Did you mention the license of the assets ? [ Yes ] The license of our code is mentioned466 in the README.md and included in a LICENSE.txt file in our git repository . PDB467 data are free of all copyright restrictions and made fully and freely available for both468 non-commercial and commercial use.469 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ Yes ] 470 As a URL in the Abstract.471 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re472 using/curating ? [ N/A ] Our data are proteins.473 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable474 information or offensive content ? [ N/A ] Our data are proteins.475 5 . If you used crowdsourcing or conducted research with human subjects ... 476 ( a ) Did you include the full text of instructions given to participants and screenshots , if477 applicable ? [ N/A ] 478 ( b ) Did you describe any potential participant risks , with links to Institutional Review479 Board ( IRB ) approvals , if applicable ? [ N/A ] 480 ( c ) Did you include the estimated hourly wage paid to participants and the total amount481 spent on participant compensation ? [ N/A ] 482

Summary: This paper uses deep learning to recover the orientations of molecules measured in the context of single particle cyroEM. The proposed method first estimates the pairwise distances between the angles of molecules based on the their projections. This is accomplished by embedding the projections into a feature space that was trained such that the cosine similarity between two features is equal to the distance between the orientations of the two molecules. Next, the authors solved an optimization problem (3) to recover the orientations, up to global ambiguities, from the pairwise distances.