Reconstructing continuous distributions of 3D protein structure from cryo-EM images
Authors: Ellen D. Zhong
Tristan Bepler
Joseph H. Davis
Bonnie Berger
Venue:  (2020)
Keywords: generative models
proteins
3D reconstruction
cryo-EM

Abstract: Cryo-electron microscopy (cryo-EM) is a powerful technique for determining the structure of proteins and other macromolecular complexes at near-atomic resolution. In single particle cryo-EM, the central problem is to reconstruct the 3D structure of a macromolecule from $10^{4-7}$ noisy and randomly oriented 2D projection images. However, the imaged protein complexes may exhibit structural variability, which complicates reconstruction and is typically addressed using discrete clustering approaches that fail to capture the full range of protein dynamics. Here, we introduce a novel method for cryo-EM reconstruction that extends naturally to modeling continuous generative factors of structural heterogeneity. This method encodes structures in Fourier space using coordinate-based deep neural networks, and trains these networks from unlabeled 2D cryo-EM images by combining exact inference over image orientation with variational inference for structural heterogeneity. We demonstrate that the proposed method, termed cryoDRGN, can perform ab-initio reconstruction of 3D protein complexes from simulated and real 2D cryo-EM image data. To our knowledge, cryoDRGN is the first neural network-based approach for cryo-EM reconstruction and the first end-to-end method for directly reconstructing continuous ensembles of protein structures from cryo-EM images.

Published as a conference paper at ICLR 2020
RECONSTRUCTING CONTINUOUS DISTRIBUTIONS OF 3D PROTEIN STRUCTURE FROM CRYO-EM IMAGES
Ellen D. Zhong
Tristan Bepler
Joseph H. Davis* MIT
Bonnie Berger* MIT
MIT zhonge@mit.edu
MIT tbepler pepplrrmitte t.. edu
jhdavi i sêmi edu
babêm edu
ABSTRACT
Cryo-electron microscopy (cryo-EM) is a powerful technique for determining the structure of proteins and other macromolecular complexes at near atoric resolution. In single particle cryo-EM, the central problem reconstruct the 3D structure of a macromolecule from 10--7 noisy and randomly oriented 2D) projection images. However, the imaged protein complexes may exhibit structural variability, which complicates reconstruction and is typically addressed using discrete clustering approaches that fail to capture the full range of protein dynamics. Here, we introduce novel method for cryo-EM reconstruction that extends naturally to modeling continuous generative factors of structural heterogeneity. This method encodes structures in Fourier space using coordinate-basse based deep neural networks, and trains these networks from unlabeled 2D) cryo-EM images by combining exact inference over image orientation with variational inference for structural heterogeneity. We demonstrate that the proposed method, termed cryoDRGN, can perform ab initio reconstruction of 3D protein complexes from simulated and real 2D cryo-EM image data. To our knowledge, cryoDRGN is the first neural network- based approach for cryo-EM reconstruction and the first end-to-end method for directly reconstructing continuous ensembles of protein structures from cryo-EM images.
INTRODUCTION
Cryo-electron microscopy (cryo-EM) is Nobel Prize- Prize-winning technique capable of determining the structure of proteins and macromolecular complexes at near-atomic resolution. single particle cryo-EM experiment, a purified solution of the target protein biomolecular complex is frozen in a thin layer of vitreous ice and imaged at sub-nanometer resolution using an electron microscope. After initial preprocessing and segmentation of the raw data, the dataset typically comprises 10"-7 noisy projection images, Each image contains a separate instance of the molecule, recorded as the molecule's 's electron density integrated along the imaging axis (Figure [ A major bottleneck in cryo-EM structure determination is the computational task of 3D reconstruction, where the goal is to solve the inverse problem of learning the structure, i.e. the 3D electron density volume, which gave rise to the projection images. Unlike classic tomographic reconstruction (e.g. MRI), cryo- EM reconstruction is complicated by the unknown orientation of each copy of the molecule in the ice. Furthermore, cryo-EM reconstruction algorithms must handle challenges such low signal to noise ratio (SNR), unknown in-plane translations, imperfect signal transfer due to microscope optics, and discretization of the measurements. Despite these challenges, continuing advances in hardware and software have enabled structure determination at near-atomic resolution for rigid proteins (Kühlbrandt) (2014); Scheres (2012b); Renaud al. (2018) Li al. (2013)).
extremely
Many proteins and other biomolecules are intrinsically flexible and undergo large conformational changes perform their function. Since each cryo-EM image contains unique instance of the molecule of interest, cryo-EM has the potential to resolve structural heterogeneity, which is experi- mentally infeasible with other structural biology techniques such as X-ray crystallography. However, this heterogeneity poses a substantial challenge for reconstruction each image is no longer of the same structure. Traditional reconstruction algorithms address heterogeneity with discrete clustering
"Corresponding authors

---

Published as a conference paper at ICLR 2020
approaches, however, protein conformations continuous and may be poorly approximated with discrete clusters (Malhotra & Udgaonkar (2016); Nakane et al. (2018).
Here, introduce neural network-based re- construction algorithm that learns a continu- low-dimensional manifold over protein's ' conformational states from unlabeled 2D cryo- EM images. We present end-to-end learning framework for generative model over 3D vol- umes using an image encoder-volume decoder neural network architecture. Extending spatial- VAF, we formulate our decoder as function of 3D Cartesian coordinates and unconstrained la- tent variables representing factors of image vari- ation that we expect to result from protein struc- tural heterogeneity (Bepler et al. (2019)). All inference is performed in Fourier space, which allows us to efficiently relate 2D projections to 3D volumes via the Fourier slice theorem. By formulating our decoder as function of Cartesian coordinates, we can explicitly model the imag- ing operation to disentangle the orientation of the molccule during imaging from intrinsic protein structural heterogeneity. Our learning framework avoids errant local minima in image orientation by optimizing with exact inference over a discretization algorithm. The unconstrained latent variables are trained in the standard variational autoencoder approach. We present results on both real and simulated cryo-EM data.
1047 projection images
3D electron density
Figure I: Cryo-EM reconstruction algorithms tackle the inverse problem of determining the 3D electron density volume from 10*-7 noisy images. Each image is noisy projection of unique instance of the molecule suspended in ice at a random orientation. Algorithms must jointly learn the volume and the orientation of each particle image. Example image from Wong et al. (2014).
SO(3) IR using branch and bound
BACKGROUND AND NOTATION
2.1 IMAGE FORMATION MODEL
Cryo-EM aims to recover a structure of interest V R IR consisting of an electron density at each point in space based on a collection noisy images XI.... produced by projecting (i.e. integrating) the volume in unknown orientation along the imaging axis. Formally, the generation of image X can be modeled as:
X(Tosry)= = g* / V(R t) dr. + noise
r= (Fas Ty.
(I)
where V is the electron density (volume), E SO(3), the 3D rotation group. an unknown orientation of the volume, and (tz, ty, 0) is unknown in-plane translation, corresponding to imperfect centering of the volume within the image. The image signal is convolved with g. the point spread function for the microscope before being corrupted with rrepeency-dependent noise and registered discrete grid of size DxD, where D is the size of the image along one dimension.
The reconstruction problem simplified by the observation that the Fourier transform of 2D projection of V is 2D slice through the origin of V in the Fourier domain, where the slice is perpendicular to the projection direction. This correspondence is known as the Fourier slice theorem (Bracewell 1956)). In the Fourier domain, the generative process for image X from volume V can thus be written:
Ake (S(t))(R)) +e
where 9 Fg is the contrast transfer function (CTF) the microscope, 5(t) is phase shift operator corresponding to image translation by t in real space, and A(R) operator corresponding to rotation by R and linear projection along the z-axis in real space. The frequency-dee deepenncydeepeddent noise is typically modelled as independent, zero-centered Gaussian noise in Fourier space. Under this model, the probability of of observing image X with pose ó (R, from volume 1/ is thus:
(+):, is a linear slice
2

---

Published as a conference paper at ICLR 2020
1 exp Σ -
p -)
where / two-component index over Fourier coefficients for the image, or is the width of the Gaussian noise expected at each frequency, and Z is normalization constant.
2.2 TRADITIONAL CRYO-EM RECONSTRUCTION
To recover the desired structure, cryo-EM reconstruction methods must jointly solve for the unknown volume V and image poses Φ (Ra,44 Expectation maximization (Scheres 2012a) and simpler variants of coordinate ascent are typically employed to find a maximum a posteriori estimate of V marginalizing over the posterior distribution of di i.e.:
N
log / p(X:lo, Vi(())o + logp(V)
MAP arg max
(4)
Intuitively, given V(n), the estimate of the volume at iteration images first aligned with V(n) (E- step), then with the updated alignments, the images are backprojected to yield V(++1) (M-step). This
iterative refinement procedure sensitive to the initial estimate of V as the optimization objective is highly nonconvex; stochastic gradient descent is commonly used for ab initio reconstruction to provide an initial estimate ((()) (Punjani et al. (017)).
Given sample heterogeneity, the standard approach in the cryo-EM field is to simultaneously recon- struct K independent volumes. Termed multiclass refinement, the image formation model is extended to assume images are generated from Vi...., VK independent volumes, with inference now requiring marginalization over o; 's and class assignment probabilities T. 's:
TJ ((X.))) (
arg MaX VL Vi.... VK
log
+ Σ logg(Vy)
(5)
j=1
While this formulation is sufficiently descriptive when the structural heterogeneity consists of small number of discrete conformations, it suffers when the heterogeneity complex or when conformations lie along a continuum of states. In practice, resolving such heterogeneity is handled through hierarchical approach refining subsets of the imaging dataset with manual choices for the number of classes and the initial models for refinement. Because the number and nature of the underlying structural states are unknown, multiclass refinement error- prone, and in general, the identification and analysis of heterogeneity is an open problem in single particle cryo-EM.
METHODS
We propose neural network-based reconstruction method, cryoDRGN (Deep Reconstructing Gener- ative Networks), that can perform ab-initio unsupervised reconstruction of a continuous distribution 3D volumes from unlabeled 2D images (Figure 2 We formulate image encoder-volume decoder architecture based on the variational autoencoder (VAE) (Kingma & Welling (2013). where protein structural heterogeneity modeled in the latent variable. While standard VAE assumes all sources of image heterogeneity are entangled in the latent variable, we propose an architecture that enables modelling the intrinsic heterogeneity of the volume separately from the extrinsic orientation of the volume during imaging. Our end-to-ent training framework explicitly models the forward image formation process to relate 2D views to 3D volumes and employs two separate strategies for inference: variational approach for the unconstrained latent variables and global search over (O(3) x RR² for the unknown pose of each image. These elements are described in further detail below.
"Reconstruction" is used interchangeably in the cryo-EM literature to refer to either the full pipeline from ab-initio model generation followed by iterative refinement of the model via expectation maximization solely the latter. We focus on the former case.

---

Published as a conference paper at ICLR 2020
3.1 GENERATIVE MODEL
We design a deep generative model to approximate a single function, V R - IR, representing a n n-dimensional manifold of 3D electron densities in the Fourier domain. Specifically, the volume V is modelled as probabilistic decoder po [V]k, 2), where 6 are parameters of multilayer perceptron (MLP). Given Cartesian coordinates & I and continuous latent variable 2, the decoder outputs distribution parameters for Gaussian distribution over (kk, =), :.),i... i.e. the electron density of volume V= at frequency l in Fourier space. Unlike a standard deconvolutional decoder which produces a separate distribution for each voxel of a D lattice given the latent variable, following spatial spatial-VAE, we model a function over Cartesian coordinates (Bepler et al.] (2019).. Here, these coordinates explicitly treated each pixel's location in 3D Fourier space and thus enforce the topological constraints between 2D views in 3D via the Fourier slice theorem.
By the image formation model, each image corresponds to an oriented central slice of the 3D volume in the Fourier domain (Section 2). During training, the 3D coordinates of an image's 's pixels can be explicitly represented by the rotation of DxD lattice initially on the x-y plane. Under this model, the log probability of an image, X, represented as a vector of size DxD, given the current MI.,, latent pose variables R E S3(3) and i E TR? and unconstrained latent variable, 2, is:
log logp(XX|R,t t, = log.olXXXXi p(X'((((R))) Logg log po (VI c 22)
(6)
where i indexes over the coordinates of a fixed lattice co. Note that X = 5(-1) is the centered image, where S is the phase shift operator corresponding to image translation in real space. We define co as vector of 3D coordinates of a fixed lattice spanning [-0.5, 0.5) on the x-y plane to represent the unoriented coordinates of an image's pixels.
Instead of directly supplying k, fixed positional encoding of k is supplied to the decoder, consisting of sine and cosine waves of varying frequency:
pe (2 (kz) = sin(k;DD(2/DD) 2iD), i i= D/2;; E k
pe (2++)) D2 i 1,... D/2; E k
(8)
Without loss of generality, we assume a length scale by our definition of co which restricts the support of the volume sphere of radius 0.5. The wavelengths of the positional encoding thus follow a geometric series spanning the Fourier basis from wavelength to the Nyquist limit (2/ D) of the image data. While this encoding empirically works well for noiseless data, we obtain better results with slightly modified featurization for noisy datasets consisting of geometric series which excludes the top 10 percentile of highest frequency components of the noiseless positional encoding.
3.2 INFERENCE
We employ standard VAE for approximate inference of the latent variable 2, but global search to infer the pose & = (R, 1) using a branch and bound algorithm.
Variational encoder: As each cryo-EM image is noisy projection of instance of the volume at random, unknown pose (viewing direction). the image encoder aims to learn a pose-imvarians representation of the protein's 's structural heterogeneity. Following the standard VAE framework, the probabilistic encoder qe [x]x is a MLP with variational parameters and Gaussian output with diagonal covariance, Given an input cryo-EM image X, represented as a DxD vector, the encoder MLP outputs 1z x and E Σzlx- statistics that parameterize approximate posterior to the intractable true posterior p x) The prior
is standard normal, ().
Pose inference: We perform a global search over SO(3) x IR² for the -ixiimum-likelhoodd pose for each image given the current decoder MLP and a sampled value of from the approximate posterior. Two techniques are used to improve the efficiency of the search over poses: (1) discretizing the search space on uniform grid and sub-dividing grid points after pruning candidate poses with
4

---

Published as a conference paper at ICLR 2020
Decoder
V2(k)
R
Posttional crooding
Input DaD imags (Founer spece)
Reconstracted
R'U,J0)"
DrD lattice
xry
Figure 2: CryoDRGN model architecture. We use a VAE to perform approximate inference latent variable & denoting image heterogeneity. The decoder reconstructs an image pixel by pixel given x and pe(k), the positional encoding of 3D Cartesian coordinates. The 3D coordinates corresponding to each image pixel are obtained by rotating DxD lattice on the x-y plane by R. the image orientation. The latent orientation for each image is inferred through branch and bound glohal optimization procedure (not shown).
branch and bound (BNB), and (2) band pass limiting the objective to low frequency components and incrementally increasing the k-space limit at each iteration (frequency marching). The pose inference procedure encodes the intuition that low-frequency components dominate pose estimation, and is fully described in Appendix A.
In summary, for a given image X,, the image encoder produces M=|8; and = A sampled value of the latent ~ (1 xx)x,) is broadcast to all pixels. Given and the current decoder, BNB orientational search identifies the maximum likelihood rotation R; and translation fę for X, The decoder po then reconstructs the image pixel by pixel given the positional encoding of R. co and zi. The phase shift corresponding to &; and optionally the microscope CTF ğ; is then applied on the reconstructed pixel intensities. Following the standard VAE framework, the optimization objective the variational lower bound of the model evidence:
C(X;: () = E ,x (log po (X.,22]
(9)
where the expectation of the log likelihood is estimated with one Monte Carlo sample. By comparing many 2D slices from the imaging dataset, the volume can be learned through feedback from these single views. Furthermore, this learning process is denoising as overfitting to noise from a single image would lead to higher reconstruction error for other views. We note that the distribution of 3D volumes models heterogeneity within a single imaging dataset, capturing structural variation for a particular protein or biomolecular complex, and that separate network is trained per experimental dataset. Unless otherwise specified, the encoder and decoder networks are both MLPs containing 10 hidden layers of dimension 128 with ReLU activations. Further architecture and implementation details are given in Appendix A.
4 RELATED WORK
Homogeneous cryo-EM reconstruction: Cryo-EM reconstruction is typically accomplished in two stages: generation of initial low-resolution model followed by 2) iterative refinement of the initial model with a coordinate ascent procedure alternating between projection matching and refinement of the structure. practice, initial structures can be obtained experimentally (Leschziner & Nogales (2006)), inferred based on homology to complexes with known structure, or via ab-initio reconstruction with stochastic gradient descent (Punjani et al. (2017)) Once an initial model is generated, there are many tools for iterative refinement of the model (Scheres (2012b); PPnjaani et al. (2017): Hohn et al. (2007); Lyumkis,, Dmitry et al. (2013); Tang et al. (2007)). For example, [Scheres) (2012a) presents a Bayesian approach based on a probabilistic model of the image formation process and refines the structure via Expectation Maximization. Frequency marching is used extensively in existing tools to speed up the search for the optimal pose for each image (Scheres (2012b); [Banntt jet al.] (2016): Punjani et al. (2017)). CryoSPARC implements a branch and bound optimization scheme, where their bound probabilistic lower bound based on the noise characteristies from the
5

---

Published as a conference paper at ICLR 2020
image formation model (Punjani et al. (2017)) Ullrich al. (2019) propose differentiable voxel- based representation for the volume and introduce a variational inference algorithm for homogeneous reconstruction with known poses.
Heterogeneous cryo-EM reconstruction: In the cryo-EM literature, standard approaches for ad- dressing structural heterogeneity mixture models of discrete, independent volumes, termed multiclass refinement (200); [2010);Lyykki, Lyumkis, Dmitry et al. (2013). These mixture models assume that the clusters are independent and homogeneous, and in practice require many rounds of expert- guided hierarchical clustering from appropriate initial volumes and manual choices for number of clusters. More recently, Nakane et al. (2018) extend the image generative model to model the protein as sum of rigid bodies (determined from a homogeneous reconstruction). thus imposing structural assumptions on the type of heterogeneity. Frank Ourmazd (2016) aim to build a continuous manifold of the images, however their approach requires pose supervision and final structures are obtained by clustering the images along the manifold and reconstructing with traditional tools. Recent theoretical work for continuous heterogeneous reconstruction includes expansion of discrete 3D volumes in a basis of Laplacian eigenvectors (Moscovich et al. (2019) and a general framework for modelling hyper-volumes (Lederman et al. (2019) e.g. tensor product of spatial and temporal basis functions (Lederman & Singer (2017)) To our knowledge, our work is the first to apply deep neural networks cryo-EM reconstruction, and in doing so, is the first that can learn continually heterogeneous volume from real cryo-EM data.
Neural network 3D reconstruction in computer vision: There is large body of work in computer vision on 3D object reconstruction from 2D viewpoints. While these general approaches have elements in common with single particle cryo-EM reconstruction, the problem in the context of computer vision differs substantially in that 2D viewpoints are not projections and viewing directions are typically known. For example. Yan et al. (2016) propose a neural network that can predict a 3D volume from single 2D viewpoint using only 2D image supervision. Gadelha et al. (2017) learn a generative model over 3D object shapes based on 2D) images of the objects thereby disentangling variation in shape and pose. Tulsiani et al. (2018) also reconstruct and disentangle the shape and pose of 3D objects from 2D images by enforcing geometric consistency. These works attempt to encode the viewpoint 'projection' operation explicitly in the model in a manner similar to our use of the Fourier slice theorem.
Using spatial (i.e. pixel) coordinates as features to a convolutional decoder to improve generative modeling has been proposed many times, with recent work computing each image as a function of fixed coordinate lattice and latent variables (Watters et (019)). However, directly modeling a function that maps spatial coordinates to values less extensively explored. In CocoNet, the authors present deep neural network that maps 2D) pixel coordinates to RBG color values. CocoNet learns an image model for single images, using the capacity of the network memorize the image, which can then be used for various tasks such as denoising and upsampling (Bricman & Ionescu (208)). Similarly, Spatial- proposes a similar coordinate-bais based image model enforce geometric consistency between rotated 2D images in order to learn latent image factors and disentangle positional information from image content (Bepler et al. (2019)). Our method extends many of these ideas from simpler 2D image modelling to enable 3D cryo-EM reconstruction in the Fourier domain.
Coordinate-basse based neural networks in computer vision:
RESULTS
Here, we present both qualitative and quantitative results for 1) homogeneous cryo-EM reconstruction. validating that cryoDRGN reconstructed volumes match those from existing tools; 2) heterogeneous cryo-EM reconstruction with pose supervision, demonstrating automatic learning of the latent manifold that previously required many sxpert-guided rounds of multiclass refinement; and 3) fully unsupervised reconstruction of continuous distributions of 3D protein structures, a capability not provided by any existing tool.
This is not the meaning of projection in the context of this work, where it refers to integration along the imaging axis.

---

Published as a conference paper at ICLR 2020
5.1 UNSUPERVISED HOMOGENEOUS RECONSTRUCTION
We first evaluate cryoDRGN on homogeneous datasets, where existing tools are capable of recon- struction. We create two synthetic datasets following the cryo-EM image formation model (image size D=128, 50k projections, with and without noise), and use one real dataset from EMPIAR- 1-028 consisting of 105, 25,247 images of the 80S ribosome downsampled to image size D=90. The encoder network is not used in homogeneous reconstruction. As a baseline for comparison, we perform homogeneous ab-initio reconstruction followed by iterative refinement in cryoSPARC (Punjani et al. (2017)) We compare against cryoSPARC as a representative of traditional state-of-the-ad tools, which all implement variants of the same algorithm (Section 2). Further dataset preprocessing and training details are given in Appendix B.
Dataset
We find that cryoDRGN inferred poses and re- constructed volumes match those from state of- the-art tools. The similarity of the volumes to the ground truth can be quantified with the with the Fourier shell correlation (FSC) curve Re- constructed volumes and quantitative compar- ison with the FSC curve is given Figure S5. Pose error to the ground truth image poses are given T For the real cryoEM dataset (no ground truth), the median pose difference between cryoDRGN and cryoSPARC: reconstructions is 0.002 for rotations and 1.0 pixels for translations, and the resulting volumes are correlated above FSC cutoff of 0.5 across all frequencies.
Method
No Noise
SNR=0.1
cryoSPARC
0.0009
0.47
0.002
0.64
cryoDRGN
0.0004 0.27
0.003 0.38
Table 1: Homogeneous reconstruction pose accuracy quantified by median rotation/rranslation error the ground truth image poses. Rotation/taanlation defined as the Frobenius/L2 norm after alignment.
5.2 HETEROGENEOLS RECONSTRUCTION WITH POSE SUPERVISION
Next, we evaluate cryoDRGN for heterogeneous cryo-EM reconstruction on EMPIAR- 10076,
real dataset of the E. coli large ribosomal subunit (LSU) undergoing assembly (131,899 images, downsampled to D=90) (Davis et al. (2016)). Here, poses are obtained through alignment to an existing structure of the LSU and treated as known during training. In the original analysis of this dataset, multiple rounds of discrete multiclass refinement with varying number of classes followed by human comparison of similar volumes were used to identify 4 major structural states of the LSU. We train cryoDRGN with a 1-D latent variable treating image pose as fixed to skip BNB pose inference. As baseline, we reproduce the published structures originally obtained through multiclass refinement with CPRRS Further baseline and training details are given in Appendix C.
We find that CryoDRGN automatically identi- fies all 4 major states of the LSU (Figure 3a). Quantitative comparison with FSC curves and additional volumes along the latent space are shown Figure S7. We compare the cryo- DRGN latent encoding 2z|X for each image to the MAP cluster assignment in cryoSPARC and find that the learned latent manifold aligns with cryoSPARC clusters (Figure T)). Cryo- DRGN identifies subpopulations in some of the cryoSPARC clusters (e.g. Class D), which is par- titioned by subsequent round of cryoSPARC multiclass refinement (Figure S8). Published structures A and F correspond impurities in the sample. CryoDRGN correctly assigns im- ages from these impurities to distinct clusters, but does not learn their correct structure since the poses inferred from aligning the LSU tem- plate structure are incorrect.
a)
cryoDRGN
cryoSPARC
3000
4000
1000
-3
-2
Iatant encoding,
Figure 3: a) Volumes generated at values the latent (at dashed lines) match the published volumes of the major states B-E of the LSU. b) Distribution of images the latent space, colored by cluster assignment from discrete multiclass reconstruction in cryoSPARC.
'The FSC curve measures correlation between volumes as a function of radial shells in Fourier space. The field currently lacks rigorous method for measuring the quality of reconstruction. In practice, however, resolution is often reported 1/ko where ky arg max, FSC(k) < C and C some fixed threshold.
7

---

Published as a conference paper at ICLR 2020
Ground truth
cryoDRGN
STYOSPARC
2ouume vdlume
Figure Left: Ground truth volume containing a continuous circular motion. Middle: Reconstructed structures from cryoDRGN match the ground truth volumes with the correct continuous deformation. We visualize 10 structures (superimposed) sampled at the depicted points in the latent space. The distribution of images in the latent space (visualized in 2D with PCA) matches the topology of the true data manifold. Right: Reconstructed volumes from discrete 3-class reconstruction in cryoSPARC and the distribution of images over the three reconstructed volumes.
Dataset
cryoDRGN
cryoDRGN+tilt
cryoSPARC
Linear ID motion
2.50(0.62)
2.35(0.36)
3.60(2.27)
Linear 2D motion
4.44(2.50)
2.93(1.02)
6.90(3.77)
Circular ID motion
4.05(2.40)
2.63(0.74)
4.87(2.17)
Discrete 10 class
4.95(3.16)
2.58(1.00)
5.69(5.15)
Table 2: Reconstruction accuracy quantified by an FSC=0.5 resolution metric between the reconstructed volumes corresponding to each image and its ground truth volume. We report the average and standard deviation across 100 images in the dataset (lower better; best possible is 2 pixels).
5.3 UNSUPERVISED HETEROGENEOUS RECONSTRUCTION
We test the ability of cryoDRGN to perform fully unsupervised heterogeneous reconstruction from datasets with different latent structure. We generate four datasets (each 50k projections, D=64) from an atomic model of a protein complex, containing either a ID continuous motion, 2D continuous motion, 1D continuous circular motion, mixture of 10 discrete conformations (Figure S7). We train cryoDRGN with a 1D latent variable for the linear 1D dataset and a 10D latent variable for the other datasets. As baseline, we perform multiclass reconstruction in cryoSPARC sweeping K=2-5 classes. We compare against K=3, which had the best qualitative results.
We also propose modification to cryoDRGN in order train on tilt series pairs datasets. Tilt series pairs is variant of cryo-EM which, for each image X,, a corresponding image X; is acquired after tilting the imaging stage by known angle. This technique was originally employed to identify the chirality of molecules (Belnap al. (1997), which is lost the projection from 3D 2D. We propose using tilt series pairs to encourage invariance of g with respect to pose transformations for given V. (and incidentally identify the chirality of V2). We make minor modifications to the architecture as described in Appendix D.
In Figure 4 show that cryoDRGN reconstructed volumes for the circular 1D dataset qualitatively match the ground truth structures. Note that while we only visualize 10 structures sampled along the latent space, the volume decoder can reconstruct the full continuum of states. In contrast, cryoSPARC multiclass reconstruction, discrete mixture model of independent structures, is only able to reconstruct (originally unaligned) structures which resemble the ground truth. Volumes contain blurring artifacts from clustering images from different conformations into the assumed- homogeneous clusters in the mixture model. Results for the remaining datasets are given in Figures S10-13.
We quantitatively measure performance on this task with an FSC resolution metric computed between the MAP volume for each image V V= and the ground truth volume which generated each image, averaged across images in the dataset (Table (S4). We find that cryoDRGN reconstruction accuracy much higher than state-of-the-ar discrete multiclass reconstruction in cryoSPARC, with further improvement achieved by training tilt series pairs.
8

---

Published as a conference paper at ICLR 2020
6 CONCLUSIONS
We present a novel neural network-based reconstruction method for single particle cryo-EM that learns continuous variation in protein structure. We applied eryoDRGN real dataset of highly heterogeneous ribosome assembly intermediates and demonstrate automatic partitioning of structural states. In the presence of simulated continuous heterogeneity, continuous representation of structure along the true reaction coordinate, effectively disentangling imaging orientation from intrinsic structural heterogeneity. The techniques described here may also have broader upplicability to image and volume generative modelling in other domains of computer vision and shape reconstruction.
show that cryoDRGN learns
ACKNOWLEDGMENTS
We thank Ben Demeo, Ashwin Narayan, Adam Lever, Roy Lederman, and Kotaro Kelley for helpful discussions and feedback. This work was funded by the National Science Foundation Graduate Research Fellowship Program, NIH grant R01-GM08 1871, NIH grant R00-AG050749, and the MIT J-Clinic for Machine Learning and Health.
REFERENCES
Alex Barnett, Leslie Greengard, Andras Pataki, and Marina Spivak. Rapid solution of the cryo-EM reconstruction problem by frequency marching. arXiv.org, October 2016.
William T Baxter, Robert A Grassucci, Haixiao Gao, and Joachim Frank. Determination of signal-to-
noise ratios and spectral SNRs in cryo-EM low-dose imaging of molecules. Journal of structural biology, 166(2): 132, May 2009.
D M Belnap, N H Olson, and T S Baker. A method for establishing the handedness of biological macromolecules. Journal of structural biology, 120(1):44-55 October 1997.
Tristan Bepler, Ellen Zhong, Kotaro Kelley, Edward Brignole, and Bonnie Berger. Explicitly disentangling image content from rotation and translation with spatial-VAB. Neural Informational Processing Systems (NeurIPS), 2019.
Ronald N Bracewell. Strip integration in radio astronomy. Australian Journal of Physics, 9(2): 198-217, 1956.
Paul Andrei Brieman and Radu Tudor lonescu. CocoNet: A deep neural network for mapping pixel coordinates to color values. arXiv.org, May 2018.
Joseph Davis, Yong Zi Tan, Bridget Carragher, Clinton S Potter, Dmitry Lyumkis, and James R Williamson. Modular Assembly of the Bacterial Large Ribosomal Subunit. Cell, 167(6):1610- 1622.e15, December 2016.
Joachim Frank and Abbas Ourmazd. Continuous changes in structure mapped by manifold embedding of single-partin particle data in cryo-EM. Methods (San Diego, Calif.), 100:61-61 -67, May 2016.
Matheus Gadelha, Subhransu Maji, and Rui Wang. 3d shape induction from 2d views of multiple objects. In 2017 International Conference on 3D Vision (3DV), pp. 402-411, IEEE, 2017.
Krzysztof M Gorski, Eric Hivon, Anthony J Banday, Benjamin D Wandelt, Frode K Hansen, Mstvos Reinecke, and Matthia Bartelmann. Healpix: a framework for high-resolution discretization and fast analysis of data distributed on the sphere. The Astrophysical Journal. 622(2):759, 2005.
Ralph VL Hartley, more symmetrical fourier analysis applied transmission problems. Proceed- ings of the IRE, 30(3):144-1500 1942.
Michael Hohn, Grant Tang, Grant Goodyear, P R Baldwin, Zhong Huang, Pawel A Penczek, Chao Yang, Robert M Glaeser, Paul D Adams, and Steven J Ludtke. SPARX, new environment for
Cryo-EM image processing. Journal of structural biology, 157(1):47-55, January 2007.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.

---

Published as a conference paper at ICLR 2020
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In The 2nd International Conference on Learning Representations (ICLR), 2013.
Werner Kühlbrandt. Cryo-EM enters a new era. eLife, 3te03678, August 2014.
Roy R Lederman and Amit Singer. Continuously heterogeneous hyper-objects in cryo-EM and 3-D movies of many temporal dimensions. arXiv.org, April 2017.
Roy R Lederman, Joakim Andén, and Amit Singer. Molecules: on the Representation and Recovery of Dynamical Structures, with Application to Flexible Macro-Molecular Structures in Cryo-EM. arXiv.org, July 2019.
Andres E Leschziner and Eva Nogales. The orthogonal tilt reconstruction method: an approach to generating single-class volumes with no missing cone for ab initio reconstruction of asymmetric particles. Journal of structural biology, 153(3):844-299, 2006.
Xueming Li, Paul Mooney, Shawn Zheng, Christopher R Booth, Michael B Braunfeld, Sander Gubbens, David A Agard, and Yifan Cheng. Electron counting and beam-induced motion correction enable sear-atomic-reoolle 2013.
single-partie cryo-EM. Nature methods, (0(6):584-590, June
Lyumkis. Dmitry, Brilot, Axel F, Theobald, Douglas L, and Grigorieff, Nikolaus. Likelihood- based classification of cryo-EM images using FREALIGN. Journal of structural biology, 183(3): 377-388, September 2013.
Pooja Malhotra and Jayant B Udgaonkar. How cooperative are protein folding and unfolding transitions? Protein science: publication of the Protein Society, 25(11):1924-194 November 2016.
Amit Moscovich, Amit Halevi, Joakim Andén, and Amit Singer. Cryo-EM reconstruction of continuous heterogeneity by Laplacian spectral volumes. arXiv.org, July 2019.
Takanori Nakane, Dari Kimanius, Erik Lindahl, and Sjors Hw Scheres. Characterisation of molecular motions in cryo-EM paglecp data by multi-body refinement in RELION. eLife, 7:e36861,
June 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Veto,o, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Ali Punjani, John L. Rubinstein, David Fleet, and Marcus A Brubaker. cryoSPARC: algorithms for rapid unsupervised cryo-EM structure determination. Nature methods, 14(3):290-22) 296, March 2017.
Jean-Paul Renaud, Ashwin Chari, Claudio Ciferri, Wen-Ti Liu, Hervé- Wille--iliiam Rémigy, Holger Stark, and Christian Wiesmann. Cryo-EM drug discovery: achievements, limitations and prospects. Nature reviews. Drug discovery, 777))471-492, July 2018.
Sjors H W Scheres. A Bayesian view cryo-EM strueture determination. Journal of molecular biology, 415(():406-414 448444466188, January 2012a.
Sjors H W Scheres. RELION: implementation of Bayesian approach to cryo-EM structure determi-
nation. Journal of structural biology, 180(3):51--530, December 2012b.
Sjors HW Scheres. Maximum-likelihowe methods in cryo-em, part ii: Application experimental data. Methods in enzymology, 482:295, 2010.
Guang Tang, Liwei Peng, Philip R Baldwin, Deepinder S Mann, Wen Jiang, Ian Rees, and Steven J Ludtke. FMAN2: an extensible image processing suite for electron microscopy. Journal of structural biology, 157(1):38-44 46, January 2007.
Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik. Multi-view consistency supervisory signal for learning shape and pose prediction. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
10

---

Published as a conference paper at ICLR 2020
Karen Ullrich, Rianne den Berg, Marcus Brubaker, David Fleet, and Max Welling. Differentiable probabilistic models of scientific imaging with the Fourier slice theorem. arXiv.org. June 2019.
Nicholas Watters, Loic Matthey, Christopher Burgess, and Alexander Lerchner. Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VABs, arXiv.org, January 2019.
Wilson Wong, Xiao-Chen Bai, Alan Brown, Israel S Fernandez, Eric Hanssen, Melanie Condron, Yan Hong Tan, Jake Baum, and Sjors H W Scheres. Cryo-EM structure of the Plasmodium falciparum 80S ribosome bound to the anti-protozoan drug emetine. eLife, 3:e01963, June 2014.
Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective Transformer Nets: Learning Single-Vies View 3D Object Reconstruction without 3D Supervision. arXiv.org, December 2016.
Anna Yershova, Swati Jain, Steven M Valle, and Julie C Mitchell. Generating Uniform Incremental Grids on SO(3) Using the Hopf Fibration. The International Journal of Robotics Research, 29(7): 801-812, May 2010.

---

Published as a conference paper at ICLR 2020
A APPENDIX METHODS
A.1 BRANCH AND BOUND IMPLEMENTATION DETAILS
We perform a global search over SO(3) x R for the maximum-likellenn pose for each image given the current decoder MLP. Two techniques are used to improve the efficiency of the search over poses: (1) discretizing the search space on uniform grid and sub-dividing grid points after pruning candidate poses with branch and bound, and (2) band pass limiting the objective to low frequency components and incremenaally increasing the k-space limit at each iteration (frequency marching).
Our branch and bound algorithm for pose optimization given in Algorithm 1. Briefty, we discretize SO(3) uniformly using the Hopf fibration Yershova et al. (210)) a predefined base resolution of the grid and incrementally increase the grid resolution by sub-dividing grid points. At each resolution of the grid, the of candidate poses pruned using branch and bound (BNB) optimization scheme, which alternates between a computationally inexpensive lower bound on the objective function evaluated at all grid points and an upper bound consisting of the true objective evaluated on the best lower-bound candidate. Grid points whose lower bound is higher than this value are excluded for subsequent iterations. In our case, the loss is evaluated on low-frequency components of the image; specifically, Fourier components with |k| < kmaz is an effective lower bound, as it is both inexpensive to compute and captures most the power (and thus the error). This bound encodes the intuition that now-frequency components dominate pose estimation. We concomitantly increase kmaz at each iteration of grid subdivision.
At each iteration, some poses excluded by BNB, and the remaining poses are further discretized. Although BNB is risk-free in the sense that the optimal pose at a given resolution will not be pruned, application of it is not risk-free as candidate pose with high loss at given resolution doesn't t guarantee that its neighbor in the next iteration will not have a lower loss. Irrespective, in practice, we find that at sufficiently fine base resolution, we obtain good results on tractable timescale (hours on a single GPU)
We reimplement the uniform multiresolution grids SO(3) based on Yershova al. (2010), using the Healpix Gorski et al. (2005) grid for the sphere and the Hopf fibration to uniformly lift the grid to SO(3). The base grid on (3)3) contains 576 orientations. We use the ordinary grid for translations containing 72 points with an extent of 20 pixels for D=128 datasets. We subdivide the grid 5 times for final resolution of 0.92 degrees for the orientation and 0.08 pixels for the translation. For D=64 datasets, we use translational grid with extent of 10 pixels.
Algorithm CryoDRGN branch and bound with frequency marching
procedure OPTPHI(X, V/2)
Find the optimal image pose given the current decoder
2: kmin &- 12, kmar &- D/2, Niter 5
3: T P==50m 50(3)= x IR22 grid base resolution
4:
k - Kmin
5: for iter = 1 ... Niter do
6:
for Φ; Φ do
> Compute lower bound all grid points
7:
Ib(():) loss between X and SLICE(V, de) at k < k
8:
6² + arg min
9:
uh < loss between X and SLICE(V, o²) (6*)aa at < kmaz T Fnew < 0
D. Compute upper bound
10:
11:
for o; E Φ do
Subdivide grid points below the upper bound
if (b(::)) < ub then
13:
Pnew + Inew U SUBDIVIDE(())
14:
T Tb Unew
15:
4- 1: + (kiman kmin)/(Nat - 1)
b Increase frequency band limit
16:
return 0"
'The difference in loss between nearby poses could be incorporated into the BNB lower bound, but this would require assumptions about the smoothness of the loss with respect to pose, We leave this detail for future work.
12

---

Published as a conference paper at ICLR 2020
A.2 TRAINING DETAILS
Given imaging dataset, XI....NN, summarize three training paradigms of cryoDRGN. 1) For homogeneous reconstruction, we only train the volume decoder Pe and perform BNB pose inference for the unknown Φ; 's for each image. As an intermediate task, we can perform heterogeneous reconstruction training the image encoder qe and the volume decoder po with known o; to skip BNB pose inference. 3) For fully unsupervised heterogeneous reconstruction, we jointly train q and pe to learn continuous latent representation, performing BNB pose inference for the unknown pose of each image.
Unless otherwise specified, the encoder and decoder networks are both MLPs containing 10 hidden layers dimension 128 with ReLU activations. fully connected architecture is used instead of convolutional architecture because the images are not represented in real space.
Instead of representing both the real and imaginary components of each image, we use the closely- related Hartley space representation (Hartley (1942). The Hartley transform of real-valued functions equivalent to the real minus imaginary component of the FT, and thus is real valued. The Fourier slice theorem still holds and the error model is equivalent.
In this work, we simplify the image generation model to Gaussian white noise. Therefore, for a given image, the negative log likelihood for reconstructed slice from the decoder corresponds to the mean squared error between the phase-shifted image and the oriented slice from the volume decoder. We leave the implementation of colored noise model to future work.
We use the Adam optimizer (Kingma & Ba (2014)) with learning rate of 5e-4 for experiments involving noiseless, homogeneous datasets, and 1e-4 for all other experiments. All models implemented in Pytorch (Paszke et al. (2017))
B HOMOGENEOUS RECONSTRUCTION
B. DATASET PREPARATION
Simulated datasets: From ground truth 3D volume, we simulated datasets following the cryo-EM image formation model by 1) rotating the 3D volume in real space by R. where R E SO(3) is sampled uniformly, 2) projecting (integrating) the volume along the z-axis, 3) shifting the resulting 2D image by f, where & is sampled uniformly from [0 10 pixels, and 4) optionally adding noise to an SNR of 0.1, a typical value for cryo-EM data (Baxter et al. (2009). Following convention the cryo-EM field, we define SNR the ratio of the variance of the signal to the variance of the noise. We define the noise-free signal images to be the entire DxD image. 50k projections were generated for each dataset with image size of D=128.
Real dataset: To generate the real cryo-EM dataset for homogeneous reconstruction, images from AMPIAR-10022 (Wong et al. (014)) were downsampled by factor of by clipping in Fourier space. The images were then phase flipped' in Fourier space by their contrast transfer function, a given real-valued function with range [-1,1] determined by the microscopy conditions, i.e. the Fourier components are negated where the CTF is negative.
B.2 TRAINING
For each dataset, we train the volume decoder (10 hidden layers of dimension 128) minibatches of 10 images with random orientations for the first epoch to learn a volume with roughly correct spatial extent, followed by 4 epochs with branch and bound (BNB) pose inference (30 min/epoch noiseless, 80 min/noisy datasets). Since BNB pose inference is the bottleneck during training, we employ multiscale training protocol, where after 4 epochs with BNB pose inference, the latent pose is fixed, and we train a separate, larger volume decoder (10 hidden layers of dimension 500) for 15 epochs with fixed poses to "refine" the structure to high resolution (20 min/epoch). Training times reported for 50k, D=128 image datasets trained Nvidia Titan V GPU.
13

---

Published as a conference paper at ICLR 2020
B.3 SUPPLEMENTARY RESULTS
Ground truth volume
Example image
cryoDRGN
cryoSPARC
FSC to ground truth
No noise
09-
50S ribosome (synthetic)
SOS ribosome (synthetic) noise
3.0
0.4
froguorr
SNR-0.1
80S
0.1-
ribosome (real)
Figure S5: Left: CryoDRGN unsupervised homogeneous reconstruction on 2 simulated datasets and real cryo- EM dataset matches state-of-the-art. Right: Fourier shell correlation (FSC) curves between the reconstructed volume and the ground truth volume for the synthetic ribosome datasets.
C HETEROGENEOUS RIBOSOME RECONSTRUCTION WITH POSE SUPERVISION
Dataset preparation: We used the dataset from EMPIAR-10076 which contains 139,899 images of the coli large ribosomal subunit (LSUU) in various stages of assembly (Davis et al. (2016)) Images were downsampled to D2128 by clipping in Fourier space. Poses were determined by aligning the images to a mature LSU structure obtained from homogeneous reconstruction of the full resolution dataset in cryoSPARC, i.e. "a consensus reconstruction".
In the original analysis of this dataset, multiple rounds of multiclass refinement in sweeps of varying number of classes followed by expert manual alignment and clustering of similar volumes were used to identify classes, labeled A-F consisting of 4 major structural states of the LSU (classes B-E) and 2 additional structures of the 70S and 30S ribosome, class A and F, respectively.
Baseline:
Since the published dataset did not contain the corresponding image cluster assignments, we perform multiclass refinement cryoSPARC using the published structures of the 6 major states, low pass filtered to 2 as initial models, to reproduce the results and obtain image cluster assignments. Aside from class and F (low population impurities in the sample), the remaining structures correlate well with the published volumes (Figure (6).
D
E
FSC published volumes
published
eryoSPARC
Figure S6: Reconstructed volumes from cryoSPARC multiclass refinement using the published structures of major states, low pass filtered to A5Aas initial models. Right: FSC curves between the cryoSPARC reconstructed and published volumes.
14

---

Published as a conference paper at ICLR 2020
cryoDRGN training: We train cryoDRGN with I-D latent variable in minibatches of 10 images for 200 epochs, treating image pose as fixed (11 min/epoch on Nvidia Titan GPU). To simplify representation learning for 96, we center and phase flip images before inputting to the encoder. We encode and decode a circle of pixels with diameter D=128 instead of the full 128x 128 image.
C.1 SUPPLEMENTARY RESULTS
FSC cryoSPARC volumes
3000
0.6
1000
Iatent sacurding, Pltt
frequency
Figure S7: Left: Latent encoding for each image of the dataset from EMPIAR-10076. Bottom: Volumes from 12 sampled values along the latent space (dashed lines). Right: Fourier shell correlation (FSC) curves for 4 structures against the published volumes for classes B-E from corresponding structural states of the large ribosomal subunit during assembly (Davis et al. (0066).
35-
CY9SDARC
3.5
30-
D.1
D2
3.0
2.-
1.5
1.0
0.5
-3.00
-0.75 75
-0.50
25
0.00
0.25
0.50
6.75
1.00
0.5
labent encoding
latent encoding
Figure S8: The latent encoding aligns with cluster assignments from successive round of multiclass refinement cryoSPARC on the subset of images from class D and E.
D FULLY UNSUPERVISED HETEROGENEOUS RECONSTRUCTION
D. DATASET PREPARATION
Linear ID motion: We generated dataset containing one continuous degree of freedom as follows: From an atomic model of protein complex, single bond in the atomic model was rotated while keeping the remaining structure fixed, and atomic models were sampled along this reaction coordinate. 1000 projections with random rotations and in in-plane translations were generated for each model, yielding a total of 50k images, approximating a uniform distribution along a continuous reaction coordinate.
Linear 2D motion: We extended the linear ID motion dataset by introducing a second degree of freedom from rotating bond in the atomic model that connected a different protein in the complex. Similar to the ID motion dataset, from a starting configuration, the original bond was rotated +/- N degrees, and 50 models sampled along this reaction coordinate. Then from the starting
15

---

Published as a conference paper at ICLR 2020
conformation, the second bond was rotated +/- 90 degrees, and 50 additional models sampled along the second reaction coordination. 500 projections were generated from each model, yielding a total of 50k images.
Circular ID motion: For this dataset, we rotated a bond full 360 degrees and sample 100 models along this circular reaction coordinate. 500 projections were generated from each model, yielding total of 50k images.
Discrete 10 class: For this dataset, sampled 10 random configurations for the proteins in the complex. 5000 projection images were generated from each model, yielding a dataset containing a mixture of 10 discrete states.
For all four datasets, random rotations were generated uniformly from SO(3), and translations were sampled uniformly from [-5, 5] pixels. The image size was D=64 with absolute spatial extent of 720Aand Nyquist limit of 22.5A. Schematics of the simulated motions are given in Figure $9
Linear
Linear motion
180-
Circular motion
10 discrete
Figure 89: Ground truth atomic model and the heterogeneity introduced for different datasets.
D.2 TILT SERIES PAIRS
Tilt series pairs is a variant of cryo-EM in which, for each image X,, corresponding image X is acquired after tilting the imaging stage by known angle. This technique was originally employed to identify the chirality of molecules (Belnap et al. (1997)) which is lost in the projection from 3D to 2D and therefore cannot be inferred from standard cryo-EM. Inferential procedures such as expectation maximization converge to one handedness the other depending In multiclass reconstruction, different classes are not guaranteed to possess the same handedness even if there is high relatedness between structures. We remark this experimental technique as we propose using series pairs to encourage invariance of with respect to pose transformations for given V. (and incidentally also to identify the chirality of V2). To train on tilt series pairs, the encoder is split into two MLPs, the first learning an intermediate encoding of each image, and the second mapping the concatenation of the two encodings the latent space. We use an 8 layer MLP with output dimension 128 for the former and 2 layer MLP with input dimension 256 for the latter. All hidden layers have dimension 128. For branch and bound, the combined loss over both images is evaluated for each grid point of SO(3) IR?. To generate the image X associated with X,, prior to rotating the volume by R, rotate the volume by constant 45 degrees around the x-axis.
their initialization.
16

---

Published as a conference paper at ICLR 2020
D.3 TRAINING
We trained cryoDRGN in minibatches of images for 40 epochs without tilt series pairs and 20 epochs with tilt series pairs. We trained I-D latent variable for the linear ID motion dataset, and 10-D latent variables for the remaining datasets. Random angles were used for the first epoch of training learn roughly the correct spatial extent of the volume and BNB pose inference used for the remaining epochs. The runtime was 120 min/epoch vs min/epoch with and without BNB pose inference, respectively, Nvidia Titan V GPU.
D.4 SUPPLEMENTARY RESULTS
cryoDRGN
cryoDRGN tilt
cryoSPARC
Ninoges
Figure S10: Reconstruction results for the linear ID dataset by cryoDRGN and by discrete multiclass recon- struction cryoSPARC. Top: Reconstructed structures from cryoDRGN sampled along the latent space (at depicted points) matches the ground truth variation. The predicted latent encoding correlates with the ground truth latent degree of freedom. Middle: CryoDRGN results with tilt series Bottom: Reconstructed volumes and the distribution of images clusters from discrete multiclass reconstruction in cryoSPARC. Volumes visualized at high low isosurface, showing artifacts the cryoSPARC structures.
17

---

Published as a conference paper at ICLR 2020
eryoDRGN
PC2
PC1
SryoDRGN + tilt
PC2
PC1
cryoSPARC
1900
Figure S11: Reconstruction results for the circular ID dataset by cryoDRGN and by discrete multiclass reconstruction in cryoSPARC. Top: Reconstructed structures from cryoDRGN sampled along the latent space (at depicted points) matches the ground truth variation. The distribution of images in the latent space matches the ciruclar topology the true data manifold. Middle: CryoDRGN results with tilt series Bottom: Reconstructed volumes and the distribution of images over clusters from discrete multiclass reconstruction in cryoSPARC. Volumes are visualized at high and low isosurface, showing artifacts in the cryoSPARC structures.
18

---

Published as a conference paper at ICLR 2020
cryoDRGN
PC1
PC1
SryoDRGN tilt
20006
eryoSPARC.
5000
cryoSPARC das
Figure S12: Reconstruction results linear dataset by cryoDRGN and by discrete multiclass reconstruc- tion in cryoSPARC. Top: Reconstructed structures from cryoDRGN sampled along the latent space (at depicted points) roughly matches the ground truth variation, however the distribution of images in the latent space does recapitulate the truc data manifold well. Middle: CryoDRGN results with tilt series reconstructs the truc structural variation and the distribution of images in the latent space matches the topology of the true data manifold. Bottom: Reconstructed volumes the distribution images over clusters from discrete multiclass reconstruction cryoSPARC. CryoSPARC volumes are visualized at high and low isosurface, showing artifacts at low isosurface
19

---

Published as a conference paper at ICLR 2020
eryoDRGN
eryoDRGN tilt
eryoSPARC
Figure S13: Reconstruction results the dataset containing 10 discrete structures by cryoDRGN by discrete multiclass reconstruction in cryoSPARC. Top: The majority reconstructed structures from cryoDRGN sampled along the latent space (at depicted points) matches the ground truth structures, however some are incorrect (red boxes), and the learned data manifold not well separated into clusters. Middle: CryoDRGN results with tilt series reconstructs the 10 structures and clusters the images in the latent space accordingly. Bottom: Reconstructed volumes from discrete multiclass reconstruction in cryoSPARC: and the distribution of images over clusters. CryaSPARC learns out 10 structures correctly.
eryoSPARC
Dataset
K=2
K=3
K=4
K=5
Linear 1D motion
5.11(3.82)
3.60(2.27)
7.40(4.16)
7.59(4.58)
Linear 2D motion
6.89(2.21)
6,90(3.77)
5,98(2.10)
6.76(4.47)
Circular ID motion
5.16(2.70)
4.87(2.17)
7.50(3.32)
4.62(1.93)
Table Relationship between number classes in cryoSPARC and reconstruction accuracy quantified by an FSC=0.5 resolution metric hetween the reconstructed volumes corresponding to each image and its ground truth volume. We report the average and standard deviation across 100 images in the dataset (lower is better; best possible is 2 pixels).
cryoDRGN 7-D=2
cryoDRGN+tilt
Dataset
7-D=1
z-D=10
z-D=1
z-D=2
z-D=10
Linear ID motion
2.50(0.62)
2.34(0.12)
2.35(0.36)
2.43(0.26)
Linear 2D motion
7.16(4.69)
4.38(3.15)
4.44(2.50)
3.38(1.18)
2.97(1.24)
2.93(1.02)
Circular ID motion
5.61(4.36)
4.95(2.91)
4.05(2.40)
3.12(0.96)
2.65(0.67)
2.63(0.74)
Table S4: Relationship between z dimension in cryoDRGN and reconstruction accuracy quantified by an FSC=0.5 resolution metric between the reconstructed volumes corresponding each image and its ground truth volume, We report the average and standard deviation across 100 images the dataset (lower better; best possible 2 pixels).
20


Reviews:
[{"review_id": "rJetvQrJcB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- The authors proposed a novel method for cryo-EM reconstruction that extends naturally to modeling continuous generative factors of structural heterogeneity. To address intrinsic protein structural heterogeneity, they explicitly model the imaging operation to disentangle the orientation of the molecule by formulating decoder as a function of Cartesian coordinates.\n\n- The problem and the approach are well motivated. \n\n- This reviewer has the following comments:\n1) VAE is known to generate blurred images. Thus, based on this approach, the reconstruction image may not be optimal with respect to the resolution which might be critical for cryo-EM reconstruction. What's your opinion?\n2) What's the relationship between reconstructed performance, heterogeneity of the sample and dimensions of latent space?\n3) It would be interesting to show any relationship, reconstruction error with respect to the number of discrete multiclass.  \n4) How is the proposed method generalizable?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "S1lOMkH6FH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce cryoDRGN, a VAE neural network architecture to reconstruct 3D protein structure from 2D cryo-EM images. \n\nThe paper offers for a good read and diagrams are informative.\n\nBelow are comments for improvement and clarification.\n\n> Consider explaining cryoSPARC in detail given that is the state-of-the-art technique and to which all the cryoDGRN results are compared.\n\n> In Figure 4 and the related experiment,  how are a) the cryoSPARK volumes related to cryoDRGN volumes, b) what do the clusters mean in cryoSPARK and how do they compare with the corresponding outputs of cryoDRGN\n\n> What would runtime comparisons be for cryoSPARK and cryoDGRN, for an unsupervised heteregeneous reconstruction?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyxRW6rTFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "~The authors build a new method to recapitulate the 3D structure of a biomolecule from cryo-EM images that allows for flexibility in the reconstructed volume.~\n\nI thought this paper is very well written and tackles a difficult project. \n\nThere is a previous work that these authors should cite:\n\nUllrich, K., Berg, R.V.D., Brubaker, M., Fleet, D. and Welling, M., 2019. Differentiable probabilistic models of scientific imaging with the Fourier slice theorem. arXiv preprint arXiv:1906.07582.\n\nHow does your method compare to this paper? In Ullrich et al., they report “Time until convergence, MSE [10^-3/voxel], and Resolution [Angstrom]). I think these statistics would be useful to report in your work, as they are more familiar with folks in the cryoEM field.\n\nIn Equation 3, how does one calculate Z, the normalization constant?\n\nFor the decoder, how large of the 3D space are you generating? What are the units? Are you using voxels to represent atomic density? What is the voxel size? Is it the same as on Page 11?\n\nI think more description of the neural network architecture would be useful (more than what is reported on page 12).\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a generative approach to reconstruct 3D images for cryo-electron microscopy (cryo-EM).

All reviewers really liked the paper, appreciate the challenging problem tackled and the proposed solution.

Acceptance is therefore recommended. 
Decision: Accept (Spotlight)