Mixture of neural fields for heterogeneous reconstruction in cryo-EM
Authors: Axel Levy
Rishwanth Raghu
David Shustin
Adele Rui-Yang Peng
Huan Li
Oliver Biggs Clarke
Gordon Wetzstein
Ellen D Zhong
Venue: NeurIPS 2024 poster (2024)
Keywords: cryogenic electron microscopy
neural representations
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We demonstrate ab initio reconstruction of conformational and compositional heterogeneity in cryo-EM datasets with neural fields.

Abstract: Cryo-electron microscopy (cryo-EM) is an experimental technique for protein structure determination that images an ensemble of macromolecules in near-physiological contexts. While recent advances enable the reconstruction of dynamic conformations of a single biomolecular complex, current methods do not adequately model samples with mixed conformational and compositional heterogeneity. In particular, datasets containing mixtures of multiple proteins require the joint inference of structure, pose, compositional class, and conformational states for 3D reconstruction. Here, we present Hydra, an approach that models both conformational and compositional heterogeneity fully ab initio by parameterizing structures as arising from one of K neural fields. We employ a hybrid optimization strategy and demonstrate the effectiveness of our approach on synthetic datasets composed of mixtures of proteins with large degrees of conformational variability. We additionally demonstrate Hydra on an experimental dataset imaged of a cellular lysate containing a mixture of different protein complexes. Hydra expands the expressivity of heterogeneous reconstruction methods and thus broadens the scope of cryo-EM to increasingly complex samples.

Mixture of neural fields for heterogeneous reconstruction in cryo-EM
Axel Levy"
Rishwanth Raghu" Princeton University rraghu@princeton. edu
Stanford University axl exlevy@stanf ord. edu
David Shustin'
Adele Rui-Yang Peng Princeton University
Huan Li
Princeton University
Columbia University
dshustin@princetonn edu
adelep@princeton. edu
h131700columbie. edu
Oliver Biggs Clarke
Gordon Wetzstein
Columbia University
Stanford University
oc21888cumee cow.euma.m..o..mm.aae....eeee
Ellen D. Zhong
Princeton University
thonge@princeton. edu
Abstract
Cryo-electron microscopy (cryo-EM) is experimental technique for protein structure determination that images an ensemble of macromolecules in near- physiological contexts. While recent advances enable the reconstruction of dynamic conformations of a single biomolecular complex, current methods do not adequately model samples with mixed conformational and compositional heterogeneity. In par- ticular, datasets containing mixtures of multiple proteins require the joint inference of structure, pose, compositional class, and conformational states for 3D reconstruc- tion. Here, we present Hydra, an approach that models both conformational and compositional heterogeneity fully ab initio by parameterizing structures arising loss function and
from one of K neural fields. We employ new likelihood demonstrate the effectiveness our approach on synthetic datasets composed of mixtures of proteins with large degrees of conformational variability. We addition- ally demonstrate Hydra on an experimental dataset of cellular lysate containing a mixture of different protein complexes. Hydra expands the expressivity of het- erogeneous reconstruction methods and thus broadens the scope of cryo-EM increasingly complex samples. Webpage: https: htttps://yydees //ty///y/aaa./// cs princeton.ce.. edu
Introduction
Structural information is key to understanding the function of macromolecular complexes, making protein structure determination a crucial tool in basic structural biology and rational drug design. Among experimental structure determination methods, cryogenic electron microscopy (cryo-EM) unique in its capability to reveal dynamic information about large macromolecular complexes in near-native states.
Equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In single particle cryo-EM, a set of biomolecular complexes (i.e. particles) is Hash-frozen and imaged with transmission electron microscope. Each collected image consists of noisy, randomly oriented projection of an individual particle of unknown identity or compasition, frozen in an unknown state. Reconstruction algorithms processing these data without information from upstream algorithms are called ab initio heterogeneous reconstruction methods. Classical reconstruction techniques use a Bayesian approach to optimize a finite number of voxel-based representations [50] 45]. These algorithms enabled biologists to process datasets made of mixture of different protein compositions and, thus illuminate the molecular details fundamental biological processes. These methods, however, tend to aggregate all the proteins of the same composition in single class, despite these proteins being trapped in different conformational states, due to thermal fluctuations prior to the freezing step. Another line of work has extensively studied the possibility of reconstructing continuous motion from EM datasets, using linear combinations of voxel arrays [43] neural- based representations
25 26 Gaussian mixture models [5] 20] or combining voxel array with a flow field 44. These methods sometimes leverage a structural prior about proteins (e.g., proteins are made of a fixed number individual atoms) and enable the reconstruction of molecular motion at high resolution. However, none of these methods can handle datasets containing different types of proteins, thereby strongly limiting their application, especially in the context of in situ cryo-EM.
[68
69
As of today, there exist approaches to simultancously reveal compositional (discrete) and confor- mational (continuous) heterogeneity in cryo-EM datasets, potentially limiting the structures that can be revealed from the data. Unraveling this information poses nontrivial problem that sequential strategies cannot solve. Due to their low nignal-o--nois--aan depending on the type of protein they contain prior to the reconstruction. Because orientations are un- known, the problem cannot be solved by handling compositional heterogeneity before conformational heterogeneity as consensus poses (i.e., orientations) are inaccurate for large motions.
cryo-EM images cannot be clustered
Here, we introduce Hydra, a neural-based method for ab initio heterogeneous reconstruction in cryo-EM. Inspired by the success of implicit neural representations in cryo-EM, we extend the neural field representation of DRGN-AI [26] with a mixture model of K neural fields. Using a new likelihood-basse loss function, simultancously optimize orientations, conformations and class assignments and circumvent the pitfalls of sequential approaches. We demonstrate that our method allows neural-basen methods handle strong compositional heterogeneity and enables the simultaneous reconstruction of compositional and conformational heterogeneity with state-of-the-art accuracy. In an experimental cryo-EM dataset of cell lysate mixture, reveal three compositional states in single pass, fully ab initio. We therefore make the following contributions:
We develop a mixture of neural fields model for ab initio heterogeneous reconstruction in cryo-EM;
We demonstrate that our method improves the expressiveness of neural-based methods for handling strong compositional heterogeneity;
We enable simultaneous reconstruction of conformational and compositional heterogeneity beyond the limits of existing methods;
We demonstrate the reconstruction of multiple protein complexes from experimental dataset of an unpurified sample.
2 Related Work
2.1 Heterogeneous Reconstruction in Cryo-EM
Discrete Variability. Cryogenic electron microscopy offers the potential to reveal the structure of macromolecules in heterogeneous samples, where multiple types or multiple conformations are mixed together [24]. The first reconstruction algorithms handling 3D variability modeled the set of particles as a finite set of static structures. RELION [50] popularized the Bayesian approach to tackle heterogeneous reconstruction and the later introduced multi-body refinement tool [34] offered the possibility to segment static density maps and model continuous motion as a combination of rigid transformations. The software suite was recently improved with the Blush regularization tool 22 4 leveraging data-driven prior to enable the reconstruction of small protein-nucleie acid complexes (S 40 kDa). CryoSPARC 45 accelerated the inference with stochastic gradient descent and, in the

---

extension, modeled continuous motion with linear combination of density maps. These EM reconstruction but do not have the capability to methods represent the state of the art for cryo-EM represent complex non-linear motion and tackle the discrete heterogeneity (e.g., with "multi-class ab initio") before the continuous variability (e.g., with 3DVA or [21)), leading to inaccurate pose estimation for states exhibiting large motion.
3DVA 4
Non-Linear Methods for Heterogeneous Reconstruction. In the past years, significant progress has been made in revealing non-linear dynamics from cryo-EM data. Manifold embedding [ and Laplacian methods [33] are among the first attempts model non-linear continuous motion but have only been applied to a small number of macromolecular complexes [8] 77. HEMNMA [20] used a
decomposition over the low-energy normal modes of an atomic model, thereby leveraging prior over the structure and the dynamics of macromolecules. Herreros et al. [17] proposed the use of 3D Zernike polynomials and eliminated the need for pseudo-atomic models. CryoDRGN 66 used neural networks to continuously represent deformable density maps as well a variational auto-encoding framework to estimate conformational states. E2GMM [5]. cryoFold [71, and DynaMight [51] followed this mcoder-decoder framework and used Gaussian mixture models to represent density maps, thereby reducing the memory footprint of previous non-linear methods. 3DFlex [44 introduced the use of a parametric flow field to smoothly deform canonical 3D density maps, leveraging the knowledge that energetically favorable deformations tend to preserve the local geometry of proteins. Although these methods demonstrated the ability to reconstruct molecular motions and heterogeneity, they all need a coarse initialization of the density map, or the poses to be provided by an upstream reconstruction algorithm. In practice, this ab initio step is error prone in the presence of large conformational motions.
Ab Initio Heterogeneous Reconstruction. Recent works investigated the problem of reconstructing an ensemble of density maps where poses are unknown. The preliminary cryoDRGN method [68] tackled the ab initio reconstruction problem by combining traditional pose search algorithms with the encoder-decoder neural-based architecture, which was refined in cryoDRGN2 (70]. Multi-
CryoGAN [5] sidestepped pose estimation by casting the reconstruction problem as a distribution matching problem and successfully revealed 3D variability in synthetic cryo-EM datasets. Rosenbaum et al. [49 showed that the auto- suto-encoding framework could be applied to jointly estimate poses and conformations of atomic models, and Levy al. (25) demonstrated a fully autoencoding framework for ab initio heterogeneous reconstruction on real benchmark datasets. DRGN-AT (26) recently introduced hybrid pose search strategy combined with an autodecoding architecture to handle low-signal datasets and demonstrated ab initio heterogeneous reconstruction on challenging datasets containing a significant number of junk images. These methods extend neural-based reconstruction to scenarios where poses cannot be reliably estimated by any upstream algorithms, but have a limited capability to represent mixtures of biomolecular complexes, due to the limited capacity of a single neural representation.
Here, we propose to represent the landscape of accessible structures via an ensemble of K neural net- works, each specialized in representing the variability within single compositional state. By jointly estimating structures and orientations, our method handles datasets that exhibit strong compositional heterogeneity and large motions.
2.2 Neural Fields for Large and Dynamic Scenes
Dynamic Scenes. In graphics, neural networks have also been used as light-weight, differen- tiable representations of continuously defined signals (e.g., occupancy fields (31), signed distance
functions (37 6), surface light fields (66), latent representation of appearance [52] 53], radiance
fields B2 and light fields (54). In order handle -epe-depen and represent dynamic
scenes, two approaches have been explored, as described in 39 Similar to the cryo-EM EM method 3DFlex 4. deformation-bass
approaches apply a spatially- varying deformation to some canonical radiance field [38 42 59]. Methods following this strategy recover detailed dynamic scenes but are unable to model any topological variations. Similar to cryoDRGN (69, modulation-butl based approaches directly condition the radiance field of the scene on some latent vectors encoding temporal or dy- namic information KK 27 [2]. These techniques are capable of modeling arbitrary deformations, topological changes, and other complex phenomena, but require additional regularization strategies
3

---

to avoid trivial, non -la-ssuuee solutions. HyperNeRR B9 was introduced as combination of both approaches and enabled photo-realistic reconstruction of dynamic scenes with topological changes.
Large-Scale Scenes. Due to their limited capacity, neural networks have essentially been used to represent single objects or scall-scale scenes. Several methods [46 47] have looked into the possibility of using an ensemble of neural fields represent large scenes such as neighborhood in the city of San Francisco 57.. Another approach is to provide extra capacity with coarse 3D) [2 55 or a block-coordinate multi-scale decomposition [2]] These works focus on static reconstruction, and if dynamic objects are present in the scene, these are simply masked out [57]. Ost et al. [36] enabled the representation of complex, dynamic multi-object scenes by decomposing them into their static and dynamic parts and learning one neural radiance field per dynamic object.
grid of latent codes
Here, we use the modulation- based approach to handle the continuous motion of proteins and an ensemble of neural fields to increase the representational capacity of our model, allowing it to reconstruct compositional mixtures of proteins. In contrast with the works cited above, we need to estimate which compositional state each image belongs to which done using a variational approach- - while simultaneously optimizing the ensemble of neural networks and individual image poses.
2.3 Adaptive Mixtures of Experts
mixture of experts (MoE) model uses an ensemble of neural networks to process given input [9] Doing so, the input space can be partitioned into subspaces which only one of the neural networks needs to become "expert". The mechanism with which the input space gets partitioned is usually referred to as the gating mechanism. Masoudnia and Ebrahimpour [29] partitions MoE approaches into two groups, depending on the gating mechanism and both how and when this mechanism is involved: (1) mixtures of explicitly localized experts (MELE) cluster the input data before the experts' training phase starts while (2) mixtures of implicitly localized experts (MILE) jointly optimize the expert networks and the gating mechanism. In cryo-EM, clustering methods operating directly on images are usually ineffective because of the high level of noise and because of the presence of other nuisance variables (orientation and conformation), making the MELE approach irrelevant.
Jacobs et al. 19 examined the use of different error functions (or loss functions) to optimize the expert networks and the gating mechanism, and the best performance was obtained using the negative log-probability of Gaussian mixture model [35] Several architectures have been explored, and one of the most applied is the mixture of MLP expeerterts (MME) [63 91 where mulli-layes perceptrons (MLPs) are used for both the experts and the gating networks. Our approach can be viewed as an application of the MILE method where, in Jacobs et al. 19), the error function corresponds to the negative log- bog-probability of a Gaussian mixture model. Due to the high level of noise, the gating mechanism ignores the content of input images and relies autodecoding framework.
3 Methods
In this section, we first describe the image formation model in single particle cryo-EM 3.1 We then define a 3D variability model that models both compositional and conformational heterogeneity (3.2h. Based on these two first sections, we describe our system with a latent variable model (3.3 and explain our optimization method (3.4). The method is schematically described in Figure]
3.1 Image Formation Model
In cryo-EM, purified solution of macromolecules is flash-frozen inside thin layer of vitreous ice. Each molecule gets trapped in a random orientation with respect to the microscope and in random conformational state, approximately following the Boltzmann distribution ambient temperature (2). The sample is exposed to an electron beam and individual 2D projections are extracted from micrographs via step known as "particle picking". The reconstruction task therefore starts with a given set of N images (or particles). Each image 1; can be modeled as 60 50
1 C. P V1 + Tk.
(1)

---

(b)
d
Forward Modill
Neural Fields
V
Confer-
K
25,k
matlons
K
Pi.k
Poses
Picked
k 2
Partilee
I.
Hith 1)
Clasa
Probabiilies
Loss Neg. Log Likelhood
Optimized Parameters
PEN 1IN--3
Figure 1: Overview of Hydra. (a) Schematic representation of the space of energetically plausible density maps heterogeneous cryo-EM dataset. We approximate this space with finite union of low now-dimensional manifolds. compositional states (or classes) are labeled by The "conformation" within class ls refers to intrinsic coordinates within k-th manifold. (b) Optimization pipeline. The conformations, poses, class probabilities and neural fields are optimized such as to maximize the likelihood of the observed images ("picked particles") under the model described in Section
where C; models the the Contrast Transfer Function (CTF), V2 is a scalar 3D field (IR3 _ IR) known the "electron scattering potential" the "density map", di (R,, t;) is "pose" (R, SO(3), t, E RR) and P projects V, on a 2D grid:
{.V R [.-mmm Lxx, Bm., - by 11²) (m, n) E (l,...,
P(R,)) Vi
(2)
Y models isotropic Gaussian noise (77² N(0,0²)) In typical experiment, N can vary between 10 and 10?; The signal-to-noise ratio can vary between 10 and 10-2.
3.2 Heterogeneity Model
Structural heterogeneity among the macromolecules originate (1) from continuous motion along a small number of degrees of freedom, or (2) from discrete compositional changes. We refer to the first kind of heterogeneity as "conformational heterogeneity" and to the second one as "compositional heterogeneity".
To model this mathematically, we make the assumption that all the density maps V, belong finite union of low-dimensional manifolds:
Vi E (1....... N). V. (V(2; e Rd, & E (1...., K]),
where K N, d E N and, for all k., 0s E θ is an unknown parameter. In other words, we assume that there exist K compositional states and that the conformational motion of state k can essentially be described with d degrees of freedom.
3.3 Latent Variable Model
We statistically describe the set of observed images with latent variable model. Here, the latent variables are the poses 6,, the conformations 2; and the class identities k,, while {0x {0x) shared parameters.
k= is set of
Given the image formation model described by Eq. and the heterogeneity model described by Eq. each observed image can be seen as a sample from a multivariate mixture model:
// [|olk k))(=(k; A)N(C, PaV((; ddodz.
p(le) Σ p(ks k)
(4)
We parameterize probability distribution over the class identity, such that:
exp(st.t)
Vi E (1....,N), N), E (1..... K}. p(ks k) = softmax(s:)x = Σ, exp(sis)
(5)
5

---

Kg where s; E IR*K is free parameter called the "score". For each k, we parameterize point estimates of the continuous latent variables:
p K; = k) = =600 - 0i,k), Φi E (O(33 x R
(6)
p(2 K2 = k)) = =6(2- -1,,),
2i,k E Rd
Under the model in Equation and the variational parameterization described above, the negative log-likelihood of an image is given by
G(Horn)es 1K
( {0x k==1)
1 2g || Ci &or. V(zia; 0x) log(softtax(ss))))
=- log Σ exp
In our implementation, each low-dimensional manifold V(:; 02. ) is implemented with residual MLP.
We provide further details on the architecture of the neural networks in the supplementary materials.
3.4 Hybrid Optimization Strategy
We aim at minimizing the negative log-likelihood of the set of observed images,
L Σ (I (0x))E))))
(8)
over ((oLx), (. {0x]). All the parameters are optimized using combination of gradient- based optimization and exhaustive search. Here, we use an autodecoding framework and do not amortize the inference of latent variables with an encoder, following the observation in Levy et al. 26 that encoders tend to memorize images in highly-noisy setups.
We handle the minimization over (=,,a), } and {0x} with stochastic gradient descent. The minimization over the poses (dbt,.) more challenging, due the existence of local minima. We take inspiration from the two-step pose estimation strategy introduced in DRGN-AI (26) and adapt it cope with the simultaneous representation of multiple maps. First, optimization done with hierarchical pose search (HPS) in alternation with stochastic gradient descent on the other variables. On a given random minibatch of indices I (L, NN.
poses is
Vi I Vk (L....,k}, K}.
Φi argmin ||4 C2 * PoV(2, dEN
where Q2 is predefined grid in the space of poses (SO(3) x )R²). See (26 and the supplementary for more details on how the minimization can be done efficiently with hicrarchical strategy. This robust but compurationallly-xpenna x 10 and 10 After that, most poses are located in the basin of attraction of their global optimum and switching to stochastic gradient descent (SGD) makes computation more efficient while improving pose accuracy (poses are not constrained to belong to predefined grid of fixed resolution during SGD).
strategy is used for a predefined number of steps, usually between
At the end of optimization, the estimated class of image is simply given by the index k; of the largest entry in 8, E R
Experiments
We run three experiments to evaluate Hydra. In Section 4.1 we show that our method improves the expressiveness neural-based methods and can reveal strong compositional heterogeneity fully ab initio. In Section 4.2 we use Hydra to reveal compositional heterogeneity in an experimental dataset containing protein complexes diverse sizes, in a single run. In Section 4 we demonstrate the simultaneous reconstruction of compositional and conformational heterogeneity.

---

(4)
(b)
(G)
(d) Ground Trulh
1=1
K=3
K= 5
(# partices)
11,300
(7,000))
(1,030)
(3.000)
11,000y
(1,000)
F1
21
11 11.000
(1 10000
(1,000)
Figure 2: Hydra captures strong compositional heterogeneity in the tomotwin3 dataset. (a-c) Reconstructed densities and estimated conformations with K E {1, 11, We report the number of particles in each class between parenthesis. We represent density maps using isosurfaces. (a) With K 1 (DRGN-AI), the model fails reconstruct the three density maps, in spite of using d 8 dimensions represent conformations. (b) With = (d 2). Hydra recovers the three density maps with perfect classification accuracy. (c) With K = 5 (d = 2). the model is over-parameterized and 2 classes out of 5 end up empty at the end of optimization. (d) Ground truth density maps for the tomotwin3 dataset.
ARI 1
Per-image FSC 1
Model
All
PDB 6up6
PDB 6id1
PDB 4cr2 1
Hydra (K = 3)
1.00
0.25 + 0.03
0.394 + 0.001
0.367 0.001
Hydra (K = 5)
1.00
0.25 + 0.03
0.396 + 0.001
0.367 + 0.001
DRGN-AI (K = 1)
0.59
0.0242 = 0.003
0.040 = 0.005
0.042 0.004
CryoDRGN2
0.36
0.08 + 0.01
0.070 0.006
0.3 0.1
CryoSPARC
1.00
0. 0.284
0.367
0.338
Table 1: Hydra captures compositional heterogeneity in challenging synthetic dataset and outperforms other neural-bascd methods. The classification accuracy evaluated each method using the adjusted Rand index (ARI) (18). To evaluate each method's 's reconstruction quality, we use the mean area under the Fourier shell correlation (FSC) curve for 20 images per class (we report #1 standard deviation). We bold the best result, and underline the second best result.
4.1 Ab initio reconstruction of compositional heterogeneity
We evaluate Hydra on tomotwin3, synthetic dataset of 3,000 images emulating a protein sample containing multiple species with static structures. We selected the 6th, 7th, and 8th largest proteins by atomic weight from the TomoTwin training dataset
48), which correspond entries 6up6, 6id1, and 4cr2 in the RCSB PDB (Protein Data Bank) 3 We rendered 1,000 synthetic images of each specimen following the cryo-EM image formation model. For each protein structure, we simulated density maps with the ChimeraX molmap command (30 (88) We padded the density maps to a box size of D = 384 pixels and centered them. We sampled 1,000 orientations per ground truth class uniformly from SO(3). We projected the density maps using Eq. and applied translation vector uniformly chosen from a 30-pixel-wide box centered at the origin. We applied CTF sampled from EMPIAR- 1247 [10] and added Gaussian noise (SNR 0.01). We downsampled each image to box size of D = 128, yielding 3,000 images in total (samples shown in Figure ST).
We compare the performance of Hydra to several state-of-the fie-the-rt art ab initio methods for resolving compositional heterogeneity in cryo-EM datasets, including DRGN-AI 266,, cryoDRGN2 (70), and cryoSPARC 45 Hydra only uses 2 dimensions for the conformational space, while DRGN-AI and Cry are evaluated with more expressive 8-dimensional space. We test Hydra using the correct number of classes (K 3, the true number of specimens present in the sample) and
7

---

Hydra (K = 4)
Junk
(b)
CryoSPARC Labels
RyR
p97
RyR
p97
Cill
Junk
Total
Hydra Labols
Rlyf
00,00r
34
86
13r
p87
7,003
182
7,500
Cill
7,717
10
Junk
822
841
2,405
1,000
Total
00,007
7,849
10,403
0,788
05.009
(c)
(d)
DRGN-AI
1.0
p9/
0.6
CIII
UMAP2
0.4
0.0
DC
27.3A
13.64
9.1A
6.84
Junk
Resolution
UMAP1
Figure 3: Hydra captures compositional heterogeneity in real dataset containing mixture of membrane and soluble protein complexes. (a) Density maps obtained with Hydra (K 4) on the Ryanodine receptor dataset. (b) Confusion matrix between Hydra and cryoSPARC K heterogeneous refinement (three classes representing RyR were combined for analysis). (c) Fourier shell correlation (FSC) between the Hydra density maps and refined cryoSPARC density maps. (d) Left: latent space plot and right: representative density maps from each of the latent space clusters from DRGN-AL.
using an over-parameterized setup (K 5 in Figure 2 and = 7 in Figure S2 For DRGN-AI and CryoDRGN2, we classify each image using k-means clustering the conformational space with clusters. We evaluate per-image FSC for each class using the dataset's ground truth class labels.
In Table D we report results for each configuration of Hydra and other methods, (best of three replicas with distinct seeds for Hydra and DRGN-AI, full results in Table 3 We observe that other methods with implicit neural volume representations, including DRGN-AL and CryoDRGN2, fail to capture the compositional heterogeneity of the dataset. Hydra matches the classification quality of cryoSPARC, which models independent, discrete mixtures. Hydra also outperforms all methods on per-image FSC, distributional volume reconstruction quality metric based on the Fourier Shell Correlation (FSC) (68). We provide additional results with larger version of this dataset in Figure S3 and additional metrics (including aototrranslation accuracy) in Table
4.2 Ab initio reconstruction of an experimental cryo-EM mixture dataset
We evaluate our method on an experimental dataset of protein mixture from red blood cell lysate subjected to consity-gradien centrifugation followed by chemical cross-linking. Through an ex- haustive, expert-driven data processing pipeline in cryoSPARC [45], we determined that the dataset consisted of substantial compositional heterogeneity from three main component protein complexes: the membrane proteins ryanodine receptor (RyR) and mitochondrial respiratory chain complex III (CIII), and dimeric complex of the soluble calosin-containing
protein (p97). A sweep of K-values from cryoSPARC ab-initio reveals that K 2 5 is necessary in order for cryoSPARC to reconstitute distinct RyR, p97, CIII, and junk classes (Figure S4).
In Figure we compare the performance of DRGN-AI (6 with Hydra. Figures 3 and S5 show that DRGN-AI successfully partitions the dataset into two clusters corresponding the RyR and non-RyR particles; however, DRGN-AI AI is unable to learn distinct shapes for all three discrete structures and instead appears learn structural artifacts from the non-RyR particles that match the overall RyR shape. By contrast, Hydra with K = -4 successfully separates the particles into the three component protein complexes and a fourth junk class (Figures B and [S6). As baseline for comparison, we also carried out the typical workflow of first generating poses from consensus reconstruction in cryoSPARC, followed by fixed-pose DRGN-AI to separate the heterogeneity. As can seen in Figure S7 DRGN-AT recovers a high resolution RyR density but fails to learn non-RyR protein structures when training from poses from homogeneous reconstruction. multi-step reconstruction with DRGN-AI AI does not either reveal the CIII class (Figure S8).

---

22
Class
Class 3
2/
Figure 4: Hydra effectively recovers both compositional and conformational heterogeneity in ribosplike dataset. Particles within each latent space are colored by class. Representative density maps are generated from the latent points denoted in white dots.
4.3 Ab initio reconstruction of conformational and compositional heterogeneity.
We prepare synthetic dataset exhibiting both compositional and conformational heterogeneity (ribosplike). For each of the pre-catalytic spliceosome, 80S ribosome, and SARS-CoV-2 spike protein, 5,000 images are generated from different conformational states the macromolecule along 1-dimensional trajectory of motion [1 64 61] yielding dataset of 15,000 images (See SI
Section F for additional details and Figure SI for sample images).
As seen in Figure 4 our method with K 3 nearly perfectly separates the particles into their three classes. Within each neural field, the con- formational change of the dominant particle type is well captured both by the conformational space. Due to the trimeric structure of the spike protein, observe three continuous manifolds in the conformational space, though the major- ity of spike particles are aligned one of the symmetries. Tables and compare the perfor- mance of Hydra with other ab initio baselines, demonstrating superior performance in classi- fication accuracy as measured by the Adjusted Rand Index (ARI) and volume reconstruction quality (per-image FSC). It additionally achieves the lowest pose error amongst all methods. Representative reconstructions for baseline methods are shown in the supplementary material (Figure
Model
Img-FSC t
ARI
Pose Err. J
Hydra (K=3)
0.414
0.997
1.070
DRGN-AI CryoDRGN2
0.207
0.994
45.379
0.399
0.986
1,2120
CryoSPARC
0.344
0.972
1.576
Table 2: Hydra outperforms state-of-the-art meth-
ods on the ribosplike dataset. Quantitative results include reconstruction quality (per-image FSC), par- ticle classification (ART), and median pose accuracy (gendesic distance between rotations, degrees).
Discussion
This work introduces Hydra, method that expands the expressiveness of neural-based models for heterogeneous reconstruction in cryo-EM. Hydra models structures as arising from 1 of K neural fields and is designed capture heterogeneity datasets containing a mixture of multiple species. Notably, our method runs fully ab initio, i.e. does not rely on pre-estimated poses, which are not straightforward to obtain cases of strong compositional heterogeneity. We evaluated Hydra on both synthetic and experimental datasets, showed improved performance over previous neural-based methods on compositionally heterogeneous datasets, and demonstrated that could simultaneously handle compositional and conformational heterogeneity.

---

Our method can be seen as an instance of mixture of experts model, where the gating mechanism handled by an autodecoding framework, due the low signal-to-noise ratio the input data. However, predicting classes with neural network can be viewed classification task and may be easier than pose or conformation estimation. We view the possibility of using (potentially pretrained) neural network for classification, thereby making use of amortized inference, as an exciting avenue for future work.
One important limitation of our approach is the need to specify the number of classes K prior reconstruction. Other methods that cope with compositional heterogeneity, like RELION [50] and cryoSPARC [45, possess the same requirement, and this is solved in practice by running the algorithm several times with sweep on the hyperparameter K. As this process is time and energy- consuming (a single reconstruction can rum for up to 4 GPU-days on high-end GPUs), more efficient methods for hyperparameter selection, for example, with an adaptive and/or hierarchical strategy, an impactful direction for future work. We would also like to note that, in its current implementation, Hydra must give the same latent dimension (d) all the classes, but this constraint could be relaxed with, for-example, dictionary-based representation for the conformations.
With Hydra, we broaden the scope datasets that cryo-EM reconstruction algorithms can process. Our method would be especially suitable for data collected in situ (i.e., inside the cell), where mixtures of dynamic biomolecular complexes coexist. This data is usually acquired by cryogenic electron tomography (cryo-ET), where the sample is progressively tilted throughout the data collection process. Extending our method to enable subtomogram averaging of cryo-ET ET data another potential future direction that we hope can enable future discoveries in structural biology and facilitate the design of new therapeutic compounds.
Acknowledgments
The author(s) are pleased acknowledge that the work reported on in this paper substantially performed using the Princeton Research Computing resources at Princeton University which is consortium of groups led by the Princeton Institute for Computational Science and Engineering (PICSeiE) and Office of Information Technology's 's Research Computing. The Zhong lab gratefully acknowledges support from the Princeton Catalysis Initiative, Princeton School of Engineering and Applied Sciences, and Generate Biomedicines. The funders had no role in study design, data collection and analysis, decision publish or preparation of the manuscript.
10

---

References
[I] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- Bartin-Buualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021.
[2] Lars V Bock and Helmut Grubmuller. Effects of cryo-em cooling on structural ensembles. Biophysical Journal, (21(3)148a, 2022.
[3] Stephen Burley, Charmi Bhikadiya, Chunxiao Bi, Sebastian Bittrich, Chen, Gregg Crichlow, Cole H Christie, Kenneth Dalenberg. Luigi Di Costanzo, Jose M Duarte, et al. Resb protein data bank: powerful new tools for exploring 3d structures of biological macromolecules for basic and applied research and education in fundamental biology, biomedicine, biotech- nology, bioengineering and energy sciences. Nucleic acids research, 49(D1):D477-D451 2021.
[4] Alister Burt, Bogdan Toader, Rangana Warshamanage, Andriko von Kugelgen, Euan Pyle, Jasenko Zivanov, Dari Kimanius, Tanmay AM Bharat, and Sjors Scheres. An image processing pipeline for electron cryo-tomography in relion-5. bioRxiv, pages 2024-04, 2024.
[5] Muyuan Chen and Steven J Ludtke. Deep learning-based mixed-dimensional gaussian mixture model for characterizing variability in cryo-em. Nature methods, 18(8):930-96,, 2021.
[6] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling- In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5939-5948, 2019.
[7] Ali Dashti, Peter Schwander, Robert Langlois, Russell Fung, Wen Li, Ahmad Hosseinizadeh, Hstau Y Liao, Jesper Pallesen, Gyanesh Sharma, Vera A Stupina, et al. Trajectories of the ribosome as brownian nanomachine. Proceedings of the National Academy of Sciences, 1 1499 2014.
[8] Ali Dashti, Ghoncheh Mashayekhi, Mrinal Shekhar, Danya Ben Hail, Salah Salah, Peter Schwan- der, Amedee des Georges, Abhishek Singharoy, Joachim Frank, and Abbas Ourmazd. Retrieving functional pathways of biomolecules from fingle-pariccle snapshots. Nature communications, 11(1):4734, 2020.
[9] Reza Ebrahimpour, Ehsanollah Kabir, Hossein Esteky, and Mohammad Reza Yousefi. View- independent face recognition with mixture of experts. Neurocomputing, 71(4-6):103--1107 2008.
[10] Ryan Feathers, Erica K. Richael, Kayla A. Simanek, J. Christopher Fromme, and Jon Paczkowski. Structure of the rhir-pqse complex from pseudomonas aeruginosa reveals mechanistic insights into quorum-sensing gene regulation. Structure, 30(12):6626-163..e 2022. ISSN 0969-2126. doi: https://doi..rg/10.. http:///..........00..../....../ 1010000000001111222220000...0000 //www www.sseiieccdieccc..cm..
URL https
[11] Joachim Frank and Abbas Ourmazd. Continuous changes in structure mapped by manifold embedding of single-particle data in cryo-em. Methods, 100:61-67, 2016.
[12] Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12479-12488, 2023.
[13) Guy Gafni, Justus Thies, Michael Zollhofer, and Mauthis Nießner. Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 8649-8658, 2021.
[14] Krzysztof Gorski, Eric Hivon, Anthony Banday, Benjamin D Wandelt, Frode K Hansen, Mstvos Reinecke, and Matthia Bartelmann. Healpix: A framework for high-resolution dis- cretization and fast analysis of data distributed on the sphere. The Astrophysical Journal, 622 (2):759, 2005.
11

---

[15] Harshit Gupta, Thong H Phan, Jaejun Yoo, and Michael Unser. Multi-cryogan: Reconstruction of continuous conformations in cryo-em using generative adversarial networks. European Conference Computer Vision, pages 429 444. Springer, 2020.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[17] David Herreros, Roy R Lederman, James Krieger, Amaya Jiménez-Moreno, Marta Martínez, David Myška, David Strelak, Jiri Filipovic, Ivet Bahar, Jose Maria Carazo, et al. Approximating deformation fields for the analysis of continuous heterogeneity of biological macromolecules by 3d zernike polynomials. IUCrJ, 8(6):922-1005, 2021.
[18] Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification, 2:193-218, 1985.
[19] Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, (1):79-87, 1991.
[20] Qiyu Jin, Carlos Oscar S Sorzano, José Miguel de La Dosa-Treein, José Román Bilbao-Castir Rafael Nüñez-Rammrrzz Oscar Llorea, Florence Tama, and Slavica Jonić. Iterative elastic 3d-to-2d alignment method using normal modes for studying structural dynamics of large macromolecular complexes. Structure, 22(3):496-506, 2014.
[21] Dari Kimanius, Kiarash Jamali, and Sjors Scheres. Sparse fourier backpropagation cryo-em reconstruction. Advances in Neural Information Processing Systems, 35: 35:12395-12408 2022.
[22] Dari Kimanius, Kiarash Jamali, Max E Wilkinson, Sofia Lövestam, Vaithish Velazhahan, Takanori Nakane, and Sjors HW Scheres. Data-driven regularisation lowers the size barrier of cryo-em structure determination. bioRxiv, pages 2023-10, 2023.
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988) 2014.
[24] Roy R Lederman and Amit Singer. Continuously heterogeneous hyper-objects in cryo-em and 3-d movies of many temporal dimensions. arXiv preprint 17XY::1704.02899, 2017.
[25] Axel Levy, Gordon Wetzstein, Julien NP Martel, Frederic Poitevin, and Ellen Zhong. Amor- tized inference for heterogeneous reconstruction in cryo-em. Advances in neural information processing systems, 00111-3008000 2022.
[26] Axel Levy, Frederic Poitevin, Jake D Johnston, Francesca Vallese, Oliver Biggs Clarke, Gordon Wetzstein, and Ellen D Zhong. Revealing biomolecular structure and motion with neural ab initio cryo-em reconstruction. biorxiv:preprint, 2024.
[27] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5531, 2022.
[28] Julien NP Martel, David B Lindell, Connor Z Lin, Eric R Chan, Marco Monteiro, and Gordon Wetzstein. Acorn: Adaptive coordinate networks for neural scene representation. arXiv preprint arXiv:2105.02788, 2021.
[29] Saced Masoudnia and Reza Ebrahimpour. Mixture of experts: literature survey. Artificial Intelligence Review, 42:275-293, 2014.
[30] Elaine C Meng. Thomas D Goddard, Eric F Pettersen, Greg S Couch, Zach J Pearson, John H Morris, and Thomas E Ferrin. Uesf chimerax: Tools for structure building and analysis. Protein Science, 32(1 1)e44992, 2023.
[31] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oceupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4460-4470, 2019.
12

---

[32] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
[33] Amit Moscovich, Amit Halevi, Joakim Andén, and Amit Singer. Cryo-em reconstruction of continuous heterogeneity by laplacian spectral volumes. Inverse Problems, 36(2):224003, 2020.
[34] Takanori Nakane, Dari Kimanius, Erik Lindahl, and Sjors HW Scheres. Characterisation of molecular motions in cryo-em single-particle data by multi-body refinement in relion. elife, 7: e36861, 2018.
[35] Steven Nowlan and Geoffrey E Hinton. Evaluation of adaptive mixtures of competing experts. Advances in neural information processing systems, 3, 1990.
[36] Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2856-2865, 2021.
[37] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
[38] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualle Nerfies: Deformable neural radiance fields. In Proceedings of the JEEF/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[39] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo -arail, and Steven M Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106. 1102222000..128,.. 2021.
[40] Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Con- volutional occupancy networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 523-540. Springer, 2020.
[41] Clemens Plaschka, Pei-Chun Lin, and Kiyoshi Nagai. Structure of a pre-catalytic spliceosome. Nature, 546:617-621, 2017.
[42] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10318-10327, 2021.
[43] Ali Punjani and David J Fleet. 3d variability analysis: Resolving continuous flexibility and discrete heterogeneity from single particle cryo-em. Journal of structural biology, 213(2): 107702, 2021.
[44] Ali Punjani and David J Fleet. 3dflex: determining structure and motion of flexible proteins from cryo-em. Nature Methods, 20(6):860- 870, 2023.
[45] Ali Punjani, John Rubinstein, David J Fleet, and Marcus A Brubaker. cryospare: algorithms for rapid unsupervised cryo-em structure determination. Nature methods, 14(3):290-296 2017.
[46] Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. Derf: Decomposed radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14153-14161, 2021.
[47] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF international conference on computer vision, pages 14335-14345, 2021.
[48] Gavin Rice Thorsten Wagner, Markus Stabrin, Oleg Sitsel, Daniel Prumbaum, and Stefan Raunser. Tomotwin: generalized 3d localization of macromolecules in cryo-electron tomograms with structural data mining. Nature Methous, 20(6))871 880, Jun 2023. ISSN 1548-7105 doi: 019-884441522-02-1888883 URL :/ttsp:://doi.oo org/ 10. 1 1038/ 00388//445500 s415944152220 023 01878-2
13

---

[49] Dan Rosenbaum, Marta Garnelo, Michal Zielinski, Charlie Beattie, Ellen Clancy, Andrea Huber, Pushmeet Kohli, Andrew W Senior, John Jumper, Carl Doersch, al. Inferring a continuous distribution of atom coordinates from cryo-em images using vaes. arXiv preprint arXiv:2106. 14108, 2021.
[50] Sjors HW Scheres. Relion: implementation of bayesian approach to cryo-em structure determination. Journal of structural biology, 180(3):519-33 530, 2012.
[51] Johannes Schwab, Dari Kimanius, Alister Burt, Tom Dendooven, and Sjors Scheres. Dynamight: estimating molecular motions with improved reconstruction from cryo-em images. bioRxiv, pages 2023-10, 10, 2023.
[52] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießnner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2437-2446, 2019.
[53] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-awwan neural scene representations. Advances in Neural Information Processing Systems, 32, 2019.
[54] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 14:19933-99325 2021.
[55] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11358-11367, 2021.
[56] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kei.. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances neural information processing systems, 33:7537-7547, 2020.
[57] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[58] Guang Tang, Liwei Peng, Philip R Baldwin, Deepinder S Mann, Wen Jiang, Ian Rees, and Steven J Ludtke. Eman2: an extensible image processing suite for electron microscopy. Journal of structural biology, 157(1):38 46, 2007.
[59] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12959-12970, 2021.
[60] Miloš Vulović, Raimond BG Ravelli, Lucas van Vliet, Abraham J Koster, Ivan Lazić, Uwe Lücken, Hans Rullgärd, Ozan Öktem, and Bernd Rieger. Image formation modeling in cryo- electron microscopy. Journal of structural biology. 183(1):19-33 2013.
[61] Alexandra C. Walls, Young-Jun Park, M. Alejandra Tortorici, Abigail Wall, Andrew McGuire, and David Veesler. Structure, function, and antigenicity of the sars-cov-2 spike glycoprotein. Cell, 181(2):281-292cm 2020. ISSN 0092-8674. doi: https://oi.oog URL UtL.ittts///www...cacddeecc...o
10.1016/j.elll2022.000 p11/$0092867440000262
[62] Alexandra Walls, Young-Jun Park, M Alejandra Tortorici, Abigail Wall, Andrew T McGuire, and David Veesler. Structure, function, and antigenicity of the sars-cov-2 spike glycoprotein. Cell, 181(2)2281-99) 2020.
[63] Steven Richard Waterhouse, Classification and regression using mixtures of experts. PhD thesis, Citeseer, 1998.
14

---

[64] Wilson Wong, Xiao-chen Bai, Alan Brown, Israel S Fernandez, Eric Hanssen, Melanie Condron, Yan Hong Tan, Jake Baum, and Sjors HW Scheres. Cryo-em structure of the Plasmodium falciparum 80s ribosome bound to the anti-protozoan drug emetine. eliffe, 3:e03080, jun 2014. ISSN 2050-084X. doi: 10.7554/eL.ife.000008 URL httpps://doi org/10 7054///...55///// 70l..5554eel.. eLife 03080
[65] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9421-9431, 2021.
[66] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural Information Processing Systems, 33:2492-2502,, 2020.
[67] Anna Yershova, Swati Jain, Steven M Lavalle, and Julie C Mitchell. Generating uniform incremental grids on so (3) using the hopf fibration. The International journal of robotics research, 29(7):801-812, 2010.
[68] Ellen D Zhong, Tristan Bepler, Joseph H Davis, and Bonnie Berger. Reconstructing continuous distributions of 3d protein structure from cryo-EM images. In International Conference Learning Representations (ICLR), May 2020.
[69] Ellen D Zhong, Tristan Bepler, Bonnie Berger, and Joseph Davis. Cryodrgn: reconstruction of heterogeneous cryo-em structures using neural networks. Nature methods, 18(2):176-185, 2021.
[70] Ellen D Zhong, Adam Lerer, Joseph H Davis, and Bonnie Berger. Cryodrgn2: Ab initio neural reconstruction of 3d protein structures from real cryo-em images. In Proceedings of the 2021.
IEEE/CVF International Conference on Computer Vision, pages
[71] Ellen D) Zhong, Adam Lerer, Joseph H Davis, and Bonnie Berger. Exploring generative atomic models in eryo-em reconstruction. arXiv preprint 137.v22000.01331, 2021.
15

---

Appendix
A Architectural Details
Each manifold of density maps V(, 0x) is implemented as neural network. Conditioned on conformation 2 R 0(:;0)): R - IR represents the Hartley transform of the 3D electron scattering potential of single particle. The frequency coordinate k E [-0.5, 0.5] expanded in sinusoidal basis using Fourier features [56) (64 base frequencies are randomly sampled from 3D Gaussian distribution of standard deviation 0.5). The neural network contains 3 hidden residual layers [6] of size 128 with ReLU nonlincarities, without any normalization schemes.
B Pose Estimation
"Hierarchical pose search" (HPS) is done on predefined grid over SO(3) x IR² (4,608 rotations and 49 translations 1-10 pix., 10 pix.). The first poses minimizing the reprojection error are kept and refined with local search over 8 neighbors during 4 additional steps. The images are band- limited during pose search and the cutoff frequency increases linearly from kimin to Aima (kimin - Amax 16 in (image length) ")). Grids parameterized using the Hopf fibration [67], product of the Healpix [14] grid on the 2-sphere and a regular grid on the circle.
Once 2,000,000 images have been fed to the model (with a minimum of 2 epochs of pose search), the pose estimation strategy switches stochastic gradient descent (SGD). The poses di.k are initialized with the last poses estimated by hierarchical pose search.
C Conformation Estimation
The conformations Φi are independently optimized by SGD. They are initialized randomly from d-dimensional Gaussian distribution of standard deviation 0.1. Unless stated otherwise, the dimension d of the conformations is 2.
D Score Estimation
The scores E R are optimized by SGD and converted into probability vectors of dimension K with softmax operator.
E Optimization Parameters
We use the Adam optimizer 23 without weight decay and with the following learning rates: for the scores, 0.01 for the conformations, 0.001 for the poses, and 0.0001 for the weights of the neural networks.
F Synthetic Datasets
Simulation of compositional heterogeneity. We first describe the construction of tomotwin3, the synthetic dataset with compositional heterogeneity only. We selected the 6th, 7th, and 8th largest proteins by atomic weight from the training dataset used in TomoTwin
[48). which correspond to entries 6up6, 6id1, and 4cr2 in the RCSB PDB (Protein Data Bank) 3 We simulate density maps
using grid spacing of 1.5 Å/px and resolution of 3.0 7/px. We pad each density map to box size of D = 384 pixels and center it. We sample 3,000 orientations uniformly from SO(3), and 3,000 translation vectors 30-pixel- box around the origin (1,000 poses for each PDB entry). We project each density map by applying corresponding orientation, projecting using Eq. and applying the corresponding translation vector. We apply contrast transfer function (CTF) uniformly sampled from EMPIAR-11247 [10) representative cryo-EM dataset. We add Gaussian noise to each image to reach signal-to-noise ratio (SNR) of 0.01. We downsample each image to image size of D 128 pixels.
for each entry using the ChimeraX molmap command 30 58
16

---

We evaluated Hydra on tomotwin3 with 2-dimensional conformational space. We perform HPS on 100,000 images (33 epochs), followed by 100 epochs of SGD pose optimization. We set the batch size to 64 during SGD pose optimization. The score table learning rate is set to 0.01, and a is set to 0.1. We evaluate Hydra using both the correct number of classes (K' = 3) and an overparametrized configuration (K" = 5). All other parameters are set to default values.
We train CryoDRGN2 v3.3.0 for 30 epochs using an 8-dimensional latent space, an encoder width of 1024, 3 encoder layers, and a decoder width of 1024, We evaluate DRGN-AI using an 8-dimensional conformational space. 100k images (33 epochs) of HPS followed by 100 epochs of SGD pose estimation. cryoSPARC ab initio was run with K = 3 classes, All other parameters are set to default values.
When calculating volume metrics, we compare the reconstructed density map to a downsampled ground truth density map (D = 128 pixels). All experiments on tomotwin3 are run one NVIDIA A100 GPU. Our experiments required 4h00min for K = 3 and 6h20min for K = 5. DRGN-AI ran in 1h20min and cryoDRGN2 in 1h50min.
Simulation of conformational and compositional heterogeneity. For the pre-catalytic spliceosome, we obtain a trained cryoDRGN model from Zenode and generate 500 density maps of box size D=256 at equally spaced points along the first principal component of the latent space [4] [69]. For the 80S ribosome, 500 density maps of box size D=256 at equally spaced points along a linear trajectory connecting the embeddings of particles 34570 and 60629 in the latent space [64 69]. For the SARS-CoV-2 spike protein, a cryoDRGN model was trained on a filtered subset of the dataset of Walls et al. [62],
likewise obtain trained cryoDRGN model from Zenodo and generate
for 25 epochs with circular image mask of dimension 64, latent dimension of 8, and all other hyperparameters at default. Then, we generate 500 density maps of box size D=256 at equally spaced points along the first principal component of the latent space [61].
We next match the pixel sizes of the density maps by standardizing them to that of the spliceosome. In particular, we downsample the ribosome density maps to D=228 and pad back to D=256, and downsample the spike protein density maps to D=198 and pad back to D=256. Then, we generate and apply soft mask for each density map, at thresholds of 0.03, 0.05, and 0.1 for the spliceosome, ribosome, and spike protein, respectively, and with 25A of dilation and 15A cosine edge. Next, we normalize density map intensity values such that the signal ratios between different particle types match approximate "true" signal ratios. We calculate the true signal ratios by running the ChimeraX molmap command on the PDBs of each particle (spliceosome SNRL, ribosome 3J79, spike 6VXX and 6V YB), summing the density within each density map, and calculating signal ratios (where the signal of the spike protein is taken to be the average over the two PDBs). Then, the intensities of the 500 ribosome density maps and 500 spike protein density maps are scaled such that the ratios of total intensities across all density maps for ribosome to spliceosome, and spike to spliceosome, match the true ratios.
Finally, we generate images from all the density maps. For each density map, 100 projection images are generated with uniformly sampled rotations from SO(3) and uniformly sampled in- plane translations within [-20, 20] pixels. CTF is applied to each image, with values drawn (with replacement) from the experimental CTF values of the spliceosome dataset [4I. Noise is added to all images at an SNR of 0.1, with the variance of the signal computed over all 15k particles. Lastly, the images are downsampled from D=256 to D=128
We train Hydra for 100,000 images of HPS before 100 epochs of SGD. We train cryoDRGN2 for 90 epochs. All other hyperparameters for all methods are set to their defaults. For single-class DRGN-AI and cryoDRGN2, the predicted classes for Adjusted Rand Index calculation are obtained by k-means clustering the latent space with k=3. Image-FSC is computed over 50 images equally spaced along the true conformational trajectory, per each of the particle types (150 images total). Experiments for Hydra were run using 2 NVIDIA V100 GPUs, and experiments for baselines were run using I NVIDIA V100 GPU.
httpps//zancdooorg/eeertt//////
17

---

G Real Dataset
Dataset details. 85,656 particles were manually picked from one round of 2D classification on cryoSPARC of larger dataset of 148,596 particles. The particles were downsampled from box size of = 300 pixels and 1.66 A/pixel to D = 150 pixels at 3.32 A/pixel.
DRGN-AI and Hydra. All training for DRGN-AI and Hydra were carried out A100 NVIDIA GPUs with 80GB memory. For ab initio DRGN-AI, we ran DRGN-AI with default parameters (latent dimension d 8), with 500k images for HPS followed by 100 epochs of SGD. We verified that single-class DRGN-AI could separate the RyR and non-RyR particles by selecting the particles belonging to the non-RyR cluster (as visualized in the latent space) for downstream single-class DRGN-AI analysis, and a second run of single-class DRGN-AL with default parameters, 500k images of HPS and 100 epochs SGD on the non-RyR particles recovered p97 density map and non-p97 cluster in the latent space (followed by a third round of DRGN-AI with the same parameters on the non-p97 particles). For Hydra, we trained the model with million images for HPS followed by 100 epochs of SGD and latent dimension d = 2. After an initial sweep of the hyperparameters for learning rate and high-entropy prior using K 3 or K = 4, the optimal parameters for this dataset were determined to be a learning rate of 0.1 and c = 10.0. In order to reduce the resource demand of Hydra, we lowered the hypervolume dimension to 128 and reduced the number of hidden layers in the hypervolume to 2, well as decreased the SGD batch size to 64. For fixed-pose DRGN-AL with consensus cryoSPARC poses, poses from cryoSPARC homogeneous refinement job were used as input for optimization by single-class DRGN-AI with default parameters.
CryoSPARC processing. We performed sweep of K-values for 3D classification using cryoSPARC ab initio from K = 1 through K = 8 (all other parameters set to default values), and determined that K 5 is insufficient to separate the junk particles from protein. K 6 ab initio yielded the best separation of particles into the four different classes, with three different RyR density maps produced. We performed heterogeneous refinement on the best ab initio K 6 replicate for comparison to Hydra. In accordance with best practices for data processing, we performed ab initio with the downsampled D 150 particles but used undownsampled particles boxed D 300 to generate the most accurate poses during refinement against the low-resolution density maps generated by ab initio.
H Additional Files
We provide the following supplementary files:
Three movies illustrating the continuous motion of the ribosome, spliceosome and spike, as shown in Figure 4 The movies were obtained by sampling 9 points on linear trajectory of the latent space, for each compositional state.
The 10 volumes shown in Figure for the ribosome and the spliceosome, mrc format.
18

---

I
Additional Figures
(e)
Fig. S1: 25 representative sample images from each of the referenced three datasets. (a) Sample images for the tomotwin3 synthetic dataset, 128, A/pix. (b) Sample images for the experimental ryanodine receptor dataset, D 150, 3.32 A/pix. (c) Sample images for the ribosplike synthetic dataset, D 128, 4.24 A/pix.
Reconstructed Maps Class
Predicted Class
Class
Class
Ground Truth Label
3
Total
6up6
1.000
1,000
1,000
1,000
Total 1.000
1.000
1,000
3,000
Fig. S2: Results on the tomotwin dataset (3k particles) with a larger-than-oppe than ohammll value for K (K = 7). Reconstructed maps the left and confusion matrix on the right. Hydra is able to accurately reconstruct the three states in the dataset but four classes end up empty.
Reconstructed Maps
Predicted Class
Class
Class
Class
Ground Truth Label
2
Total
6up6
10,000
0
10.000
61d1
10,000
0
0
10,000
4cr2
0
10,000
10.000
Total
10,000
10,000
10,000
30.000
Fig. S3: Results on the tomotwin datasct (30k particles, K = 3).
19

---

(a)
CryoSPARC replicate labels
CryoSARRC repiicate 2
RyR
p97
Cill
Junk
Total
RyR
60,464
60,591
7,714
7,896
19
80
9,492
1,501
11,092
Junk
52
98
1,305
4,622
6,077
Total
60,538
7,900
10,917
6,301
85656
(b) cryoSPARC ab-initio replicate 1
(c) cryoSPARC ab-initio replicate 2
Fig. S4: cryoSPARC ab initio K 6 shows comparable levels of classification uncertainty Hydra K 4, with cryoSPARC initio showing significant classification uncertainty between particles from the CIII and junk classes, similar to the comparison between Hydra class assignments and cryoSPARC ab initio reference labels. (a) Confusion matrix between two K = 6 replicates of cryoSPARC ab initio. (b) and (c): reconstructed density maps from the cryoSPARC ab initio replicates.
-30
-2.5
0.0
2.5
5.0
75
PC1 00.33)
Fig. 85: Sampling of the DRGN-AT latent space from the ryanodine receptor dataset shows two main
clusters corresponding to RyR and non-RyR particles. Left: density maps sampled using k-means clustering with K 20 on the latent space; right: latent space plots.
20

---

(a)
p97
RyR
Cill
Junk
-1.
PC1
(b)
(c)
Cryustaac labels for DRGN-AI' CII
Cryoshwac labels for DAGN-WI junk
05
65-
02
experimental ryanodine receptor dataset. (a) Latent space plots corresponding each class from Hydra. (b) Additional top and views of each density map generated from Hydra K 4. with a cutaway view of CIII showing resolution of transmembrane helices. (c) Bar plots showing agreement between class assignments for Hydra K 4 and cryoSPARC K = 6 heterogeneous refinement (particles from the three recovered RyR classes were combined for analysis). normalized by total number of particles in each Hydra class.
Fig. S6: Additional qualitative information for Hydra
21

---

(a) cryoSPARC homogeneous refinement
CSPSC Rocee sdun:: 5.x44
L0
(b) DRGN-AI fixed pose
Fig. S7: The typical processing workflow of generating consensus reconstruction followed by DRGN-AI heterogeneous reconstruction fails to capture the shape of non-RyR densities, as the cryoSPARC consensus reconstruction conceals compositional heterogeneity and yields a high-resolution density for RyR only. (a) Homogeneous refinement of the entire ryanodine receptor dataset against cryoSPARC ab initio K = 1 aligmment of the entire dataset (left); right: FSC curve. (b) single-class DRGN-AT fixed pose with poses from the cryoSPARC homogeneous refinement; left: densities from k-means 20 sampling the latent space; right: latent space plots.
(a) DRGN-A round
(b) DRGN-AI round 2
(c) DRGN-AI round 3
25,396 non-RyR particles selected for downstteam processing
80,260 RyR particies
6,905 nom-g97 particles selected for downstream processing
18,491 #97 partiless
Fig. S8: Even when using a multi-shot heterogencous reconstruction approach, DRGN-AI is only able capture RyR and p97 densities, possibly due the higher SNR of RyR and p97; the final density shows mix CIII and junk particles and no partitioning the latent spacc. (a) DRGN-AI on the full experimental ryanodine receptor datasct; resulting RyR density generated from sampling the latent space cluster labeled in blue. (b) DRGN-AI Al on the subset of particles from the latent space cluster labeled in orange from part (a); p97 density sampled from latent space cluster labeled in blue. (c) DRGN-A Al on the subset of particles from the latent space cluster labeled orange from part (b).
22

---

(a)
CryoDRGN2
UMAP1
DRGN-AI
UMAP2
UMAP1
(c)
CryoSPARC
Particle Count
5000
4859
5141
Class
Fig. S9: Qualitative results of baselines on the synthetic ribosplike dataset containing compositional and conformational heterogeneity. (a)-(b) Conformational space plots and representative K-Means elustering volumes CryoDRGN2 and single-class DRGN-AL. (c) Class volumes particle counts CryoSPARC.
23

---

J Additional Tables
All
PDB 6up6
PDB 6id1
PDB 4cr2
Model
ARI
Image-FSC
Image-FSC
Image-FSC
Replica
1.00
0.24 0.03
0.393 0.001
0.364 0.001
Hydra K 3
Replica 2
1.00
0.25 + 0.03
0.394 = 0.001
0.367 + 0 0.001
Replica 3
0.75
0.15 + 0.02
0.388 = 0.001
0.367 + 0.001
Replica
1.00
0.25 + 0.03
0.396 = 0.001
0.367 + 0.001
Hydra K = 5
Replica 2
1.00
0.27 +: 0.02
0.394 + 0.001
0.363 1 0.001
Replica
0.89
0.18 1 0.02
0.396 0.001
0.365 1 0.001
Replica 1
0.59
0.0242 + 0.003
0.0396 = 0.005
0.042 + 0.004
DRGN-AI K =
Replica
0.49
0.022 0.002
0.040 0.004
0.040 + 0.004
Replica 0.53
0.026 0.005
0.038 = 0.004
0.041 +: 0.005
CryoDRGN2 CryoSPARC
0.36
0.08 0.01
0.070 0.006
+ 0.1
1.00
0.284
0.367
0.338
Table 3: Extension of Table 1 The classification accuracy is evaluated for each method using the adjusted Rand index (ARI. To evaluate each method's reconstruction quality, we use the mean area under the Fourier shell correlation (FSC) curve for 20 images per class (we report =: standard deviation). We bold the best result, and underline the best result for the second best method.
Rotation Error +
Translation Error
4
Resolution 0.5 FSC f
6up6
6id1
4cr2
6up6
6id1
4cr2
6up6
6id1
4cr2
122.73
80.07
1.21
4.754
19.144
0.005
44.31
82.29
9.14
114,82
1.19
1.23
2,936
0.022
0.013
9.29
9.14
9.14
5
123.43
1.18
1.22
2.812
0.020
0.016
9.29
9.14
9.14
Table 4: Quantitative results on the tomotwin dataset (30k particles) for Hydra with varying K. Metries include median rotational errors (in degrees), median translation errors (in pixels), and resolution in Á FSC cutoff of 0.5, where compare the ground truth volume backprojected volume using the predicted poses (Nyquist limit is at 9.00 Å). We note high pose errors for structure 6up6 despite its good reconstruction quality, likely indicative of pose ambiguity induced by the symmetry of the molecule.
Rotation Error
Translation
Per-Image 1
Method
Splice
Ribo
Spike
Splice
Riho
Spike
Splice
Ribo
Spile
Hydra (K = 3)
0.997
1.69
0.67
0.85
0.141
0.003
0.003
0.373
0.429
0.441
(RG
0.353
CryoDRGN2
0.986
2,06
0.59
0.99
0.611
0.006
0.011
0.357
0.424
0.416
CrysPARCC
0.972
1.94
1.34
1.45
0.367
0.018
0.020
0.324
0.355
0.353
Table 5: Quantitative results on the ribosplike ike dataset. Metrics include particle classification accuracy (Adjusted Rand Index, ARI), median rotational error (in degrees), median translation error (in pixels), and reconstruction quality (per-image area under the FSC curve). Hydra outperforms state-of the-an art methods jointly capturing conformational and compositional heterogeneity.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduce a new neural-based model to cope with compositional and conformational heterogeneity in cryo-EM. Our contributions are supported by results on synthetic and experimental datasets.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are mentioned in the discussion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer: [NA]
Justification:
The paper does not introduce theoretical results requiring proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide implementation details in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
26

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
Inttps:/hydda.csr cs.
wwwp://aarcc....ccoooooooooom princeton.edi edu/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "pipsscc/puulic//uui /ttpe//ggu..eeeoooouuuummbibbi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details are provided the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report the results obtained with 3 replica for each experiment in Table 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
27

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide this information in experiments section and in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/neurips. /
Answer: [Yes]
Justification: To the best of the authors' knowledge, the paper respects the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: We do foresee any potential malicious applications uses of the work described in this paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The structures from the PDB are properly credited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
29

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "LcSDOLsR6N", "reviewer": "Reviewer_u3f9", "summary": "This submission presents Hydra, a method for handling heterogeneous cryo-EM reconstruction. Hydra can model both conformational and compositional heterogeneity and can perform ab initio reconstruction. To achieve this, it parameterizes structures as arising from one of K neural fields. In the optimization pipeline, the conformations, poses, class probabilities, and neural fields are optimized to maximize the likelihood of the observed images. The authors demonstrate the reconstruction of multiple protein complexes from an experimental dataset.", "questions": "1. How should the value of K be determined? How would the proposed model behave if K is too large?\n2. Will the multiple neural representation design lead to excessive partitioning, that is, splitting complete independent structures into unreasonable parts?\n3. How does the proposed approach compare to conventional methods like cryoSPARC and RELION in terms of reconstruction quality and efficiency?", "limitations": "n/a", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed approach can handle ab initio reconstruction, meaning it does not require pre-computed image poses.\n2. The capability of handling compositional heterogeneity is not well-explored.\n3. Results on experimental datasets are provided, with comparisons to baseline methods.", "weaknesses": "1. Compared to DRGN-AI, the proposed approach primarily changes the single neural representation to multiple ones.\n2. The determination of K seems tricky.\n3. The paper lacks comparisons with conventional approaches such as cryoSPARC and RELION, especially qualitative comparisons. More results, preferably video results, are needed.\n4. It is unclear whether the pose predictions are accurate. More visualization analysis would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank our reviewer for their comments. We hope to address their questions and concerns in the following response.\n\n**Distinction with DRGN-AI**\n\nHydra uses the same pose search strategy and autodecoding framework as DRGN-AI. It primarily differs by using several neural networks, “latent scores” (L197) and a new loss function involving a marginalization over the possible classes (Eq 7). By doing so, Hydra broadens the scope of datasets for cryo-EM reconstruction. In particular, we demonstrate that, unlike Hydra, DRGN-AI fails at processing datasets with strong compositional heterogeneity, both on synthetic (Fig 2.a, S7.b) and experimental (Fig 3.d, S2, S6) datasets.\n\nWe will clarify, in the description of the method, what elements are borrowed from DRGN-AI.\n\n**Determination of $K$**\n\nThe optimal value of $K$ can be obtained by running Hydra several times with increasing values for $K$. Although choosing a larger-than-optimal value for $K$ would lead to unnecessary computation, we find that the quality of the reconstruction does not degrade when $K$ is too large, as shown in Fig 2 (b vs. c). We provide additional results using $K=7$ on the tomotwin dataset in Fig A1 (see document attached to the shared rebuttal).\n\nWe acknowledge that this sweeping procedure requires time, energy and memory and could be avoided with a principled way of choosing $K$. However, other conventional approaches for heterogeneous reconstruction (cryoSPARC, RELION) are limited in the same way, while not being able to handle conformational heterogeneity (Fig S7.c). This limitation is currently mentioned in the Discussion section (L311-317), and potential mitigation strategies are suggested.\n\n**Comparison to Conventional Approaches**\n\nHydra is qualitatively compared to cryoSPARC on the experimental dataset (Fig S5) and on the synthetic ribosplike dataset (Fig S7). We thank our reviewer for suggesting to provide access to the reconstructed maps. We will follow this suggestion and add the maps reconstructed with cryoSPARC, DRGN-AI and Hydra to the supplementary material, in the _mrc_ format.\n\n**Metrics on Pose Estimation**\n\nTo better assess the accuracy of pose estimation, we provide both translation and rotation errors for the tomotwin (Table A1) and ribosplike (Table A2) datasets. For rotation errors, we use the geodesic distance in $SO(3)$ (angular error), in degrees, which is a more interpretable metric than the Frobenius norm. We will add these metrics to the supplementary material.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BsW1LUkEgf", "reviewer": "Reviewer_RoiP", "summary": "The paper presents a neural network-based methodology for modeling both compositional and conformational protein states in cryo-electron microscopy (Cryo-EM) 3D reconstruction. \n\nIn particular, the authors propose a fully *ab initio* approach, named Hydra, which enables the joint inference of poses, conformations, and class identities.\n\nThe novelty of this approach lies in its ability to capture both discrete (compositional) and continuous (conformational) heterogeneity within Cryo-EM datasets, without relying on pre-computed pose estimations. Previous methods have either struggled with accurate pose estimation, relied on coarse initializations and upstream algorithms, or had limited capacity to represent complex biomolecular mixtures.\n\nMoreover, the authors validate their proposed approach by comparing it against other popular methodologies, using both synthetic and real datasets, showing the potential of Hydra to capture both compositional and conformational heterogeneity.", "questions": "- Do you think that the choice of using only 2 dimensions to represent the conformational space in Hydra may have affected the performance of your method? I think that the choice is excellent for increasing the interpretability of results since it eliminates the need to use techniques like PCA to visually understand the learned latent space. My intuition is that thanks to the K neural fields, the conformal representation may be better partitioned in lower dimensional spaces, but I wonder if you also have considered a higher number of dimensions.\n\n- In Table 2, both Img-FSC and Pose errors are reported. It is unclear to me if the pose error and the Image FSC may be somehow correlated, as, intuitively, a wrong estimated pose could increase the reconstruction error. If that was the case, then I would be surprised to see such a small gap in Img-FSC with respect to CryoDRGN2 compared to the improvement in pose error.\n\n- In Table 2, Hydra shows the lowest Pose error, while DRGN-AI shows the highest despite using a similar pose estimation approach. Can this performance difference be ascribed to the mixture model, or there are other factors to take into account, e.g. the dimensionality of the conformational space?", "limitations": "- The main limitation of this work is the need to know the number of classes K in advance. This can be addressed by an exhaustive search of the optimal value of K, however, this further worsens the second main limitation that is related to the computational cost of the method. The authors address both limitations and propose a possible future direction to reduce computational cost.\n\n- Another point of view of the previous limitation is related to scalability. Given a fixed computational budget, since a new neural field is required for each additional protein class, the possible choice of the number of desired classes K is limited, potentially reducing the applicability of the method in datasets that contain a higher number of different macromolecules.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper proposes a novel approach that improves over previous methods. In particular, the proposed method extends previous work, DRGN-AI, to use mixture models of multiple neural fields instead of a single neural field to model conformal heterogeneity. In addition, the proposed approach can directly classify the reconstructed sample between 1 of K different classes, providing advantages over methods that rely on downstream classification tasks.  Bibliographic references are exhaustive and well-discussed. The provided results highlight the significance of the method, showing substantial improvements over three pre-existing methods.", "weaknesses": "Although the manuscript is generally well-written, it may be hard to follow for someone who is not an expert in the field. In particular, there are some areas where accessibility to a broader readership might be possible:\n* For me, reading some previous work was necessary to understand the context of this work sufficiently to appreciate and understand the contributions of this manuscript. A clearer introduction tailored to a broader readership could make this submission more self-contained, which I would find desirable.\n* The introduction lacks a straightforward definition of the taxonomy used throughout the paper. While the authors introduce the context of their research, in particular regarding cryo-EM, I think that the reader should be introduced to the concepts of “poses”, “conformational states”, and “compositional states”, and then to why they are relevant to the presented research and future users. Casting this into a concise but clear way will be much appreciated by future readers, I believe. After that, the authors address how their technical contributions addressed the main challenges presented to them. In my opinion, such changes will allow non EM experts to appreciate the presented work much better and potentially allow other fields to benefit from the same/similar ideas and methods.\n* The proposed method seems to rely heavily on the DRGN-AI approach from Levy et. al. Although the authors explicitly state in the introduction that this paper represents an extension of that method, throughout the paper it’s not always clear whether the methodological choices described are novel contributions or are unaltered from the previous method. For example, in Section 3.4, the sentence “We use the pose estimation strategy introduced in [25]” may lead the reader to think that the authors are experimenting with a new pose estimation approach from the literature, while it was already used in the DRGN-AI (or at least I believe so). \n* While results are clear and easy to follow, they only present standard deviations in Table 1 and not in the other tables. Moreover, it is unclear why the CryoSPARC result in Table 1 does not report any standard deviation.\n* Will the authors provide a public code repository enabling others to replicate the presented results and use the method in the context of their data and experimentation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank our reviewer for their constructive feedback and suggestions on ways to improve the clarity of the manuscript.\n\n**Clarification of Prior Work and Contributions**\n\nWe thank our reviewer for pointing out the lack of clarity in the presentation of prior works and apologize for the absence of important context. We will review our introduction and make an effort to make it clear for a broader audience.\n\n**Clarification of Field-Specific Terminology**\n\nAgain, we apologize for the oversight on using field-specific terminology (pose, conformational/compositional heterogeneity). We will clarify these terms in order to allow non-cryo-EM experts to appreciate the presented work and potentially borrow from its ideas.\n\n**Distinction with DRGN-AI**\n\nHydra uses the pose search strategy and the autodecoding framework from DRGN-AI. It primarily differs by using several neural networks, “latent scores” (L197) and a new loss function involving a marginalization over the possible classes (Eq 7). By doing so, Hydra broadens the scope of datasets for cryo-EM reconstruction methods. As suggested by our reviewer, we will clarify that we use the pose estimation strategy in DRGN-AI and that it is adapted to cope with the simultaneous representation of multiple maps.\n\n**Standard Deviations in Table 1**\n\nThe standard deviations reported in Table 1 characterize the spread of the area under the Fourier shell correlation obtained over 20 images (with different conformational states $z$) per class. This can only be measured for neural-based methods (Hydra, DRGN-AI, cryoDRGN2), hence the absence of standard deviation for cryoSPARC, which outputs a single conformational state per class.\n\n**Code Release**\n\nYes, we will release our code upon publication.\n\n**Questions**\n\n- We agree with our reviewer’s intuition. The use of $K$ neural networks increases the expressivity of the representation and probably decreases the number of latent dimensions required to capture conformational motion. Empirically, we found that a dimension of two was sufficient to obtain accurate density maps, and to capture continuous motion in the ribosplike dataset (Fig 4). However, if Hydra had to process a dataset where one of the compositional states had strictly more than two degrees of freedom, the dimensions of the latent space would have to be increased. We hypothesize that this dimension must be greater or equal to the largest number of global degrees of freedom among all compositional states. We will add a brief discussion on the choice of the latent dimension in the last section.\n\n- Our reviewer’s intuition on the correlation between pose error and image-FSC is correct. In order to provide a clearer assessment of each method’s performance, we show updated metrics for the ribosplike dataset, including rotation and translation errors, in Table A2 (see document attached to the shared rebuttal).\n\n- To show that the poor performance of DRGN-AI on the ribosplike dataset is not linked to the low dimension of the latent space, we give eight dimensions to the latent space of DRGN-AI (L244). We therefore hypothesize that Hydra’s ability to reconstruct this dataset can be ascribed to the mixture model. We provide qualitative reconstructions and a UMAP plot of the conformations obtained with DRGN-AI in Fig S7.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tDCJBimFtL", "reviewer": "Reviewer_SenM", "summary": "This work describes a new method for ab initio heterogeneous reconstruction in cryo-EM using mixtures of neural fields. This generalizes previous approaches, such as CryoDRGN and DRGN-AI, which attempts to reconstruct 3D molecular densities using a single neural field representation. The resulting method is able to handle both compositional (discrete) and conformational (continuous) heterogeneity, with each mixture component handling the continuous variability of each distinct compositional state. The performance of the method is evaluated on two synthetic and one experimental dataset.", "questions": "– On line 49, please explain the principal difference between DRGN-AI and its predecessors. This is particularly important since the proposed method is quite closely related.\n– What is meant by “tackle the discrete heterogeneity and the continuous variability *in sequential order*”? Is this referring to clustering the data, estimating the pose, and then estimating the continuous variability for each cluster? This could perhaps be clarified.\n– Another approach to manifold embedding for continuous heterogeneity (line 82) is discussed in Moscovich et al., 2020.\n– Line 130 should have “used” instead of “use”.\n– The last line in the caption for Figure 1 is missing “Section” in front of “3.3”.\n– If the dimension d is the same for each component in the mixture (eq. 3), how does this work when the number of degrees of freedom varies with each discrete structure?\n– In eq. 4, σ_2 should be σ^2.\n– Please discuss the choice of using point estimates for the pose and latent variables in eq. 6. Why is this approximation reasonable? This can be especially sensitive in the ab initio case where the can be great uncertainty in the pose estimation stage. Why do we not need to account for this here?\n– Why use the area under FSC to evaluate the accuracy of the reconstruction (Table 1)? What are the advantages compared to gold-standard resolution estimates?\n– Does the “strategy” on line 216 refer to the pose (eq. 9) estimation or to the overall estimation of pose and other variables? In other words, do we run ~1e6 iterations of pose estimation of ~1e6 iterations of pose estimation alternated with SGD?\n– In line 227, what is meant by “a single pass”? The described algorithm seems to loop over the data several times over. Single-pass approaches usually refers to moment-based methods which pass over the data once to calculate certain statistics and then use those statistics to reconstruct the molecule.\n– What is the SNR of the images described in Section 4.3? This information is important enough that it should be supplied in the main text.\n– Please provide running times for the various experiments. The metric of 4 GPU-days is cited in the Discussion, but it is not clear how this relates to the number of images (is this for the 60 000-image dataset or the 3 000-image dataset?).", "limitations": "As stated above, the main limitation of the work is its lack of validation on experimental data (for both compositional and conformational variability). It is also not clear how computationally intensive the implementation is and how this can be mitigated.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This represents a natural and well-structured extension of previous neural-field approaches to cryo-EM reconstruction. The method is well-motivated and described with an appropriate level of detail. Finally, the numerical results verify many important aspects of the proposed method. Overall, the writing is clear and easy to understand.", "weaknesses": "The most important issue is the lack of experimental validation for the combined estimation of compositional and conformational heterogeneity. While this is tested in the third experiment (Section 4.3), this is only on a synthetic dataset. As the authors are no doubt aware, however, the behavior of a reconstruction algorithm can be quite different when applied to real data. It is therefore encouraging that the authors present results on an experimental dataset (Section 4.2), but this only covers compositional heterogeneity (and not conformational). That being said, validating the full method on an experimental dataset would make a stronger case for the proposed work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank our reviewer for their constructive comments and overall positive rating of our submission.\n\n**Validation on an Experimental Dataset Combining Compositional and Conformational Heterogeneity**\n\nAs mentioned by our reviewer, processing real cryo-EM data often comes with unforeseen and significant challenges. By demonstrating Hydra’s ability to process the “ryanodine receptor” (RyR) mixture dataset, we hope to provide evidence that our method can process experimental data, in spite of the challenges and non-idealities that come with it. We agree with our reviewer on the fact that demonstrating the unique capabilities of Hydra on an experimental dataset with mixed heterogeneity, and potentially revealing motion or complexes that could not be seen before, would constitute an exceptional scientific leap. By sharing our work (and our code), we hope to provide new capabilities to the cryo-EM community and enable it to make this leap.\n\n**Questions**\n\n- We will clarify the main differences between Hydra and DRGN-AI in L49: use of several neural networks, the presence of “latent scores” and a new loss function involving a marginalization over the possible classes.\n\n- We will clarify the term “sequential order” on L79. The explanation given by our reviewer is correct.\n\n- We will add a reference to Moscovich et al (Cryo-EM reconstruction of continuous heterogeneity by Laplacian spectral volumes, Inverse Problems, 2020) on L82. Thank you for pointing this out.\n\n- We will add the missing character on L130.\n\n- We will add “Section” in the caption of Fig 1.\n\n- In the current architecture, the conformational latent vectors are stored in a $N$-by-$d$-by-$K$ array, meaning that all classes have the same latent dimension. We hypothesize that the latent dimension $d$ must be greater or equal to the largest number of degrees of freedom among all compositional states. However, it would be possible to use per-class dimensions $d_k$, for example using a dictionary of $K$ arrays of dimensions $N$-by-$d_k$. We will mention this in the Discussion section.\n\n- We will fix the typo in Eq 4, thank you for pointing it out.\n\n- For poses, the point estimates are only used in the second phase of optimization (the Stochastic Gradient Descent phase). During the first phase (Hierarchical Pose Search), poses are exhaustively searched over using the strategy described in Section B (supplementary material). This HPS phase enables Hydra to perform _ab initio_ reconstruction.\n\n- The exhaustive search (Eq 9) is only applied to poses on a predefined number of images (~1e6). The schedule describing the switch from HPS to SGD is further described in L869-871 (supplementary material).\n\n- Following our reviewer’s suggestion, we report per-class resolutions at an FSC cutoff of 0.5 for the tomotwin dataset in Table A1 (see document attached to shared rebuttal). We note that a per-image FSC metric is better suited for the ribosplike dataset as it accounts for conformational differences in the true structural distribution. \n\n- We recognize that the term “single pass” is misleading in L227. We will replace it with “single run”.\n\n- The SNR for the dataset described in Section 4.3 is 0.1. We will add this information to the supplementary information, where the generation protocol is described (Section F).\n\n- The mention of “4 GPU-days” relates to the experimental dataset (85k images). The 3,000 image dataset tomotwin3 was processed using a single NVIDIA A100 GPU and required 4h when $K=3$ and 6h20min when $K=5$. DRGN-AI required 1h20min. We will clarify this point in the supplement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s4PGATrl3V", "reviewer": "Reviewer_14wr", "summary": "In this paper, author(s) porpose Hydra, an *ab initio* approach to model conformational and compositional heterogeneity. They ahieve this by parameterizing structures of proteins as a mixture of *K* neural fields.", "questions": "How would Hydra perform on EMPIAR-10076, one widely used dataset with complex compositional heterogeneity?", "limitations": "Authors discussed the limitation in their paper.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality:\n\n- Authors propose to incorporate neural network ensemble with recent *ab initio* reconstruction method to enhance the ability to model complex heterogeneity.\n\nQuality:\n\n- Through qualitative experiments, authors demonstrate the method is capable of clearly distinguish different compositional states in the synthetic dataset.\n- Hydra is able to identify both compositional and conformational heterogeneity in real datasets.\n- The method exhibits better quantitative results compared to existing *ab initio* reconstruction methods.", "weaknesses": "Novelty:\n\n- The hierarchical pose search (HPS) method for pose estimation is proposed in DRGN-AI [1].\n- Using an ensemble of representations to model heterogeneity in protein cryo-EM reconstruction has been adopted in many previous works [2][3].\n\n[1] Levy, Axel, et al. \"Revealing biomolecular structure and motion with neural ab initio cryo-EM reconstruction.\" *bioRxiv* (2024): 2024-05.  \n[2] Punjani, Ali, and David J. Fleet. \"3D variability analysis: Resolving continuous flexibility and discrete heterogeneity from single particle cryo-EM.\" *Journal of structural biology* 213.2 (2021): 107702.  \n[3] Kimanius, Dari, Kiarash Jamali, and Sjors Scheres. \"Sparse Fourier backpropagation in cryo-EM reconstruction.\" *Advances in Neural Information Processing Systems* 35 (2022): 12395-12408.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank our reviewer for their comments. We hope to address them in the following response.\n\n**Distinction with Previous Works**\n\nHydra uses the pose search strategy and autodecoding framework from DRGN-AI [1]. It primarily differs by using several neural networks, “latent scores” (L197) and a new loss function involving a marginalization over the possible classes (Eq 7). Compared to DRGN-AI, Hydra broadens the scope of datasets that can be handled. We will clarify that the pose estimation strategy was already used in DRGN-AI (it is only adapted to consider the simultaneous representation of several classes).\n\nWe thank our reviewer for pointing out the relevant reference [3], which we will cite. Although the references [2] (3DVA) and [3] reconstruct a combination of 3D arrays, those are not density maps but rather vectors with the same dimension as density maps, representing the basis of a low-dimensional linear space. The conformation heterogeneity is then represented in the linear space spanned by these vectors (the “structural basis” [3]). Unlike Hydra, 3DVA (cited on L34 and L76) does not handle compositional heterogeneity (L38).\n\n**EMPIAR-10076**\n\nWe thank our reviewer for this suggestion. EMPIAR-10076 (assembling ribosome) would indeed be a relevant dataset to demonstrate our method. We will run additional experiments on this dataset and try to include results in the supplement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ppNrDSP9JH", "reviewer": "Reviewer_qUkh", "summary": "This paper introduces a novel method, Hydra, for ab initio heterogeneous cryo-EM reconstruction. Different from existing approaches, Hydra separately models conformational and compositional heterogeneity by integrating K-parameterized neural fields to represent cryo-EM density maps. Furthermore, Hydra employs a hybrid optimization strategy to optimize particle poses, heterogeneity, and density map representations concurrently. The authors assess the efficacy of Hydra on three datasets comprising various protein complexes (two synthetic, one experimental). Extensive experimental results indicate that Hydra outperforms three baseline methods regarding reconstruction quality, particle classification, and pose estimation accuracy.", "questions": "My major concerns have been listed in the weakness section in this review, along with some of my confusion and questions. I list the important ones:\n- Why is the synthetic dataset limited to 3000  particle images in a very low SNR of 0.01 for the tomotwin3 dataset? In my experience, this limited number usually causes poor reconstruction results.\n- Can you provide the 3D resolutions of the reconstruction results to facilitate a more quantitative assessment?\n- Have you tested how Hydra handles classes with a small ratio of particle numbers, given that the combination ratio of the three types of particles is not explored?\n- The differences between the various states in Figure 4 are very subtle. How do you ensure that the observed surface changes are due to different conformations and not due to varying density map thresholds?\n- Could you include qualitative comparisons with cryoDRGN-AI and cryoDRGN2, as these methods also address conformational heterogeneity without explicit particle classification?\n- How do you sample the different conformation states in the conformational latent space? (like the white circular points in the latent space in Figure 4.)\n-The median Frobenius norm metric ignores translation errors. Could you also report the in-plane and out-of-plane angle errors, along with the translation error, similar to DRGN-AI?\n- What is the protocol for running CryoSPARC on a synthetic dataset, and what advantages does Hydra offer if CryoSPARC performs well in Table 1?\n- How is the runtime efficiency of the Hydra algorithm compared to other algorithms?", "limitations": "The authors have addressed the limitations of Hydra in the main paper.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "This paper aims to tackle ab-initio reconstruction – simultaneously estimating poses and reconstructing 3D structure, which is one of the most challenging problems in cryo-EM. The scope of the problem has been further extended to a more challenging setting by assuming the captured particle images exhibit the motions of structure and conformational heterogeneity. Thus, the problem setting is novel and challenging.\n\nThis paper proposes Hydra, the first ab-initio heterogeneous cryo-EM reconstruction method based on a mixture of neural networks to estimate conformational and compositional heterogeneity in the training process simultaneously.", "weaknesses": "**Synthetic Dataset**\n\nThe number of images in the tomotwin3 synthetic dataset is too small to match the real cryo-EM setting; there are only 3000 particle images with a very low SNR of 0.01 (tomotwin3). In a real scenario, there would be 50,000 to more than 100,000 particles with conformational variability. Without reporting the 3D resolutions of the results, it is very hard to quantitatively evaluate the reconstruction results. Additionally, this synthetic dataset lacks conformational variability, which hinders the evaluation of conformational variability recovery. The combination ratio of the three types of particles is not explored either; it remains unclear if Hydra would be sensitive to a class with only a small ratio of particle numbers. In Section 4.3, the dataset settings that include pre-catalytic spliceosome, 80S ribosome, and SARS-CoV-2 spike protein are unrealistic. To sum up, I recognize a significant gap between this synthetic dataset and real datasets, and I feel the experiment is not sufficient to evaluate Hydra adequately.\n\n**Reconstruction Quality Evaluation**\n\nFor qualitative evaluation, the differences between the various states in Figure 4 are very subtle, making it difficult to judge whether the surface changes are due to different conformations or the result of applying different thresholds to the density map. Also, I argue that cryoDRGN-AI and cryoDRGN2 also account for conformational heterogeneity without explicitly classifying particles; their qualitative results should also be compared in Figure 4.\n\nFor quantitative evaluation in Table 1 and Table 2, the authors only use Img-FSC to compare reconstruction quality. To the best of my knowledge, prior work such as cryoSPARC and cryoDRGN reports widely used 3D resolution calculated by thresholding the FSC curve between two half maps of the reconstruction for experimental datasets or between the reconstructed 3D density map and the synthetic ground truth density map (available for synthetic datasets).\n\n**Choice of Metrics for Pose Evaluation**\n\nThe authors use the median Frobenius norm as the metric to compare pose error, which ignores the translation part. Referring to DRGN-AI, the in-plane and out-of-plane angle error, along with the translation error, should be reported. Additionally, showing the angle distribution of poses for comparison could be beneficial.\n\n**Ablation Study**\n\nThis paper misses the ablation study of the number of K. I would like to know how to determine K in real cases. In this paper, the authors run CryoSPARC multiple times to determine the best K for H.\n\n**Miscellaneous**\n\n1.\tThe chirality of the reconstruction results for the pre-catalytic spliceosome in Figure 4 appears to be incorrect.\n2.\tThe construction of the ribosplike dataset involves a mixture of three different proteins. In real single-particle cryo-EM experiments, this scenario seems rare as the purified samples are carefully prepared and should not contain or only contain a very small ratio of undesired particles that can be easily filtered out in 2D classification. Maybe in cryo-ET, Hydra can perform one-for-all sub-tomo averaging?\n3.\tWhat’s the protocol to run CryoSPARC on a synthetic dataset? If it performs so well in Table 1, what is the advantage of Hydra?\n4.\tThis paper seems rushed. In Lines 57 – 62, the first three contributions have basically the same meaning; please consider rephrasing them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank our reviewer for their comments on the significance and the difficulty of the problem our method addresses. We hope to address their concerns in the following response.\n\n**Synthetic Datasets**\n\nTo further validate Hydra on the tomotwin dataset, we generated a larger version of this dataset (30,000 particles) with the same SNR. Results are shown in Figure A2 (see document attached to the shared rebuttal).\n\nFor synthetic data like tomotwin, we use the ground-truth maps to quantitatively evaluate the reconstruction results. Following our reviewer’s suggestions, we provide 3D resolutions for the larger tomotwin dataset (30,000 particles) in Table A1 of the attached document.\n\nThe tomotwin and ribosplike datasets are built with a uniform distribution over the three possible states, while the three states in the experimental dataset (RyR) are present in very different proportions (RyR: 71%, p97: 8%, CIII: 13%, junk: 8%, according to cryoSPARC and Hydra, Fig 3.b). We demonstrate Hydra’s ability to process all these datasets, confirming the possibility to handle different types of distributions over classes.\n\nWe acknowledge the existence of a gap between the SNR in synthetic datasets vs. experimental datasets. However, for synthetic datasets, we compare Hydra to baselines (cryoSPARC, cryoDRGN2, DRGN-AI) using the same level of noise and show an improvement in reconstruction quality (Table 1, Table 2, Fig S7). Our experiments on the real dataset validate Hydra’s ability to handle realistic noise levels and to outperform existing methods (Fig 3.b, S2, S6).\n\n**Reconstruction Quality Evaluation**\n\n- We will include movies of the states shown in Fig 4 to better show their conformational changes.\n\n- Per-image FSC measures per-particle 3D resolution on synthetic datasets (where ground truth maps are available) and is a standard metric for assessing the reconstruction quality of methods that can represent conformational (continuous) heterogeneity. We refer, for example, to Fig 3 and Supplementary Figure 1 of the cryoDRGN paper [1]. For the tomotwin dataset, which only contains compositional heterogeneity, we provide per-class 3D resolutions with a threshold at 0.5 FSC in Table A1 (see document attached to the shared rebuttal).\n\n[1] Zhong, E. D., Bepler, T., Berger, B., & Davis, J. H. (2021). CryoDRGN: reconstruction of heterogeneous cryo-EM structures using neural networks. Nature methods, 18(2), 176-185.\n\n**Choice of Metrics for Pose Evaluation**\n\nIn order to better assess pose accuracy, we provide per-class angular and translation errors for the tomotwin dataset in Table A1, and for the ribosplike dataset in Table A2 (see document attached to the shared rebuttal).\n\n\n**Ablation Study**\n\nFig 2 provides an analysis on the influence of $K$. When $K$ is too low (Fig 2.a), the reconstruction is inaccurate. When $K$ is too large, some of the capacity of the model is not used, but the reconstructed states are accurate (Fig 2.b). To further stress-test Hydra, we provide additional results using $K=7$ on the tomotwin dataset in Fig A1 (see document attached to the shared rebuttal).\n\nTo obtain the optimal value of $K$, one can run Hydra several times with increasing values for $K$. Although choosing a larger-than-optimal value for $K$ would lead to unnecessary computation, the quality of the reconstruction does not degrade when $K$ is too large, as shown in Fig 2 (b vs. c).\n\nWe acknowledge that this sweeping procedure requires time, energy and memory and could be avoided with a principled way of choosing $K$. This limitation is currently mentioned in the Discussion section (L311-317), and potential mitigation strategies are suggested.\n\n**Miscellaneous**\n\n- We thank our reviewer for pointing out the wrong chirality of the spliceosome in Fig 4. Since the chirality of a molecule is not identifiable from cryo-EM projections, we showed the chirality that we got out of Hydra without modification. We will flip (mirror) the density map to show the correct chirality.\n\n- While the choice of proteins in the ribosplike dataset may be unrealistic, having a mixture of different particle types in cryo-EM is not unrealistic, as evidenced by imaging of a lysate sample in the RyR experimental dataset. We agree with our reviewer: adapting this method to sub-tomogram averaging of _in situ_ mixtures is a promising avenue for future work (L320).\n\n- We always use the default parameters of cryoSPARC, both for synthetic and experimental datasets. We will mention this in the supplements.\n\n- Table 1 focuses on the tomotwin dataset, a synthetic dataset with strong compositional heterogeneity. The Table shows previous neural-based methods fail on this dataset. CryoSPARC performs well, but is unable to reconstruct conformational heterogeneity, which is quantitatively shown in Table 2.\n\n- We will clarify the contributions in L57-62.\n\n**Questions**\n\n- We thank our reviewer for suggesting to provide access to the reconstructed maps. We will follow this suggestion and add the reconstructed maps to the supplementary material, in the _mrc_ format. To make our method fully reproducible, we will provide access to the code upon publication.\n\n- We include qualitative comparisons to DRGN-AI and cryoDRGN2 in Fig S2, S4.b, S6 and S7.\n\n- In Fig 4, the conformational states are manually selected in the latent space. In other figures, we use k-means clustering (L247, L940, caption of Fig S2, S4, S7).\n\n- We evaluate the efficiency of Hydra by comparing GPU runtimes on the tomotwin dataset using a single NVIDIA A100 GPU. CryoDRGN2 required 1h50min. DRGN-AI required 1h20min. Hydra required 4h when $K=3$ and 6h20min when $K=5$. We will include this information on runtime efficiency in the supplemental.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all our reviewers for their detailed and constructive feedback. We value their appreciation of the significance of this work **[RoiP, qUkh]**, its novelty **[14wr]** and the validation of our method on experimental data **[u3f9, 14wr]**. We appreciate them highlighting the substantial improvements demonstrated by Hydra over pre-existing methods **[RoiP, 14wr]** and the relevance of our quantitative **[SenM]** and qualitative **[14wr]** results. We are also glad they emphasized the clarity of the manuscript **[SenM]**.

We provide additional figures and tables in the attached document.

Meta-review: This paper received mixed reviews.  The authors submitted a rebuttal with some additional experimental results and there was an active discussion period with many of the issues noted in the initial reviews being addressed.

The main issues raised were around datasets and novelty.  The issues around datasets were largely resolved and clarified during the discussion phase.  The question of novelty is somewhat subjective with two reviewers (RoiP and SenM) arguing that the work makes a sufficient contribution.  After reviewing the comments and the paper, the AC agrees with RoiP and SenM that the work presents a novel and significant improvement over existing work and therefore recommends acceptance.
Decision: Accept (poster)