A Graph Neural Network Approach to Automated Model Building in Cryo-EM Maps
Authors: Kiarash Jamali
Dari Kimanius
Sjors HW Scheres
Venue: ICLR 2023 poster (2023)
Keywords: cryo-em
model building
graph neural networks
attention networks
proteins

Abstract: Electron cryo-microscopy (cryo-EM) produces three-dimensional (3D) maps of the electrostatic potential of biological macromolecules, including proteins. At sufficient resolution, the cryo-EM maps, along with some knowledge about the imaged molecules, allow de novo atomic modelling. Typically, this is done through a laborious manual process. Recent advances in machine learning applications to protein structure prediction show potential for automating this process. Taking inspiration from these techniques, we have built ModelAngelo for automated model building of proteins in cryo-EM maps. ModelAngelo first uses a residual convolutional neural network (CNN) to initialize a graph representation with nodes assigned to individual amino acids of the proteins in the map and edges representing the protein chain. The graph is then refined with a graph neural network (GNN) that combines the cryo-EM data, the amino acid sequence data and prior knowledge about protein geometries. The GNN refines the geometry of the protein chain and classifies the amino acids for each of its nodes. The final graph is post-processed with a hidden Markov model (HMM) search to map each protein chain to entries in a user provided sequence file. Application to 28 test cases shows that ModelAngelo outperforms state-of-the-art and approximates manual building for cryo-EM maps with resolutions better than 3.5 A.

Published as a conference paper at ICLR 2023
A GRAPH NEURAL NETWORK APPROACH TO AUTO- MATED MODEL BUILDING IN ERYO-EM MAPS
Kiarash Jamali, Dari Kimanius, & Sjors H.W. Scheres MRC Laboratory of Molecular Biology Cambridge, UK {k [kjamall 1i, dari, scheres @ lmb. cam. ac. uk
ABSTRACT
Electron cryo-microscopy (cryo-EM) produces three-dimensional (3D) maps of the electrostatic potential of biological macromolecules, including proteins. Along with knowledge about the imaged molecules, cryo-EM maps allow de novo atomic modeling, which is typically done through laborious manual process. Taking inspiration from recent advances in machine learning applications to pro- tein structure prediction, we propose graph neural network (GNN) approach for the automated model building of proteins in cryo-EM maps. The GNN acts on graph with nodes assigned individual amino acids and edges representing the protein chain. Combining information from the voxel-based cryo-EM data, the amino acid sequence data, and prior knowledge about protein geometries, the GNN refines the geometry of the protein chain and classifies the amino acids for each of its nodes. Application
28 test cases shows that our approach outper- and approximates manual building for cryo-EM maps
forms the with resolutions better than 3.5 A
INTRODUCTION
Following rapid developments in microscopy hardware and image processing software, cryo-EM structure determination of biological macromolecules is now possible atomic resolution for favourable samples (Nakane et al.) [020 Yip et al. 2020). For many other samples, such as large multi-component complexes and membrane proteins, resolutions around Á are typical (Cheng 2018). Transmission electron microscopy images are taken of many copies of the same molecules, which are frozen in thin layer of vitreous ice. Dedicated software, like RELION (Scheres) 2012) or cryoSPARC (Punjani et al., (2017), implement iterative optimization algorithms to retrieve the orientation of each molecule and perform 3D reconstruction to obtain a voxel-based map of the underlying molecular structure.
Provided the eryo-EM map is of sufficient resolution, it is interpreted in terms of an atomic model of the corresponding molecules. Many samples contain only proteins; other samples also contain other biological molecules, like lipids or nucleic acids. Proteins are linear chains of amino acids or residues. There are twenty different canonical amino acids that make up proteins. All of these amino acids have four heavy (non-hydrogen) atoms that make up the protein's 's main chain. The different amino acids have different numbers, types, and geometrical arrangements of their side-chain atoms. The smallest amino acid, glycine, has no heavy side chain atoms; the largest amino acid, tryptophan, has ten heavy side chain atoms. Typical proteins range in size from tens to residues. Typically, the electron microscopist knows which protein sequences are present in the sample. The task at hand is build the atomic model, which identifies the positions of all atoms for all proteins that are present in the cryo-EM map. For each residue, there are two rotational degrees of freedom in the conformation of its main chain, the & and V angles. Distinct orientations of the side chains provide additional conformational possibilities, the number of which depends on the type of amino acid (figure I].
than a thousand
CCode and weights are open-source and can be accessed https://gg thub. com 3dem/
model 1o

---

Published as a conference paper at ICLR 2023
A
Bard
Vollee
Appraac
V
X
CA
T
R
Increasing residue number
Hooden
Threonine
Figure I: (A) shows peptide backbone where the chain is ordered from left to right. Arrows mark the peptide bonds between the atom one residue and the N atom of the next. (B) shows six amino acids, pointing out the differences in the side chains, marked by the outline. (C) shows the + and v angles of the backbone and the additional rotatable bond of the side chain for threonine.
Atomic model building in cryo-EM maps is typically done manually using 3D visualisation software, (e.g. [Emsley et al. 2010 Pettersen al., 202I). followed by refinement procedures that optimize the fit of the models in the map, (e.g. Murshudov et al. 2011 Croll 2018 Liebschner et al. 2019).
Often, in areas of weak density in the map, one cannot discern the amino acid identity of residues from the map alone and sequence information has to be used to make an accurate assessment. Man- ually building a reliable atomic model de novo the reconstructed cryo-EM map is considered to be difficult for maps with resolutions worse than 4 Å. Although the task is more straightforward for maps with resolutions better than 3 Å, it still typically requires large amounts of time and a high level of expertise.
Machine learning has recently achieved a major step forward in structure prediction for individual proteins (Jumper al.) 102 Back et al., 2021). In these approaches, the sequence information of proteins and their evolutionary related homologues is used to predict their atomic structure without the use of experimental data. In addition, protein language models, which are trained in unsu- pervised fashion on the amino acid sequences of many proteins, have also provided useful results in protein structure prediction (Lin et al. 2022, Wu al. 2022a). Although these techniques not yet capable of reliably predicting structures of the larger complexes that are typically studied by explore similar approaches for auto-
cryo-EM, their success for individual proteins inspired mated model building in eryo-EM maps.
In this paper, we present a single integrated GNN that combines the voxel-based information from the cryo-EM map with information from the protein sequence through protein language model, and information from the topology of the graph through invariant point attention (IPA) (Jumper et al., 2021). For 28 test cases, we demonstrate that our approach approximates the accuracy of manual model building for maps with resolutions better than 3.5 Å.
PRIOR WORK
Automated model building. Automated approaches for atomic modeling in the related experimen- tal technique of X-ray crystallography have existed for many years (for example, Perrakis et al.) 1999 Cowtan 2006 Terwilliger et al. 20088.. Some of these approaches have also been applied to cryo-EM maps. For example, the PHENIX package builds models that are on average 47% com- plete for cryo-EM maps with resolutions worse than 3 Å (Terwilliger et al. 2018). For similar maps, MAINMAST, an approach that was designed build C main-chain traces in cryo-EM maps, often produces models with root mean squared deviations (RMSDs) in the range of tens of Á (Terashi & Kihara 2018). Relatively incomplete models, with large residuals, have limited the impact of these techniques automated model building in the cryo-EM field thus far.
More recently, Deeptracer (Pfab et al. 202I), the first deep-learning approach for automated atomic modeling in cryo-EM maps, was reported to outperform these earlier approaches. Deeptracer uses U-Nets (Ronneberger et al. 2015) to construct an atomic model de novo in the cryo-EM map. In contrast to work, Deeptracer does not integrate the sequence information with the U-Net, and does not use a graph representation of the protein chain during model refinement. Instead, Deep- tracer treats the entire problem segmentation and classification problem. Thereby, it also does
2

---

Published as a conference paper at ICLR 2023
not have support for refining already built models performing multiple recycling steps. Although Deeptracer predicts amino acid types for each residue, it only builds atoms for the main chains.
There have also been reports dock and morph the output from protein structure prediction pro- grams, like AlphaFold2 (Jumper et al., 202T). to fit cryo-EM maps (He et al. 2022 Terwilliger ct al, 2022). Such approaches suffer when proteins change their conformation in complex with others and are likely to propagate errors in the structure prediction. Thus, it seems sensible to design a neu- ral network approach that integrates both the cryo-EM map and the sequence and protein structure primitives to produce a more reliable structure. This is the essence of our approach.
GNNs for proteins. A number of different approaches modeling proteins with GNNs have been proposed recently. This includes modeling the protein with torsion angles (Jing et al. 2022). SE(3) cquivariant graph neural networks (Ganea al. 2022). and SE(3) invariant GNNs (Dauparas al. 2022). In our approach, the ordering and connectivity of residues are unknown and have to be inferred, hence representations that require this to be known a priori, such as the torsion angle representation, are inappropriate. Furthermore, the relative orientation of the model and the cryo- EM map is important model building, which makes SE(3) invariant representations ill-suited. Thus, the most natural graph representation is an SE(3) equívariant one. In this project, we choose the backbone frame representation that was first introduced AlphaFold2, but is now also used in other protein prediction networks (e.g. Wu et al. 2022a Lin et al. 2022).
3 METHODS
3.1 GRAPH INITIALIZATION
We start by identifying the positions of the C atoms 2 the nodes of our graph. This part of the pipeline is formulated problem, similar to prior work discussed in section That is, the cryo-EM map V E RN,, where N the number of voxels, has an associated binary target T {0. 1) where represents the existence of a C atom in the voxel and O the lack of it. Since the minimum distance of two C atoms is 3.8 Å, resampling the cryo-EM voxel maps with a pixel size 1.5 Á ensures that there is no voxel that contains more than one such atom. The goal then becomes to train neural network (V) T
of all residues in the map, which will form
straightforward segmentation
3.1.1 NETWORK ARCHITECTTREE
We implemented fo(V) residual network (He al. 2016 inspired by the Feature Pyramid Network (Lin et al. 2017a). First, we changed all convolutions to 3D convolutions and changed batch normalization layers (loffe & Szegedy 2015) to instance normalization layers (Ulyanov al. 2016), as the statistics should not be averaged across instances of a batch since local boxes of cryo- EM density might be from different sections the map with large differences in scale. No noticeable effects on performance were observed between ReLU and LeakyReLU (Xu et al. 2013). so ReLU was selected due to its improved computational efficiency. We also shifted the network parameters from the low-resolution part of the model to the high-resolution part, and we changed the order of operations so that global information about the structure more directly integrated into the model at an early stage (for detailed pseudo-code that contains the exact number of parameters and order of operations, see algorithm 1 in A.3.I). In this manner, large-scale structural features, which are recognized at lower resolutions, become easier to integrate with the classification task.
3.1.2 TRAINING DATASET
The dataset starts with 6351 cryo-EM maps with a resolution higher than A, downloaded from the
EMDB (Lawson al. 2016) on 01/04/2022, and the corresponding PDB files that were downloaded from RCSB PDB Burleey et al. 202T). A portion of these was manually checked for orientation is- sues, large unmodelled regions, and the existence of large modeled regions that do not correspond to the cryo-EM map. This led to 700 manually curated pairs for the first round of training. Then,
Atom names in this document follow the PDB (Protein Data Bank) naming convention. See the PDB atomie coordinate and bibliographic entry format deseription
3

---

Published as a conference paper at ICLR 2023
A
B
VSETTFGRDFNEALY
Update Vectors
GNN Output
Backbone
-0:9
Frame
6.5
1.40
0.00
0.25
0.45
Mead
Cry-Oh
Exbeiiin
Layers
Cryo-EM
Sequence Module
IPA Module
Module
Resilues
ResAdass
Residues
Resivves
Resitues
Recidutt
Redtuue
Reddues
2
Figure 2: (A) shows the schematic the GNN and how the layers iteratively refine the feature vectors. (B) illustrates how the Backbone Frame module updates the positions of the backbone atoms. Finally, (C) contains two examples of high confidence (dark blue colour) and low confidence (yellow and red) predicted backbone regions for PDB entry 7ZIM. The confidence measure is good predictor of fit to the deposited backbone model (shown in outline).
using the first trained model, we were able automatically detect issues with the rest of the map- structure pairs and prune them to ~220 structures. The pruning was based precision and 70% recall of the model output C positions compared to the ground-truth PDB coor- dinates, at a cutoff of 3 Å. This produced the dataset that was used for training the model.
cutoff of 70%
3.1.3 TRAINING
We define the true number of the C atoms, residues, in map M To ameliorate the issue with class imbalance between the number of C atoms and the number of voxels (M* < N). we chose focal loss (Lin et al. 2017b) over binary cross entropy loss, Additionally, we up-weighted the loss of voxels with C" atoms and used a combination of auxiliary loss functions to get an acceptable trade-off between precision and recall. We up- up-weight the Co containing voxels in the focal loss with the ratio between the true negatives and true positives, using the following formula:
N M + (1-xx))
(I)
M
where W. is the weight for voxel and is the characteristic function of whether the voxel contains a C atom. This formula ensures that half the loss of a map comes from empty voxels and half from the C atom containing voxels. We found that also using Tversky loss (Hashemi let al. 2018) in the later parts of training allowed higher recall at the expense of precision. Lastly, improve generalization data augmentation schemes to the cryo-EM maps. We added random colored noise to every map and performed random sharpening/dampeninng by sampling a B-factor (see Rosenthal Henderson, 2003) from uniform distribution between -30 2 and 30 λ Lastly, we applied random rotations that are integer multiples of 90" to both the cryo-EM map and the targets in order to have the model learn different orientations while avoiding interpolation effects.
variations in experimental and data acquisition settings, applied
3.2 GRAPH REFINEMENT
Next, we define GNN 9s that is trained to refine the position of all C atoms in the graph, to map each of them to an individual residue in the user- user-provided input sequence, and to provide coordinates for all the atoms in the residues:
9 (m) p()-1) v.s)
(..s)) = (x(1), F(n), G(n), p(u) p(m),
(2)
4

---

Published as a conference paper at ICLR 2023
In the above, (x) R RMx3 are the C positions at iteration n. F Mx3x4 are the affine frames defined by the C C, and N atoms of the backbone, G E R MMxx2 are the torsion angles of the side chains, P E IR M is a probability vector for all twenty amino acids for each residue, and S E IRMx1280 are protein sequence embeddings of all residues in the input sequence, and O E RM per-residue confidence prediction. The backbone affine frames (F) and the side-chain torsion angles (G) are similar to what is used in AlphaFold2 (Jumper et al. [202], see figure 2 Together (X, F. G, P) provide enough information to calculate all-atom coordinates for the proteins in the cryo-EM map.
The overall training objective is to acquire Jo (x(x), p(), V.S) R G","", where X* is the set of Ca positions in the training data, F* and G are caleulated from the atom coordinates for every residue in the training data, and P E {0, M* x20 is one-hot encoding of the amino acid classes in the training data. At iteration n = 0, the graph is initialized with M nodes with random F(0), and with X set to the coordinates of those voxels predicted to contain C atoms by the graph initialization step in section 3.1
3.2.1 NETWORK ARCHITECTURE
The GNN consists of eight consecutive layers, each of which contains three main modules that are based on the attention algorithm (Bahdanau et al. 2014). Each module extracts information relevant to its modality and updates the feature encoding of the graph nodes with a residual connection, (see figure 2A and algorithm 9 in A.4.2).
The first module is the Cryo-EM Attention module, which allows the GNN to look at the density around each C atom with convolutional neural networks (CNN), as well as the density linking to its neighbouring nodes, to update its representation. This is accomplished with mix of graph- based attention module and feature extraction using CNNs. The edge features that determine the attention keys are calculated based on CNN embeddings of cuboids (clongated cubes in the direction of the neighbour) extracted from the cryo-EM map spanning each edge that connect a node with each of its neighbours. These regions of the cryo-EM map capture the connectivity between residues, e.g. peptide bond or side-chain:side-hhain interactions, and inform the attention scores. The query and value vectors are generated by the feature vector of each node. Additionally, cube centered each node is extracted from the cryo-EM density with an orientation defined by the backbone affine frame of each node and passed through a convolutional feature extraction network. Finally, the extracted features from the centered cubes are concatenated with the attention output and projected to create the new feature representation. For precise details about the algorithm, see algorithm 10 in A.4.2
The second module is the Sequence Attention module, where each node searches against the input sequence embedding to find the relevant entries that best fit its features. This is a conventional encoder-only transformer module, similar to that used in Devlin et al. (2018). The user-provided sequence is embedded using a pre-trained protein language model (ESM-Ib) (Rives et al. 202T] Lin et al. 2022). which only uses the primary sequence, instead of multiple sequence alignments (MSAs). Generally, MSAs improve the results of protein prediction algorithms by giving them access to co-evolutionary information (Jumper et al., (021), and they need less learnable parameters than algorithms that rely on a single sequence only (Rao et al.] 2021 Lin et al.) 2022] Wu et al., 2022b). However, because the cryo-EM map already provides sufficient information about the global fold of the proteins, we chose not to use MSAs, as their calculation would have made using our approach more difficult. For additional details about this module, see algorithm Il in A.3.2
The third module is the Spatial Invariant Point Attention (IPA) module, which allows the network to update its representation based on the geometry of the nodes in the graph. This module is inspired by the module with the same name in AlphaFold2 (Jumper et al., (021]. although simplifications have been made to better fit the problem at hand. Each node predicts query points based on its current representation in its own local affine frame; these points are transformed into the global affine frame (by application of F' to the predicted point); the distance calculated between each node's 's query points and its neighbours; and based on the sum of the distances of the query points to the neighbouring nodes, each of these nodes get an attention score that is used to update the central node's 's representation (see algoritht 12 in A.3.2). Essentially, this module queries parts of the graph where it expects specific nodes to be and then uses the distance of the neighbouring nodes to its query point to collect information from the other nodes.
5

---

Published as a conference paper at ICLR 2023
Since the three main modules are applied sequentially in eight layers, the representations from each module allow the other modules to gradually extract more information from their inputs. For ex- ample, using the cryo-EM density, the network is able find better orientation for its backbone as well as a more accurate set of probabilities for its amino acid identity, which lets it search the sequence more accurately with the sequence attention module. This process of improvement con- tinues while the positions of the atoms also get optimized using these representations through the application of the Backbone Frame module.
The Backbone Frame module (seen in figure 2B) takes as input the representation of each graph node that is the result of the sequential operation of the three main modules described above and outputs three vectors that describe the change in position of the C C. and N atom positions with respect to the network's current backbone affine frame. The shift in position is applied to the backbone atoms and the new backbone affine frame is calculated using Gram-Schmidt, similar to Algorithm 21 in AlphaFold2 (Jumper et al. 202T). Because the three shifts may distort the geometry of the peptide plane, the new coordinates for the backbone are calculated by aligning a peptide plane with ideal geometry (from literature bonds) with the shifted positions (see algorithm 16 in A.3.2
3.2.2 TRAINING
Multiple loss functions define the tasks of the different modules and the resulting losses opti- mized jointly with gradient descent. Most losses are calculated at each intermediate layer of the GNN so that it is able learn the correct structure as early in the layers as possible. The most important losses are C root mean squared deviation (RMSD) loss; backbone RMSD loss; amino- acid classification loss; local confidence score loss; torsion angles loss; and full atom loss. A full definition of all losses is given in Appendix A
The main training loop consists of taking PDB structure, extracting just the C atoms, distorting them with noise, initializing the backbone frames for each node randomly, and then having the network predict the original PDB structure. Cryo-EM maps are augmented with the addition of noise and hharpening/dampeningg.. similar to the training of the graph initialization network as described in section 3.1.3] Because the initial Ca positions are noisy, with an RMSD to the deposited model of 0.9 A on average, one important loss function is
Lc
- N Σ
RMSD(x; e:))
(3)
where x; E 3R3 are the true Ca positions from the dataset, Io is the graph neural network, and e, NO,, J/)) Note that /2 R 0.9 (this comes from the average norm of a Gaussian distributed vector). Denoising node positions has been shown to be a powerful training paradigm in other use cases as well (e.g. Godwin et al. 2021 }. In addition to noise added to the C" positions, sometimes the graph initialization step misses some residues or adds extra ones. In order to have the GNN be able to deal with these scenarios, during training 10% of the residues are randomly removed and replaced with randomly generated peptides that are between 2-5 residues long. The network is then also trained to be able to predict whether or not node actually exists in the model, or if it is extra. All classification losses focal loss (Lin et al. 2017b). This includes the amino acid classification loss, the sequence match loss, and the edge prediction loss.
An important feature of this network is that it also gives measure of its confidence in its output per residue. This is trained by having the network predict its backbone loss per residue. This output (referred to in 3.2 as O) is then normalized and saved in the B-factor section of the mmCIF file outputs. This output is useful in pruning regions of the model where the network is not confident in the postprocessing step (see section 3.3). Generally, we observe that orde-ord high- rishuresolution parts of the cryo-EM map have higher confidence values than regions with disordered and lower resolution (see figure 2C
The side chain atoms are generated through the prediction of their rotatable torsion angles with respect to the backbone frame (for an example, see figure [C). We noticed better results if the network predicted torsion angles for all 20 possible amino acid assignments for each residue. Then, we index into the torsion angle predictions for each residue and pick the set of angles that correspond to the predicted amino acid. To train this part of the model, for each layer, the mean squared loss of the torsion angles of the target amino acid against the true torsion angles is calculated, and at the last layer, the all-atom RMSD to the target structure is also calculated.

---

Published as a conference paper at ICLR 2023
3.2.3 RECYCLING
The output of one round the GNN denoises the positions of the C nodes and gives better orienta- tions for the backbone frames, and we observed that subsequent round of the GNN, starting from the output of the previous round, improved the results further. We, therefore, train the GNN with recycling. For every training step, randomly pick an integer {1, 2,3) 3) and the GNN r- times with gradients turned off, and then use the output to run the GNN one more time with gradi- ents. This allows the GNN learn to keep the input approximately unchanged when the positions are correct. We do not recycle the GNN features so they are recalculated with the corrected positions and orientations. The same recycling scheme, but with
also used during inference.
3.3 POSTPROCESSING
The GNN processes the C atoms into of unordered residues. Next, connect the residues into chains that define the full atomic model. In the strictest sense, not even the direction of the chains is defined by the GNN. However, using the fact that ||C,-1 N₁ < 1.4 Á (known as peptide bonds, see figure TA). we can combine the atomic coordinates predicted by the network as well as the edge prediction probabilities as a heuristic to connect residues. More concretely, the residues are tied so that the sum of peptide bond lengths across all nodes is minimized, ignoring links where the edge prediction is below a threshold of 0.5.
After the chains are connected, we use the amino acid prediction probabilities to construct an HMM profile. We then perform a sequence search using HMMER (Mistry et al.) 2013) against the given set of sequences of the model. The uncertainty of the predicted amino-acid probabilities P can vary due to several factors, e.g. map resolution or characteristics of the sequence itself, which can limit accurate mapping of the sequence onto the structure by the GNN. Due to this uncertainty, doing probabilistic search against the sequence after postprocessing gives superior performance over just assigning the highest probability amino acid. After alignment with the sequence search, residues that correspond to a "match" state (as defined in Krogh et al. 1994) are mutated to the amino acids that exist in the sequence. Based on the sequence search, we also connect separate chains that should be connected depending on both the matched sequence gap and the proximity of the chains.
Lastly, chains shorter than 4 residues are pruned and the resulting coordinates are used as the input to the GNN network again. This process continues for 3 recycling iterations and the end result undergoes a final "relaxation" step that uses physical restraint- based losses to optimize the positions of the model atoms using an L-BFGS optimizer (Liu & Nocedal 1989). This step mainly alleviates unnatural side chain distance violations and does not noticeably affect the distance metrics in section which are all based on main chain atoms.
4 RESULTS
A
B
C
Sequence recall
Sequence recall
Sequence recall
Original
80-
ModelAngelo
80
IPA Ramoved
Removed
Removed
Deeptracer
50-
50-
70-
20
20-
B-factor [A?]
50-
B-factor [
0
100
200
20
DeepTraces 80
100
200
Figure 3: (A) shows sequence recall for all residues in the test dataset as function of B-factor labels for final model outputs (after postprocessing). for Deeptracer (black) and ModelAngelo (before pruning in orange; after pruning in purple). (B) shows the same results but averaged for each PDB entry, with ModelAngelo's pruned prediction (y-axis) versus Deeptracer (x-axis). The dotted line marks the identity line. The orange marker represents PDB entry 8DTM, which is shown in figure (C) shows the result of the ablation experiment, similar in format to (A).
7

---

Published as a conference paper at ICLR 2023
Here report results for test dataset of 28 map-model pairs that were deposited to the PDB and EMDB after the cutoff date for training. Results on a smaller test set without protein chains with more than 30 % sequence similarity with any model in the training set are described in A.5 We implemented our approach in an open-source software package called ModelAngelo. Generally, the atomic models built by ModelAngelo are close the deposited PDB structures and they degrade with the resolution of the cryo-EM map. The overall resolution of the 28 test maps ranges from 1 to 3.8 A. However, flexibility in parts of the protein structures also leads to local variations in resolution across the maps. The latter is reflected in the refined B B-factors of the individual residues in the deposited PDB coordinate files, where higher B-factors indicate lower local resolution. We compare our results against the current state-of-the-art method for automated model building, Deeptracer (Pfab et al. 202T). A comparison with results obtained with the Phenix software (Terwilliger et al.) 2018), which performs worse than DeepTrracer is available in table 4 of A.2
Our main metric sequence recall: the percentage of residues for which the C atom is within Á of the deposited model, and the amino acid prediction is correct. Figure compares the performance of ModelAngelo and Deeptracer for each of the structures in the test dataset, and as a function of B-factor averaged over all residues. A more comprehensive list of metrics, for each of the 28 map- model pairs, is shown in Appendix [A.2] Over the entire test dataset, there is little difference in sequence recall between the unpruned and the pruned model from ModelAngelo, which implies that pruning removes incorrectly built parts of the model.
Ablation studies (figure 33] show that the improved results of AngelA versus Deeptracer because ModelAngelo is able to combine different modalities of information to build the model, rather than just the cryo-EM map. Removing the sequence module or the IPA module results in worse results that are still better than Deeptracer. However, removing both of these modules, such that the GNN only relies on the Cryo-EM module, results in predictions that are worse than Deep- tracer.
Figure illustrates the quality of the built models from ModelAngelo and Deeptracer for one ex- ample from the test dataset (PDB entry 8DTM); more examples are given in AA6] Because of its considerably slower than Deeptracer. Still, execution times increased complexity, ModelAngelo for the test dataset are in the range of several minutes to one hour and a half, depending on the size of the structure (see A.4.3). Given that manual de novo model building takes on the order of weeks, we do not believe this to be serious drawback.
5 DISCUSSION
Our results illustrate that combining the voxel-based information from the cryo-EM map with se- quence information and graph topology is useful for automating the intensive task of atomic mod- elling in cryo-EM maps. Below, we consider the limitations of the current implementation of Mod- elAngelo, and outline lines of future research to overcome them.
Sensitivity to resolution. Even though the network has multiple different modalities of input, rel- atively low resolutions of the cryo-EM map will affect the results. The graph initialization by the CNN, and the amino acid classification that provides the information for mapping the sequence onto the main chain, are obvious examples that benefit from higher-resolution maps. Poor amino acid classifications may also lead to errors in the sequence assignment in the postprocessing step, which may then feed into the subsequent HMM sequence alignment and lead incorrect chain assign- ments. This is more likely to happen for complexes with many similar sequences. In practice, we observe that ModelAngelo's performance starts degrading resolutions worse than 3.5 A, see Ap- pendix A.3 A similar trend is also typical for manual model building. It may be possible to combine the embedding of information from relatively low-resolution cryo-EM maps (e.g 10 Á) with meth- ods for protein structure prediction, such AlphaFold2 (Jumper et al. (02II. which would extend beyond what is possible for manual building. Despite the observation that ModelAngelo already uses some ideas from AlphaFold2, such an approach for building in low-resolution maps, which would blur the boundaries between experimental structure determination and prediction, would re- quire major changes to the approaches outlined in this paper.
Nucleic acids. Many large complexes that are solved with cryo-EM comprise nucleic acids as well as proteins, e.g. ribonucleic acids (RNA) in spliceosomes and ribosomes, deoxyribonucleic acids
8

---

Published as a conference paper at ICLR 2023
29.1
121
Correct Incorrect
A
BC
B
C
D
E
Figure 4: Comparison of the deposited model for PDB entry 8DTM (C), Deeptracer's prediction (D) and AngelAngelo's pruned model (E). The deposited model is coloured according to the refined B- factor, Meanwhile, the predictions are coloured orange where their amino acid prediction different from the deposited structure, and purple where it is the same. (A) shows an iso-surface of the cryo- EM map and the deposited model for high B-factor region, with correspondingly poor density. (B) shows the same for low B-factor region, where side chains are well resolved in the density.
(DNA) in replication transcription machinery. The backbone of DNA or RNA strands is made from alternating phosphate and sugar groups. The phosphorus atoms have high contrast in the cryo- EM map, which makes the segmentation problem of nucleic acids easier than that of protein residues. However, the main difficulty lies in identifying the correct sequence for the nucleobases that make up the equivalent of side chains for RNA DNA strands. There are four typical bases for both RNA and DNA: two purines and two pyrimidines. At resolutions around 3.5 A, one can distinguish the purines from the pyrimidines, but not the two purines or pyrimidines from each other. This makes sequencing DNA or RNA strands in 3.5 Á cryo-EM maps difficult. Yet, already the ability to automatically model nucleotide backbones, together with classifier to distinguish purines from pyrimidines, would alleviate the task of manual model building. Therefore, plan to add support for building RNA or DNA to ModelAngelo in the near future.
Unknown sequences. Because cryo-EM can be performed on samples that extracted from na- tive cells or tissues, it is not always obvious which proteins are present in a cryo-EM map, (for an example, see Schweighauser al. 2022). Our current implementation depends user-provided sequence file that defines all proteins present in the map. Recently, semi-automated tools to iden- tify proteins in cryo-EM maps have been reported. For example, findMySequence (Chojnowski let al. 2022) can search protein sequence databases, using amino acid classifications after back- bone model has been built in the cryo-EM map. We plan extend our approach with sequence- model to fully automate this process, by combining automated model building with searches through large sequence databases like Uniclust (Mirdita et al. (017)) using our HMM search procedure com- bined with tools such as HHblits (Remmert et al. 2012)
ACKNOWLEDGEMENTS
We thank Johannes Schwab, Joe Greener, Sofia Lövestam, David Li, Keitaro Yamashita, Garib Mur- shudov, Carola-Bibiane Schönlieb, Tanmay Bharat, Zo Ford, and Rafael Fernández Leiro for helpful discussions, and Jake Grimmett, Toby Darling, and Ivan Clayson for help with high-performance computing. This work was supported by the Medical Research Council (MC UP A025 1013 to S.H.W.S.).

---

Published as a conference paper at ICLR 2023
REFERENCES
Minkyung Baek, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong. Lisa N Kinch, Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):871- 876, 2021.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: 1409.044 2014.
Stephen K Burley, Charmi Bhikadiya, Chunxiao Bi, Sebastian Bittrich, Li Chen, Gregg V Crichlow, Cole H Christie, Kenneth Dalenberg, Luigi Di Costanzo, Jose Duarte, et al. Resb protein data bank: powerful new tools for exploring 3d structures of biological macromolecules for basic and applied research and education in fundamental biology, biomedicine, biotechnology. bioengineer- ing and energy sciences. Nucleic acids research, 49(D1):D437 D451, 2021.
Yifan Cheng. segeee-atticle cryo-em sooweem-owwwww 876-880, 2018.
how did it get here and where will it go. Science, 361(6405):
Grzegorz Chojnowski, Adam Simpkin, Diego A Leonardo, Wolfram Seifert-Davila, Dan Vivas- Ruiz, Ronan M Keegan, and Daniel Rigden. findmysequence: neural-network-basee approach for identification of unknown proteins in x-ray crystallography and cryo-em. IUCrJ, 9(1), 2022.
Kevin Cowtan. The buccaneer software for automated model building. 1. tracing protein chains. Acta crystallographica section D: biological crystallography, 6009):1002-111 2006.
Tristan Ian Croll. Isolde: physically realistic environment for model building into low-resolution
electron-density maps. Acta Crystallogaahhica Section D: Structural Biology, 74(6)5519-530, 2018.
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning based protein sequence design using proteinmpnn. bioRxiv, 2022.
Jacob Devlin, Ming- bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.04805. 2018.
Ming-Wei Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
Paul Emsley, Bernhard Lohkamp, William G. Scott, and Kevin Cowtan. Features and development of coot. Acta Crystallographica Section D Biological Crystallography, 66:486-501, 2010.
Octavian- Eugen Ganea, Xinyuan Huang, Charlotte Bunne, Yatao Bian, Regina Barzilay, Tommi S.
Jaakkola, and Andreas Krause. Independent -Equi-equi models for end-to-end rigid protein docking. In International Conference on Learning Representations, 2022. URL openreview net/ net/Forum?id:G0 9mLet.
https://
Jonathan Godwin, Michael Schaarschmidt, Alexander Gaunt, Alvaro Sanchez-Gonzalez, Yulia Rubanova, Petar Velickovic, James Kirkpatrick, and Peter W. Battaglia. Very deep graph neu- ral networks via noise regularisation. CoRR, fbs/2200.009711. 2021. URL uttpstts:://arxi
org/ httg/ba/221000 07971
Seyed Raein Hashemi, Seyed Sadegh Mohseni Salehi, Deniz Erdogmus, Sanjay P. Prabhu, Simon K. Warfield, and Ali Gholipour. Tversky as a loss function for highly unbalanced image segmentation using fully convolutional deep networks. CoRR, Abs/1803.11078 2018. URL http://
arxiv. www.vv.org//.. abs /10./.103.... 1018811.8881
Jiahua He, Peicong Lin, Chen, Hong Cao, and Sheng- You Huang. Model building of protein complexes from ceammediate-resoluttie cryo-em maps with deep learning-guide automatic sembly. Nature Communications, 13(1):1-16, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Published as a conference paper at ICLR 2023
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. PMLR. 2015.
Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional diffu- sion for molecular conformer generation. arXiv preprint drXiv:2206.01729, 2022.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-588 2021.
Anders Krogh, Michael Brown, Saira Mian, Kimmen Sjölander, and David Haussler. Hidden markov models in computational biology: Applications to protein modeling. Journal of molecular biology, 235(5): 15015)5:5111111 1994.
Catherine L Lawson, Ardan Patwardhan, Matthew L Baker, Corey Hrye, Eduardo Sanz Garcia, Brian Hudson, Ingvar Lagerstedt, Steven J Ludtke, Grigore Pintilie, Raul Sala, et al. Emdata- bank unified data resource for 3dem. Nucleic acids research, 44(D1))D336-DD4) 2016.
Dorothee Liebschner, Pavel V Afonine, Matthew L Baker, Gábor Bunkóczi, Vincent B Chen, Tris- tan I Croll, Bradley Hintze, L-W Hung, Swati Jain, Airlie McCoy, et al. Macromolecular struc- ture determination using x-rays, neutrons and electrons: recent developments in phenix. Acta Crystallographica Section D: Structural Biology, 75(10))861-877, 2019.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 2117-2125, 2017a.
Tsung-Y Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. CoRR, N0//100.00000) 2017b. URL http:://axxi..or//aaap 02002
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos San- tos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido. and Alexander Rives. Language mod- els of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022. doi: 10. 10010022222000000500000000 URL http:t/////// htwp::::/wwww... wwww www.blooxxxv..or/ccneeeee. ear 2atll//2022/07////2222222 50000
Dong C Liu and Jorge Nocedal, On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1):003-528, 1989.
Valerio Mariani, Marco Biasini, Alessandro Barbato, and Torsten Schwede. Iddt: local superposition-free score for comparing protein structures and models using distance difference tests. Bioinformatics, (2(21):2722221)) 2013.
Milot Mirdita, Lars Von Den Driesch, Clovis Galiez, Maria J Martin, Johannes Söding, and Martin Steinegger. Uniclust databases of clustered and deeply annotated protein sequences and align- ments. Nucleic acids research, 45(D1):D170 7176...6.... 2017.
Jaina Mistry, Robert D Finn, Sean R Eddy, Alex Bateman, and Marco Punta. Challenges in homol- ogy search: Hmmer3 and convergent evolution of coiled-coil regions. Nucleic acids research, 41 (12):e121-e121) 2013.
2013.
Garib N Murshudov, Pavol Skubák, Andrey A Lebedev, Navraj S Pannu, Roberto A Steiner, Robert A Nicholls, Martyn D Winn, Fei Long, and Alexei A Vagin. Refmac5 for the refinement of macromolecular crystal structures. Acta Crystallographica Section D: Biological Crystallog- raphy, 374):335-36,, 2011.
Takanori Nakane, Abhay Kotecha, Andrija Sente, Greg McMullan, Simonas Masiulis, Patricia MGE Brown, Ioana T Grigoras, Lina Malinauskaite, Tomas Malinauskas, Jonas Miehling, et al. Single- particle cryo-em at atomic resolution. Nature, 587(7832):552-156 2020.
Anastassis Perrakis, Richard Morris, and Victor S Lamzin. Automated protein model building com- bined with iterative structure refinement. Nature structural biology, 6(5):458-46) 1999.

---

Published as a conference paper at ICLR 2023
Eric F Pettersen, Thomas D Goddard, Conrad C Huang, Elaine C Meng. Gregory S Couch, Tris- tan Croll, John H Morris, and Thomas E Ferrin. Uesf chimerax: Structure visualization for researchers, educators, and developers. Protein Science, 30(1):70-888) 82, 2021.
Jonas Pfab, Nhut Minh Phan, and Dong Si. Deeptracer for fast de cryo-em protein structure modeling and special studies on cov-related complexes. Proceedings of the National Academy of Sciences of the United States of America, 118, 2021. ISSN 10916490. doi: 10.1073/PNAS. S...PP..D..
Ali Punjani, John L Rubinstein, David J Fleet, and Marcus A Brubaker. cryosparc: algorithms for rapid unsupervised cryo-em structure determination. Nature methods, 14(3):290-22 296, 2017.
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. International Conference on Machine Learning, pp- 8844-8856. PMLR, 2021.
Michael Remmert, Andreas Biegert, Andreas Hauser, and Johannes Söding. Hhblits: lightning-fast iterative protein sequence searching by hmm-hmm alignment. Nature methods, 9(2):133-775 2012.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Acudemy of Sciences, (18(15):2001223111 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Peter B Rosenthal and Richard Henderson. Optimal determination of particle orientation, absolute hand, and contrast loss in pareell electron cryomicroscopy. Journal of molecular biology, 333(4):721-745, 2003.
Sjors HW Scheres. Relion: implementation of bayesian approach to cryo-em structure determina- tion. Journal of structural biology, 180(3):519-533 2012.
Manuel Schweighauser, Diana Arseni, Mehtap Bacioglu, Melissa Huang, Sofia Lövestam, Yang Shi, Yang Yang, Wenjuan Zhang, Abhay Kotecha, Holly J Garringer, et al. Age-dependent formation of tmeml066 amyloid filaments in human brains. Nature, 1057790))330--31 2022.
Genki Terashi and Daisuke Kihara. De novo main-chain modeling for em maps using mainmast. Nature communications, -1))1-11,,0118
Thomas C Terwilliger, Ralf W Zwart, L-W Hung, Randy Read, and Paul D Adams. Iterative model building, structure refine- ment and density modification with the phenix autobuild wizard. Acta Crystallographica Section D: Biological Crystallography. 64(1):61-69, 2008
Pavel V Afonine, Nigel W Moriarty, Peter H
Thomas C Terwilliger, Paul D Adams, Pavel Afonine, and Oleg V Sobolev. fully automatic method yielding initial models from high-resolution cryo-electron microscopy maps. Nature methods, 15(11) (5(11)9905990) 2018.
Thomas C. Terwilliger, Billy K. Poon, Pavel V. Afonine, Christopher J. Schlicksup, Tristan I. Croll, Claudia Millán, Jane. S. Richardson, Randy Read, and Paul D. Adams. Improved alphafold modeling with implicit experimental information. bioRxiv, 2022. doi: 10.1101//022.01.077 475350. URL httpp::/wwwet wtttp://www.iixx..ooroo//////// http:///ww..lloooc.oooooo//////.
01. 07.07.44 475350
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint 16Xiv 2016.
12

---

Published as a conference paper at ICLR 2023
Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, Jianzhu Ma, and Jian Peng. High-resolution de novo structure pre- diction from primary sequence. bioRxiv, 2022a. doi: 10. 20211001120222072211550099999999 URLhttps:
//www biorxiv. htt.://ww......////////////////.
500999
Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al. High-resolution de novo structure prediction from primary sequence. bioRviv, 2022b.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00833 2015.
Ka Man Yip, Niels Fischer, Elham Paknia, Ashwin Chari, and Holger Stark. Atomic-resolution protein structure determination by cryo-em. Nature, 1571 2020.
A APPENDIX
A.I LOSSES
This is full description of all the losses used for training the GNN. These losses can be divided into two groups: one-step losses and auxiliary losses. One-step losses are losses that are only defined once for the final step of the GNN. However, auxiliary losses summed over each step of the GNN, though the last step has a higher weighting.
A.1.1 AUXILIARY LOSSES
These losses follows: C RMSD loss, backbone frame loss, amino-acid classification loss, edge classification loss, local confidence score loss, and existence loss.
The C RMSD loss is just the RMSD of the C atoms only. That is
L
N Σ RMSD(x,, go (x, e.))
[xs]a - [90(x; + c;)lal
d=1
The backbone frame loss is the RMSD of the three backbone atoms that define the backbone frame: Ca C, and N atoms. The over-representation of the RMSD of the C atoms, through its contribution two losses, is intended.
The amino-acid classification loss is the focal loss between the amino-acid classification logits and the larget amino-acid recorded for each node. We have observed that even after the convergence of the GNN, the auxiliary loss of this classification task is relatively high, meaning that earlier layers are not well capable of distinguishing amino acids. This could be because the C* positions are still noisy or that the network needs to pool neighbouring information more before it can be confident about the classification.
The edge classification loss is based on whether two ground truth C nodes are within 3.9 Á of each other. If they are, then we expect both nodes a and to classify each other as a neighbour. Otherwise, both need to classify the other as not neighbour.
The local confidence score loss is a simple regression loss that tries to predict the backbone frame loss described above for each residue. This gives good measure of how confident the network is in the position of the residue. This measure is then normalized between 0 and 100 based on a simple heuristic so that larger expected losses lead to lower confidence values.
Finally, the existence loss is a focal loss that classifies whether a residue is an artificially added residue or it actually exists.
13

---

Published as a conference paper at ICLR 2023
A.1.2 ONE-STEP LOSSES
The full-atom loss is the same RMSD loss defined above for the C atom, except that it is calculated for all atoms, including side chain atoms.
The torsion angles loss is defined as the L2 distance between the torsion angles of the network output and the torsion angles of the deposited model. A small loss added make sure the norm of the torsion angles is equal to 1. This is similar to the loss defined in section 1.9.1 Jumper et al. (202T.
The sequence attention loss is classification loss defined over the length of the given sequence for each residue, designed match the position of the graph node. It calculated for the last attention head of the sequence attention module as a focal loss over the calculated attention score.
The violation loss meant to constrain the network output models that are more in line with ideal bond lengths and angles. This loss is the same as defined in section 1.9.1 1 in Jumper et al. (202T)
A.2 DETAILED METRICS PER PDB ENTRY IN THE TEST DATASET
The tables below contain the resolution (in Å) and various metrics for each of the 28 PDB entries in the test dataset. Backbone RMSD and sequence recall are defined in the main text. Ca recall is the percentage residues for which the C atom is within 3 Á of the deposited model; C precision is the percentage of C atoms predicted that are within 3 À of C atom in the deposited map; IDDT the C local distance difference test as described in Mariani et al. 2003)) and sequence match is the percentage of matching amino-acids for the corresponding residues. Table I describes the results obtained by Deeptracer, tables and describe the results obtained by ModelAngelo, after and before pruning, respectively.
14

---

Published as a conference paper at ICLR 2023
PDB
Reso- lution
backbone RMSD
C
C
IDDT
Sequence match
Sequence
recall
precision
recall
7tu5
2.1
0.591
94.3
99.4
96.7
90.0
84.8
7unl
2.45
0.496
97.3
98.2
97.4
95.0
92.4
7qlu
2.7
0.546
99.3
99.2
96.2
99.2
98.5
7szi
2.7
0.635
98.3
99.0
94.2
96.3
94.6
7txv
2.7
0.837
76.3
97.8
91.2
77.3
58.9
7uck
2.8
0.866
87.8
49.0
93.6
85.5
75.1
7ode
2.84
0.960
66.6
29.0
93.1
69.9
46.5
7sba
2.9
0.829
88.4
93.9
92.2
75.2
66.5
7ugg
3.16
0.896
93.0
98.7
93.2
82.3
76.5
7pt6
3.2
0.750
92.4
99.0
93.3
91.1
84.1
7xpx
3.2
0.762
82.0
58.8
96.1
92.2
75.6
7px8
3.27
0.637
97.6
99.2
94.0
94.7
92.5
7sr8
3.3
1.026
95.4
95.3
86.6
78.8
75.1
7wug
3.3
0.778
97.1
95.8
92.5
82.5
80.0
7y9u
3.3
0.739
82.9
98.7
94.3
84.5
70.0
7u50
3.4
0.734
76.6
88.3
92.9
88.3
67.6
7sjn
3.4
1.213
70.5
95.5
85.7
53.1
37.4
7z1m
3.4
0.954
92.9
97.7
89.2
74.6
69.3
7rzy
3.5
1.544
71.4
94.2
79.7
17.9
12.8
7w91
3.5
0.977
80.6
96.1
90.2
72.8
58.7
8dtm
3.5
1,420
95.9
81.6
87.1
46.8
44.9
8a3t
3.5
1.380
76.5
96.6
81.1
50.8
38.8
8a2q
3.53
1.260
74.0
96.6
85.2
55.5
41.1
7yzk
3.57
1.293
94.8
88.4
82.7
53.2
50.5
7oix
3.6
1.339
95.1
77.9
82.0
45.7
43.4
7tvz
3.6
1.261
91.8
96.3
85.9
59.6
54.7
7pt7
3.8
1.202
86.3
96.4
86.5
57.5
49.6
7um0
3.8
1.102
86.2
98.1
86.7
67.8
58.5
Table 1: Deeptracer Results
15

---

Published as a conference paper at ICLR 2023
PDB
Reso- lution
backbone RMSD
C
C
IDDT
Sequence match
Sequence
recall
precision
recall
7tu5
2.1
0.287
96.3
99.7
98.7
99.9
96.2
7unl
2.45
0.310
99.2
94.7
98.9
99.7
98.9
7qlu
2.7
0.268
99.6
99.3
99.2
99.9
99.6
7szi
2.7
0.398
97.2
99.4
98.4
99.3
96.5
7txv
2.7
0.686
89.5
91.5
92.3
90.5
81.0
7uck
2.8
0.385
97.6
99.5
97.4
99.8
97.3
7ode
2.84
0.418
81.5
99.2
96.7
95.7
78.0
7sba
2.9
0.444
86.1
98.2
96.1
97.4
83.9
7ugg
3.16
0.468
96.3
98.6
95.9
97.7
94.1
7pt6
3.2
0.435
95.7
99.1
96.6
98.9
94.7
7xpx
3.2
0.343
85.4
97.1
97.6
99.5
85.0
7px8
3.27
0.368
99.0
98.2
97.7
99.1
98.1
7sr8
3.3
0.537
92.7
98.0
93.5
96.9
89.8
7wug
3.3
0.396
96.2
94.1
97.0
99.3
95.5
7y9u
3.3
0.388
77.9
99.0
97.7
99.1
77.2
7u50
3.4
0.396
73.7
97.7
97.0
97.6
71.9
7sjn
3.4
0.740
68.3
96.3
90.5
93.1
63.6
7z1m
3.4
0.472
90.4
99.3
95.3
97.9
88.5
7rzy
3.5
0.746
47.2
99.2
90.1
97.2
45.8
7w91
3.5
0.447
74.0
98.7
96.2
97.2
71.9
8dtm
3.5
0.557
92.0
90.9
93.8
97.3
89.5
8a3t
3.5
0.706
66.9
98.6
90.4
95.6
63.9
8a2q
3.53
0.644
74.1
94.8
90.8
94.4
70.0
7yzk
3.57
0.811
81.0
94.4
88.5
88.9
72.0
7oix
3.6
0.636
75.1
96.4
91.2
89.6
67.3
7tvz
3.6
0.563
86.5
98.3
93.3
97.7
84.5
7pt7
3.8
0,476
78.2
99.5
95.0
95.9
74.9
7um0
3.8
0.618
89.8
99.0
92.2
97.7
87.8
Table 2: ModelAngelo Pruned Results
16

---

Published as a conference paper at ICLR 2023
PDB
Reso- lution
backbone RMSD
C
C
IDDT
Sequence match
Sequence
recall
precision
recall
7tu5
2.1
0.307
99.7
98.1
98.3
96.7
96.5
7unl
2.45
0.311
99.9
84.8
98.8
98.9
98.8
7qlu
2.7
0.273
100.0
97.7
99.1
99.6
99.6
7szi
2.7
0.406
99.1
98.2
98.3
97.8
96.9
7txv
2.7
0.710
92.3
87.0
91.9
88.4
81.6
7uck
2.8
0.401
99.0
98.2
97.2
98.4
97.4
7ode
2.84
0.549
96.9
93.5
94.6
84.2
81.7
7sba
2.9
0.516
97.4
91.5
94.7
87.6
85.3
7ugg
3.16
0.495
98.9
97.1
95.4
95.4
94.3
7pt6
3.2
0.466
98.1
97.7
96.1
96.6
94.9
7xpx
3.2
0.563
98.2
90.3
93.7
88.1
86.5
7px8
3.27
0.371
99.9
96.5
97.7
98.5
98.4
7sr8
3.3
0.574
99.0
93.7
92.7
92.0
91.1
7wug
3.3
0.420
99.8
82.0
96.5
96.1
95.9
7y9u
3.3
0.719
99.1
86.0
91.7
82.8
82.1
7u50
3.4
0.848
99.1
67.2
89.6
76.3
75.6
7sjn
3.4
1.071
98.6
78.7
85.8
70.2
69.2
7z1m
3.4
0.546
98.0
97.6
93.9
91.4
89.6
7rzy
3.5
1.122
95.3
85.9
82.8
57.2
54.5
7w91
3.5
0.646
90.2
89.2
92.6
83.0
74.9
8dtm
3.5
0.649
99.5
74.8
92.1
90.5
90.0
8a3t
3.5
0.960
86.5
94.3
85.9
77.7
67.3
8a2q
3.53
0.923
98.3
82.7
86.6
74.7
73.5
7yzk
3.57
0.935
99.4
75.4
86.2
74.4
73.9
7oix
3.6
0.843
99.4
74.9
87.6
71.1
70.7
7tvz.
3.6
0.661
98.5
93.2
91.5
87.8
86.5
7pt7
3.8
0.583
88.9
97.4
93.2
86.2
76.6
7um0
3.8
0.727
98.6
93.8
90.5
90.5
89.2
Table 3: ModelAngelo Unpruned Results
17

---

Published as a conference paper at ICLR 2023
PDB
backbone RMSD
C
Ca
IDDT
Sequence match
Sequence
recall
precision
recall
7tu5
0.841
60.5
95.6
85.1
58.8
35.5
7unl
0.692
88.1
96.4
89.5
80.7
71.1
7qlu
0.862
89.5
94.5
84.7
70.5
63.1
7szi
0.843
85.7
96.4
88.1
81.7
70.0
7txv
1.028
44.6
94.3
83.5
62.8
28.0
7ode
1.103
8.0
51.9
84.9
25.7
2.0
7sba
1.127
64.7
93.6
80.8
28.4
18.4
7ugg
1.277
59.5
94.6
79.1
36.7
21.9
7xpx
0.889
69.6
86.5
83.5
40.8
28.4
7px8
1.055
83.0
90.4
80.5
60.5
50.3
7sr8
1.833
45.4
94.0
74.6
7.7
3.5
7wug
1.027
82.7
95.4
83.5
57.7
47.7
7y9u
1.068
67.5
95.7
81.8
40.8
27.5
7u50
1.049
56.5
79.1
81.0
36.6
20.7
7sjn
1.580
33.0
95.8
78.3
19.6
6.5
7rzy
2.103
24.0
92.1
70.1
8.8
2.1
7w91
1.177
53.3
95.7
80.5
47.6
25.4
8dtm
1.808
48.8
87.5
74.8
11.3
5.5
8a2q
1.503
53.0
95.3
75.1
34.5
18.3
7yzk
1.575
56.8
92.1
77.3
27.0
15.4
7oix
1.933
60.2
95.4
71.6
7.2
4.3
7tvz
1.474
39.1
95.2
76.2
31.4
12.3
7um0
1.611
54.1
95.2
75.7
16.5
9.0
Table 4: PHENIX Results
18

---

Published as a conference paper at ICLR 2023
A.3
NETWORK DETAILS
A.3.1
GRAPH INITIALIZATION NETWORK
Algorithm Segmentation Forward Pass. The fully convolutional segmentation model described in
section 3.I.I
1: function SEGMENTATION CORIITAATOOOFOWARDDOOOO
2: V (V mean V s)s/s((VV)
3: ds_0, ds_ 2, ds 3. ds_4 < imput.channel=-12. ouputt.hammels==25)
4:
t14 f conv building block(ds. 4,
5:
114 conv homwwbbllding.aa blo.ddinngbooodss__1__
hiput.channels=256. output.chammels=256)
6:
c4 - main laserle add(t14, I14), i channels=256, expansion=4, num layers=2)
7:
t13 conv building block(c4, input channels=1024, output channels=128)
8:
113 < conv cow._building.llocleeeo main mhaii__ayortupsample.dddd
2,
ourput.channels=128) 113), imput.channels=12..
9:
c3
expansion=4,
num layers=20)
t12 s conv building block(c3, inw.t. channels=512, output channels=64)
11:
I12 f conv brrilding_blockiee input_chameeee chaputtchannes-64, output_channelie
12:
c2 4- main maainnlaeertussapll.eee
I12), input chpunelchane exppeeeeeeee..e.eiieiooooooe expansion=4, num lamelayers=50)
13:
d1 conv Nonv_buildinggblockte
cpput_channel-=256. output__hannels==6))
< conv conv_building -loockidnnggbbokdds .O, muputcchannel=64,, outpur_channel==64)
15:
el main maaiiiaaycrrpssapllc... (111 111), mnput.channel=664 expansion=4, num_Jayers==0))
16:
pred f multi scale conv(cl, input channcls=256, mid channels=64, output channels=1)
return pred
Algorithm 2 Convolutional Building Block
1: function CONV BUILDING BLOCK(f,
output channels)
2: < conv3d(f, chaannels,,, output.chanmee channels, kizeel stride=1, padding=1)
instance Serancee_orm_.. nff, output channels, affine=True) 4: f relu(f)
3:
5: return f
Algorithm 3 Bottleneck
1: function BOTTLENECK(L input inppitncpaanaeteee channels, hidden. channels, expansion)
2:
f conv conv_buille www.ldiig.illcceeecc block(i, input_channed hidden_de (oddeen_channes)
3:
hidden.channels, hidden_channels) + conv www.bbuldingg.bocck
4:
conv3d(f, hidden channels, hidden channels expansion, kernel size=1, stride=1, padding=1)
5: instance_na @aatnnce_nrm__33 norm_3d(f, output affine=TTru)
6:
f f relu(f + i)
7:
return f
19

---

Published as a conference paper at ICLR 2023
Algorithm 4 Downsample Layer. Factor-two downsampling with strided convolutions.
function DOWNSAMPLE(inuu channels, output channels, f)
2:
f conv3d(f,, input channels, output channels, kernel size=3, stride=2, padding=1) instance_norm_.5 output_channels, affine=True) f f- relu(f)
3:
4:
return f 5:
Algorithm Downsample Forward Pass
1: function DOWNSAMPLE, (LOISAMMPLEFFOWWR(VV)
2: ds_0 < conv boinddigldiig..oo buildinggblooc((.. 64, hermel_size-3 padding=2)
3: ds_1 downsample(ds. O, channelsean,, output channels=664)
4: ds_2 < downsample(ds.1, input.channel==64, output_channell=Il
5: ds_3 downsample(ds 2, input.channels=12. output.channel==256
6: ds & downsample(ds 3, input channels=256, output channels=512)
7:
return ds ds 1, ds 2, ds 3, 4
Algorithm 6 Main Layer
function MAIN LAYER(f, input in channels, expansion, num layersy t- hidden.channels 3: for range layers) do
2: hidden channels
4: f bottleneck(f, input.channels, hidden_channels. expansion)
5: input channels t- hidden channels x expansion
6: return f
Algorithm 7 Upsample then add
1: function APSAMPLE ADD(f, g)
2:
H, W, D retur_shape(g'
3: upsampled +- trilincar interollonn,,,, H, W, D)
4: return upsampled +
Algorithm 8 Multi Scale Convolution
1: function BULTISCCLEE.CM SCNLE,, input.channels, mid channeasss,s, output,channely
2: f₃ f conv3d(f, soput-channels,
pernel_size=3) stride=1, padding=1)
f <- conv3d(f. input channels, mid channels, kernel size=5, stride=1, padding=2)
4:
f <- conv3d(f, input channels, mid channels, kernel size=7, stride=1, padding=3) +- concatenate(f3, fs. f7)
5:
6:
f instance.norm_ddle mid mid_channels 3, affine=True)
7: f f relu(f)
8: f ft- conv3d(f, mid channels 3, output outputt.caaneels channels, padding=1, kerne-
9:
return f
20

---

Published as a conference paper at ICLR 2023
A.3.2 GRAPH NEURAL NETWORK
Algorithm 9 GNN Forward Pass
1:
function for k in range htwp:/uiwwrrccelnn.eeeemm steps) do
STORWARRDPASSEE V, num.recycling.step
2:
3:
f zeros(batch size, 256)
4:
for i in range(num layers=8) do
5:
f f- cryo_em_attentionnix
F, V)
6:
f < scqueence_atticntionff. S)
7:
f + spatial_ipa(x, f, F)
8:
f < Transition_layerte
9:
P < amino_acii acid classification(f)
10:
O local confidence prrdicciorc)))
11:
X, F +- backbone frame module(f, F)
12:
G <- torsion angle network(())
13:
return F, O.
Algorithm 10 CryoEM Attention Module
1: function CRYO_EMATTENNTINN
F, V)
2: C < www.ctangles.eeeeee......
F, V)
3: R + get_rectangles. between _xdes((x. nodsee.. F, V)
4:
d + generate. centraatedistance... castaance.bbsee features(x))
5:
n. [z. dlc
6:
k < get cryo leellkeeeeeeeeee vew_eeeetsss))
7:
q
cryo. query value vectors(n,)
8:
2, + softmax k).
9:
Zp get.cryo.poimmffauurrecCC
10:
z f linear Janer_laayer(z.. [ze, Zy]. outputdimm=256
z' + dropout(z', probability=0.5)
layer_normn 22/22)
13:
return z
Algorithm 11 Sequence Attention Module
1: function DOQQUENCE.ATTENTIONE. S)
2:
Q < linear layer(f. output dim= x 256)
3:
Q reshape(Q, shape=(batch size, attention heads=8, feature dim=256)
4:
V <
output_dim=8 x 256)
5:
V + reshape(V, shapesseesese attention feature 256t
6:
K + lincar.layer(K, o x 256)
7:
K + reshape(K, whappeeseeueeneellleee len, attention heads=8, feature dim=256)
8:
+ softmax @)
9: + reshape(z, shape= size, 8 x 256))
10:
+ _oveer)
+- linear layerr_, oumput_dim=256)
12:
dropout(z, probability=0.5)
13:
f f layer-norm(f + z/ 22)
14:
return f
21

---

Published as a conference paper at ICLR 2023
Algorithm 12 Spatial IPA Module
function SPATIAL IPA(X; z, F)
2: d <- generate_distance
3: e <
4:
n. [z, dle
5:
V < get value vectoo()e)))())))))
6:
q get.query in local frameen..)
7:
q ー F q
D Bring to global frame Shape: M x k x
8:
X k get neighbour positions(x)
9:
z' f softmax ExE() |q x,, 13)
> Sum is over query points
10:
z' f linear Jayerr_a output_dim=256)
11:
z' f dropout(z', probability=0.5)
f- lnymr_n + 22/v2)
13: return z
Algorithm 13 Cubes Centered on Nodes
1:
function GET.CCBBES.CETEERED. NODDESSS F, V)
2:
affine grid - torch.nn. www.tiooannnnn.nnnnoooooooaaaaaa affine grid(F, shape=(17, 17, 17))
3: t wrrcchnn.uunctional.gg sam.le/ affine.grit grid)
4: return C
Algorithm 14 Rectangles Between Nodes
1: function GET RECTANGLES BETWEEN NODES(X, V)
2:
v f get vectors to nearest neighbours(x)
3: M notation_matric Batttrrxx_pottt__aaaia _t___stor__v
4:
F f- Concatenate(M1, dim=l)
5:
affine grid torch.nn.
www...ionnnnn........ affine grid(F, shape=(5, 1))
6: R t arcc..nn.unctional. iituncttonalggi...... sample(V, ufffie-grri)
7: return R
Algorithm 15 Distance Based Features
1:
function GENERATE DISTANCE BASED FEATURES(F, F)
2:
X
F -[.
3:
hew.rkeeeeeeetniiiioouure..ee k 20)
Shape: M
4: neighbour
locallffaaeeeeeeeoooooooa-aaaaae + F-1 x/e]
Shape: M x k X
5:
neighbour.distance + nneighhoour-positionn
localollaaaaaa
Shape: M
6:
NCaC. + backbone Banckbone_frmeeeee frame.to_pos(( to)
See AlphaFold2 sup. 1.8.4
7:
CoodNaaC.dsstaneee
[osCCC - x
8:
dio_daist + CCCCCCposs(e]...... NCCCC_CCssS [o. 0|1
9:
CtoN distances + [NCaC.pos(e]
NCaC pos(Cposs..... 2
10:
distances + [CatoNCaCCddistacee NtoC_distances,
11:
n +
dim=20, freq=1/50)
12:
reshape(n, shape=(batch_sizee size, 20 * * 20))
13:
n <- linear layer(n, output dim=64)
14:
f [f, n
15:
return f
Algorithm 16 Fully Connected Residual Block
1: function FC_RES BLOCK(F, output_dim)
2: <- linear_layer( output_dim)
3: - f f +
4: f f relu(f)
5:
f layer.norm(f)
6:
return f
22

---

Published as a conference paper at ICLR 2023
Algorithm 17 Backbone Frame Module
function BACKBONE FRAME. MODULE(f, F)
2:
f T- "c_res_block(f
output_dim=220
3:
f f blooef, output )
4:
f fc_res_block(f, output_dim=256)
5:
NCaC.shift + linear Jayer(f, output_dim=9)
6:
NCaC.shift +- reshape(NCaC... CeshstetCCCChttthhii shape 3, 3,3))
7:
NCaC new + Fo NCaC shift
8:
F' affine from Points(NCaC new)
See algorithm 21 in AlphaFold2 sup.
9:
F' < mirror_x_and
> Same convention in AlphaFold2
['.....1 return F
11:
Algorithm 18 Transition Layer
1: function TRANSITION LAYER(f))
2: f res block(f, output dim=256)
3: f fe block(f, output dim=256)
4: felres_block(f, output
5: return f
Algorithm 19 Cryo Key Vectors
1: function GET SET___RYO_K__ JEE.. VECTORS(()
2: + conv3d(R, in_channels=1, caacccenels,, (1ze_l)))2=)133333))
3: + reshape(f, (batch-size size, neighnorrs
4: - relu(layer.norm(f)
5:
linear layer(f, output.dim sigpoutdddmmttttttionn.m. heads x 256)
6: f dropout(f, probabílity=0.5)
7:
reshape(f, (batch size, num neighbours, attention heads, 256))
8: return f
Algorithm 20 Cryo Query and Value Vectors
function GET_CRYO_qU CURRYO_QUEYY.CO SOUFRY_VAIURRREECCEEC WAL.UR_VECTORS(F,. e)
dinppuldiimttttttioo...o heads x 256) 2: q linear Jayerrf,
heaas 3: q - reshape(q. obatch_size, attention heads=8, 256))
4:
+ linear layer(f, oulput-dim=8 256)
5:
v t víc]
7: return reshape(v, (batch size, num neighbours=20, attention heads=8, 256))
6:
Algorithm 21 Cryo Point Features
Require: global_ variables
1: attention_heade +-
2: num fen_uresures 256
3: function DEST_CRYO_POINTE POINT.FEATURES
4: bottleneck(CC, input channelhannls hidden channels, num features 4, expansion=4,
stride=2)
5: f bottleneck(f, inpu.chanee--nnumffaausss hhgpr//seennnttnnsseccccccnncnnn hidden.chaaeelesmmmfaannsss
expan-
sion=4, stride=2)
bottleneck(f. inpu.chhamel--nnnffeuurrss
hhtp://sseuu.uuuuinccncnnncnnnnn hiweeeencccaammmeiiineeeeeetuee expan-
sion=4, stride=2)
7:
bottleneck(f, input channels=uum features, hidden channels=num features expan-
4 stride=2)
8:
= reshape(batch.sizee num fun_fueures, x, 2)
9:
10:
linear layer(f, output.dime attention heads x num featuurus
11:
f + reshape(batch.sizz, attention fcatuurr
12:
return f
23

---

Published as a conference paper at ICLR 2023
A.4
COMPUTATIONAL CHARACTERISTICS
A.4.1 NUMBER OF PARAMETERS
Segmentation network: 61,158,147
GNN total: 192,090,873
Spatial IPA module: 9,659,904 (x 8 layers)
Cryo-EM module: 7,722,581 (x 8 layers)
Sequence attention module: 6,499,348 (x 8 layers)
Torsion angle predictor: 240.806
Backbone update module: 200,457
Local confidence predictor: 198,401
Exiistncee mask predictor: 198,401
Transition layer: 198,144
A.4.2 TRAINING TIME AND COMPUTE
The training was carried out on 8 NVIDIA A100 GPUs. We trained the segmentation network for 400,000 steps with batch size 16. Each step took 0.99 seconds on average, for a total training time of 4.6 days. We trained the GNN for 300,000 steps with a batch size of 16. Each step took 1.73 seconds on average, for total training time of 6 days.
A.4.3 INFERENCE TIME AND COMPUTE
Inference time depends the size of the structure and its sequence length. Below, we report the total time (in minutes) that it took to run inference on a single A100 GPU for all structures in the test set.
7ode: 19.5
7oix: 7.0
7pt6: 104.2
7pt7: 88.9
7qlu: 8.8
7rzy: 9.9
7sba: 132.7
7sjn: 7.5
7szi: 4.2
7tu5: 8.3
7tvz: 14.2
7txv: 15,6
7uck: 283.5
7ugg: 9.1
7um0: 17.0
7unl: 5.0
7wug: 21.1
7xpx: 4.8
7px8: 11.4
7y9u: 5.2
7z1m: 58.0
7sr8: 5.8
8a2q: 17.8
8a3t: 105.3
8dtm: 7.5
7u50: 7.4
7w91: 11.2
7yzk: 5.5
24

---

Published as a conference paper at ICLR 2023
A.5 SEQUENCE SIMILARITY CUTOFF DATASET
To test whether overfitting to structures that similar to those used for training affected our results, we also analyzed the relative performance of ModelAngelo and Deep Tracer on a new test set. For this, we chose 15 recent structures from the PDB that did not contain single protein with a sequence similarity of more than 30% to any of the chains in the training set. As shown in A.33aa) tables 5 and 6 the relative performance of Deep' Tracer and ModelAngelo does not change for this smaller, more unique test set. We note that structure 7ele is only partially built in the PDB deposition, which impacts its C recall.
Sequence Recall
100
80
ModelAngelo
60
40
20
New Dataset
Old Dataset
0
20
40
60
80
100
Deeptracer
Figure 5: Scatter plot of results of the new sequence identity cutoff dataset, ModelAngelo pruned compared to Deeptracer. We note that the results are similar to the main dataset presented in the paper.
25

---

Published as a conference paper at ICLR 2023
PDB
Reso- lution
backbone RMSD
C
C
IDDT
Sequence match
Sequence
recall
precision
recall
8c21
3.51
0.663
59.8
97.3
90.6
96.0
57.4
7vt0
3.40
0.719
75.4
96.5
90.7
95.0
71.6
7uab
3.70
0.610
54.8
98.5
92.6
95.4
52.3
7tnx
3.50
0.577
94.6
98.9
93.2
99.3
94.0
7u58
3.10
0.484
87.3
96.7
95.1
97.9
85.4
8dze
2.99
0.535
96.8
98.5
94.1
98.6
95.4
7cle
3.34
0.619
88.3
4.7
92.7
97.2
85.8
8aa5
2.46
0.410
73.6
99.5
97.1
96.4
71.0
7yer
3.00
0.489
94.6
98.4
95.1
96.0
90.8
7ufg
3.28
0.653
66.0
96.7
91.2
96.2
63.5
7uqx
3.30
0.533
50.5
99.5
93.8
90.2
45.6
7uwq
3.05
0.399
90.8
97.9
97.4
98.8
89.7
7uws
3.47
0.599
80.9
79.0
91.7
97.6
79.0
7wjm
3.30
0.426
98.0
99.6
96.6
98.9
96.9
7usc
3.00
0.290
98.6
98.5
98.6
99.4
98.0
Table 5: ModelAngelo sequence similarity cutoff results
PDB
Reso- lution
backbone RMSD
C
C
IDDT
Sequence match
Sequence
recall
precision
recall
8e21
3.51
1.404
66.9
93.7
82.7
54.5
36.5
7vt0
3.40
1.130
80.9
90.2
83.2
51.4
41.6
7uab
3.70
1,231
83.1
95.7
84.8
31.5
26.2
7tnx
3.50
0.975
95.2
88.0
89.0
79.1
75.3
7u58
3.10
0.807
85.7
96.8
91.8
88.7
76.0
8dze
2.99
0.719
96.4
97.5
92.0
96.2
92.7
7ele
3.34
1.324
96.2
4.9
86.6
71.6
68.8
8aa5
2.46
1.010
72.8
87.8
91.1
56.8
41.4
7yer
3.00
0.795
91.0
98.3
91.8
87.9
80.0
7ufg
3.28
1.193
71.8
96.3
85.3
64.3
46.2
7uqx
3.30
1.505
75.5
96.9
82.9
10.9
8.2
7uwq
3.05
0.904
91.8
98.8
93.0
83.6
76.7
7uws
3.47
1.157
87.9
72.9
85.4
66.0
58.0
7wjm
3.30
0.685
97.6
99.6
92.7
95.6
93.2
7usc
3.00
0.580
95.5
99.3
96.6
94.0
89.7
Table 6: Deeptracer sequence similarity cutoff results
26

---

Published as a conference paper at ICLR 2023
A.6 EXPANDED MODEL COMPARISON
Deeptracer
ModelAngelo
59%
7TXV
81%
41%
8A2Q
70%
68%
7U50
72%
Figure 6: Additional results showing comparisons between Deeptracer (left) and ModelAngelo (right). Both models viwed in color over the outline of the ground truth (deposited) model. The percentages show respective sequence recall for each model. Close-up on 7SJN highlights the incorrect chain connections made by Deeptracer. A global view of the models for 7TXV and 7U50 are shown.
27


Reviews:
[{"review_id": "zmWiyjGaXf", "reviewer": "Reviewer_K3kR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method for automated model building of protein structures in cryo-EM maps. The method utilizes amino acid sequence data and prior knowledge about protein geometries in addition to cryo-EM data. Specifically, it starts with approximate locations of individual residues using a CNN and then uses a GNN that takes in information from sequence data, cryo-EM map and initial positions of atoms to refine the full atom positions of each residue. The main innovation is utilizing several aspects of protein data in contrast to traditional sequence only or cryo-EM only approaches. The results look promising, achieving better sequence recall and backbone root mean squared deviation compared to Deeptracer, though with only 28 evaluations and a longer execution time. The main limitations discussed are sensitivity to resolution, inability to achieve full atomic modeling of nucleic acids, and inability to construct models with unknown sequences. ", "summary_of_the_review": "The method is interesting and works well. However, the manuscript needs significant work to be better presented to the ML community. Model architecture and training details are vague and experiments do not demonstrate the value of individual model components. With some significant work on the text, I think this can be a strong contribution, but I think the manuscript as currently written is of limited value to the ML community.\n\nWhat would raise my score: revise the manuscript to provide detailed architectural and training details and expand the evaluation to demonstrate the value of the components of the model and provide further characterization of the performance as it relates to structural similarity to proteins in the training set.", "strength_and_weaknesses": "Overall, the paper shows promising results. I am, however, concerned by the way the method is presented for a machine learning audience. The method section is vague and does not provide sufficient detail to be able to fully understand and reproduce the method. There is also a lack of detailed descriptions about the neural network architecture and ablation studies. The paper would benefit from further discussion, detailed explanation, and clarification.\n\t\t\n1. Other works combining sequence with cryo-EM: The authors mentioned several other works utilizing protein structure prediction programs to improve output predictions to fit the cryo-EM map (page 3). What are the main differences between those works and ModelAngelo? The authors claimed that “Such approaches are likely to propagate errors in the structure prediction. ”. What is the basis of this claim and what makes ModelAngelo not susceptible to the same problems?\n\n2. Methods section 3.1 describes the coarse location estimation module, but there are many important details left to the readers imagination. How is the dataset split into train, test and validation sets? How is the neural network evaluated and validated? What is the performance of this network? What are the choices of layers for this CNN and what is the reason behind these choices? Are there any ablation studies done for the design? \n\n3. Methods section 3.2: Again, the description of the GNN is vague. There is no description about the training, testing and validation processes nor evaluation of the model performance. Specifically how is this module designed?\n\n4. For eq(1),  χ is the characteristic function of whether x contains a Cα atom. How is this function defined? \n\n5. Time and computing resources are briefly discussed, but there does not seem to be information regarding training time for each part of the method nor the computing resources used. What kind of hardware is needed to train and generate predictions from ModelAngelo?\n\n6. The test dataset is small and the similarity between the train and test structures is unclear. It would be helpful to see evaluation of ModelAngelo on a larger number of structures and to stratify performance based on similarities to the proteins in the training set. Is it possible that some of the test proteins are structurally similar to proteins in the training set? How well does ModelAngelo perform on structures substantially dissimilar to those seen during training? Structural splits based on structural alignment scores or SCOP/CATH classes would be illuminating.\n\n7. How does ModelAngelo compare with more conventional approaches to model building such as Phenix or Rosetta?  we would like to see more metrics in comparison as well as comparison with more approaches such as Phenix and Rosetta. What about MAINMAST?\n\n8. The description of each algorithm could be more detailed. For example, how do get_cryo_key_vectors and get_cryo_point_features work? What does get_query_vector_in_local_frame mean?\t \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "yB8k7spxhw", "reviewer": "Reviewer_9Yaz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a machine learning tool ModelAngelo for automated model building in cryo-EM maps. The tool consists of a sequence of three components, including an improved Residual network for residue segmentation, a GNN-based framework for refining and an HMM for the final-mapping. These components are separately trained in a supervised fashion. ModelAngelo is evaluated in a test dataset of 28 map-model pairs, and shows better performance in sequence recall and backbone RMSD. The authors also discuss limitations and future research of the current implementation. ", "summary_of_the_review": "The paper introduces ModelAngelo for automated model building in cryo-EM maps. Compared with previous work (deeptracer), ModelAngelo largely improved deep learning techniques. It achieves better results at the cost of affordable increased complexity. Overall, I think this work is useful. ", "strength_and_weaknesses": "Strength\n    1. Automated model building in cryo-EM maps is an important problem worth studying. ModelAngelo is among the initiated works that use deep learning techniques. \n    2. Deep learning approaches used in ModelAngelo are technically sound. Some improvements in model architectures and learning objectives are successfully applied.  \n    3. The performance of ModelAngelo is empirically verified by a test dataset. \n\nWeaknesses\n    1. A mathematical formulation for the entire problem is missed. Though the problem is complex and difficult to be solved with an end-to-end framework, the original formulation is still needed at the beginning of the Methods section, followed by a brief introduction of the entire framework, i.e. how to split the entire task into several components and what specific role does each component play. I also suggest authors present a figure to illustrate the overall computing procedure of ModelAngelo.\n    2. Quantitive evaluation results in Figure 3 only reflect middle outputs rather than the final outputs. Figure 4 illustrates the comparison of final results with a single data sample. Thereby, current evaluations are not convincing enough to confirm ModelAngelo’s superiority to competitors. Is it possible for a quantitive comparison on the final outputs?\n    3. Many losses (e.g. (RMSD) loss, backbone RMSD loss, amino-acid classification loss, local confidence score loss, torsion angles loss, and full atom loss) are involved in the learning of GNN. What are the definition of these losses? How do you get the ground truth labels required (if so) for these losses?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "s8ii8kBWia2", "reviewer": "Reviewer_AJLC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work leverages recent advances in large-scale biological machine learning to the task of atomic model fitting into cryo-EM density maps. There are three stages of this pipeline: 1) a residue segmentation model that identifies C_alpha locations in the cryo-EM density map, 2) a GNN-based model which refines the C_alpha locations by sequentially attending over the cryo-EM volume, the underlying protein sequence, and the predicted C_alpha locations, and 3) a postprocessing step where HMMER is used to infer the sequence and relax the geometry of the final atomic model. The model was applied on 28 test cryo-EM maps and compared against DeepTracer.\n", "summary_of_the_review": "This is fantastic work that should be accepted at ICLR. It is a novel application of machine learning to the manual, labor-intensive task of atomic model fitting in cryo-EM maps. Empirical results are strong, and the algorithms are well-designed. The writing of the paper could however be improved.", "strength_and_weaknesses": "Strengths include novelty and good empirical performance. This work is to my knowledge the first example of integrating learning over protein sequences to the task of atomic model fitting into cryo-EM maps. The algorithm is designed to iteratively incorporate information from the cryo-EM volume, geometric considerations from modeling the protein chain, and the underlying sequence.\n\nOne weakness is that the writing could be improved. The writing is clear and explains the design of the method well, however, some statements tend to be anecdotal and not substantiated. e.g. It would be nice to show ablations of the different model components. The details of the algorithms are also missing. \n\nQuestions: \nIt was not clear to me how/when side chain atoms are fit\nThe postprocessing step is also a bit unclear. How is the chain inferred? Why is the HMMER sequence search needed if the input sequence is known?\nHow does the model deal with multiple sequences in a large protein complex?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "A_8aGN3ODd", "reviewer": "Reviewer_7rzk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for the automated building 3D protein models from cryo-EM data. The proposed model combines a CNN with a GNN component in an attention-based architectures and also requires a HMM based post-processing step.\n \nAs a first step, the authors try to approximate the location of residues by solving a segmentation task. The model is a 3D FPN trained on Cryo-EM maps in a supervised fashion, using a customized weighted loss to address data imbalance problems. Training starts using a small, manually curated subset of the data. Results are later used to prune larger datasets. \nSubsequently, the resulting map is processed by 3-stage GNN, which aims at correcting the CNN output. First module of the GNN is Cryo-EM Attention, which uses local neighborhood information by utilizing a novel graph-attention. Second is a Sequence Attention module, which is a BERT-like transformer taking as input the embedding computed by a pre-trained language model. The third and final module is Spatial Invariant Point Attention (IPA), inspired by similarly named module from AlphaFold2. The whole model is trained sequentially using several loss terms and produces 2 types of output. In the unpruned output, predicted chains are almost as-is from the output of the model, with minimal post-processing. This allows for or a larger percentage of the model to be built and gives higher recall. However, this may produce overhead for users in form of manually pruning longer sequences. \nTherefore, the authors provide an additional post-processing mechanism based on hidden Markov models, which produces lower recall results, but higher sequence matching percentages, Authors argue that this is better for end users.", "summary_of_the_review": "The paper proposes a multi-faceted method for solving an important and challenging task. The method contains several technical novelties but fails to fully clarify their details. The problem is described in sufficient detail, but several important issues with the way material is presented make it hard to completely understand the details of the implementation.", "strength_and_weaknesses": "Strength:\n1. The description of the problem in question is sufficiently thorough.\n2. The authors argue that the proposed method allows to build an atomic model de novo in the matter of hours compared to weeks in case of doing it manually.\n3. ModelAngelo outperforms current state-of-the-art DeepTracer in terms of sequence recall and RMSD, although it’s slower due to increased overall computational needs. (Personally I don’t know if there are other baselines the authors could/should compare against.)\n4. The proposed system can also provide confidence of its own predictions thanks to a special loss term.\n5. Contrary to DeepTracer, the proposed method outputs side-chain coordinates.\n6. Authors provide extensive analysis of limitation of the proposed method and possible directions of future work. \n\nWeaknesses:\n1. No schematic description of the model architecture and training procedure. This made it hard for me to be sure what is really proposed and to check if I understood the overall setup of the proposed method (which certainly has many moving parts).\n2. Overall, the proposed algorithm is rather vaguely described. \n3. The same thing is also true for the proposed loss, which I would love to see better on more thoroughly described.\n4. I do know some things about cryo-EM data and workflows, but still I would have appreciated a better description of the data that goes into the proposed model.\n5. Hard to interpret and somewhat limited description of the obtained results.\n6. No ablation studies are presented, leaving many canonical questions on how important different parts of the proposed method are completely unaddressed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "2dIuTCV2Qw", "reviewer": "Reviewer_RqeM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes ModelAngelo, a two-step pipeline to predict the protein structure using cryo-EM maps and protein sequences as input. To do this, ModelAngelo first employs a 3D CNN to predict positions of alpha carbons in protein sequences taking cryo-EM maps as input. Apart from this, they train a separate graph neural network to finish the task of position de-noising by using data augmentation methods, which can be used to fine-tune the protein structure after training. These two steps are separately trained. The employed models of CNN and GNN are task-driven but not new.", "summary_of_the_review": "Overall, because of the major concerns listed in the above weaknesses, I vote for the rejection of this paper.", "strength_and_weaknesses": "## Strengths\n1. They propose to use protein sequences as input to better the structure prediction results, which is different from previous work named Deeptracer. \n2. They employ a GNN to integrate information from different domains including Cryo-em map, protein sequence, and alpha carbon positions.\n\n## Weaknesses:\n1. This paper is not well-written and hard to follow. The problem definitions of these two steps are unclear. (1) Clear definitions of inputs, outputs, and data structures are missing. It is hard to guess how they use these different features from different domains and what information is exactly used for the second step. (2) Some operations are hard to follow. How can they use literature bonds to update the coordinates?\n2. The experimental part is not convincing. The comparison with Deeptracer is problematic because (1) Deeptracer and ModelAngelo use different training data, which means this is not a fair comparison, and (2) compared with Deeptracer, additional protein sequence information is used as input for ModelAngelo, which is a huge assumption to the problem setting.\n3. Ablation studies are missing. The positions of alpha carbons can be obtained by the first stage only. Ablation studies are needed for a better understanding of the whole framework.\n4. The proposed method is not novel enough. The methods for the first stage and the second stage are borrowed from previous papers with some modifications to better fit the problem settings, with limited novelty for the model part.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster