GraphQA: Protein Model Quality Assessment using Graph Convolutional Network
Authors: Federico Baldassarre
David Menéndez Hurtado
Arne Elofsson
Hossein Azizpour
Venue:  (2020)
Keywords: Protein Quality Assessment
Graph Networks
Representation Learning

Abstract: Proteins are ubiquitous molecules whose function in biological processes is determined by their 3D structure.
Experimental identification of a protein's structure can be time-consuming, prohibitively expensive, and not always possible. 
Alternatively, protein folding can be modeled using computational methods, which however are not guaranteed to always produce optimal results.
GraphQA is a graph-based method to estimate the quality of protein models, that possesses favorable properties such as representation learning, explicit modeling of both sequential and 3D structure, geometric invariance and computational efficiency. 
In this work, we demonstrate significant improvements of the state-of-the-art for both hand-engineered and representation-learning approaches, as well as carefully evaluating the individual contributions of GraphQA.

Submitted as conference paper at ICLR 2020
GRAPHQA: PROTEIN MODEL QUALITY ASSESSMENT USING GRAPH CONVOLUTIONAL NETWORK
Federico Baldassarre', Hossein Azizpour', David Menéndez Hurtado², Arne Elofsson'
KTH Royal Insitute of Technology
2 Stockholm University, Science for Life Laboratory and Dep. of Biochemistry and Biophysics
ABSTRACT
Proteins are ubiquitous molecules whose function in biological processes deter- mined by their 3D structure. Experimental identification of protein's 's structure can be consuminn, prohibitively expensive, and not always possible. Al- ternatively, protein folding can be modeled using computational methods, which however are not guaranteed to always produce optimal results.
GRAPHQA is a graph-based method to estimate the quality of protein models, that possesses favorable properties such as representation learning, explicit modeling of both sequential and 3D structure, geometric invariance and computational effi- ciency. In this work, we demonstrate significant improvements over the state-of- the-art for both hand-engineered and representation-leaniie approaches, as well as carefully evaluating the individual contributions of GRAPHQa components.
INTRODUCTION
Protein molecules are predominantly present in biological forms, responsible for their cellular func- tions. Therefore, understanding, predicting and modifying proteins in biological processes essen- tial for medical, pharmaceutical and genetic research. Such studies strongly depend on discovering mechanical and chemical properties of proteins through the determination of their structure.
At the high level, a protein molecule is a chain of hundreds of smaller molecules called amino acids. Identifying protein's 's amino-acid sequence is nowadays straightforward. However, the function of a protein is primarily determined by its 3D structure. Spatial folding can be determined experimen- tally, but the existing procedures are time-consuming, prohibitively expensive and not always possi- ble. Thus, several computational techniques were developed for protein structure prediction (Arnold jet al., |0066] Wang et al. 2017 Xu 2019). So far, no single method is always best, e.g. some protein families are best modeled by certain methods, also, computational methods often produce multiple outputs. Therefore, candidate generation generally followed by an evaluation step. This work focuses Quality Assessment (QA) of computationallly-errii models of protein (Lundstrom ct al. 2001 Won et al.) 2019).
QA, also referred as model accuracy estimation (MAE), estimates the quality of computational protein models in terms of divergence from their native structure. The downstream goal of QA is two-fold: to find the best model in pool of models and refine model based its local quality.
Computational protein folding and design have recently received attention from the machine learn- ing community (Wang al. 2017 Xu 2019 Jones & Kandathil, 2018 Ingraham et al.; 2019b Anand & Huang 2018: Evans et al. 2018] (OT8)AAQQN AlQuraishi (019), while QA has yet to follow. This despite the importance of QA for structural biology and the availability of standard datasets to benchmark machine learning techniques, such as the biannual CASP event (Moult et al.) 1999). The field of bioinformatics, on the other hand, has witnessed noticeable progress in QA for more than decade: from earlier works using artificial neural networks (Wallner & Elofsson, 2006) support vector machines (Ray al., 2012 Uziela et al. (016)) more recent deep learning methods based on ID-CNNs, 3D-CNNs and LSTMs (Hurtado al. 2018 Derevyanko et al. 2018 Pagès et al. 2018 Conover et al. 2019).
In this work, we tackle Quality Assessment with Graph Convolutional Networks, which offer several desirable properties over previous methods. Through extensive experiments, we show significant improvements over the state-of-the-an ahe- and offer informative qualitative and quantitative analyses.

---

Submitted as conference paper at ICLR 2020
ENOGIDER
MESSAGE PASSING
READOUT
2
Czstive
Figure I: Protein Quality Assessment. GRAPUQA predicts local and global scores from protein's graph using message passing among residues with chemical bond or spatial proximity. CASP QA algorithms score protein models by comparison with eegeriienntalyyddtermmiii conformations.
1.1 RELATED WORKS
Protein Quality Assessment (QA) methods are evaluated CASP MMoult et al. 1995) since CASP7 (Cozzetto et al. 2007). Current techniques can be divided into two categories: single- model methods which operate on single protein model estimate its quality (Wallner Elotsson, 2003]. and consensus methods that use consistency between several candidates to estimate their quality (Lundstrom et al. 200T). Single-model methods are applicable lation and in the recent CASPI3 performed comparable to or better than consensus methods for the first time (Cheng et al. 2019). Recent single-model QA works are based on deep learning, except VoroMQA that takes statistical approach atom-level contact area (Olechnovic & Ven- clovas. 2017). 3DCNN adopts a volumetric representation of proteins (Derevyanko et al. (2018). Ornate improves 3DCNN by defining a canonical orientation (Pagès al. [2018). ProQ3D uses a multi-layer perceptron on fixed-length protein descriptors (Uziela et al.] (017). ProQ4 adopts a pre-trained ID-CNN that is fine-tuned in siamese configuration with rank loss (Hurtado al., 2018). VoroMQA and ProQ3D are among the top performers of CASPI3 (Won et al. 2019)).
single protein in iso-
Graph Convolutional Networks (GCNs) bring the representation learning power of CNNs to graph data, and have been recently applied with success to multiple domains, e.g. physics (Gonzalez al., 20188). visual scene understanding (Narasimhan et al., 2018) and natural language understand- ing (Kipf & Welling 2017). Molecules can be naturally represented as graphs and GCNs have been proven effective in several related tasks, including molecular representation learning (Duvenaud et al. 2015), protein interface prediction (Fout et al.) (017), chemical property prediction (Niepert itt al. 2016 Gilmer et al. 2017 Li et al. 2018a). drug-drug interaction (Zitnik et al. 2018), drug- target interaction (Gao et al.) 2018) molecular optimization (Jin et al. 2019), and generation of proteins, molecules and drugs (Ingraham et al. 2019a; You et al.) 2018 Liu et al. 2018 et al.)
2018b Simonovsky & Komodakis 2018). However, to the best of our knowledge, GCNs have never been applied to the problem protein quality assessment.
1.2 CONTRIBUTIONS
This work is the first to tackle QA with GCN which bring several desirable properties over previous methods, including representation learning (3DCNN, Ornate), geometric invariance (VoroMQA, Ornate), sequence learning (ProQ4, AngularQA), explicit modeling of 3D structure (3DCNN, Ornate, VoroMQA) and computational efficiency.
Thanks these desirable properties, simple GCN setup achieves improved results compared to the more sophisticated state-of-the-art methods such as ProQ4. This is demonstrated through extensive experiments on multiple datasets and scoring regimes.
Novel representation techniques employed to explicitly reflect the sequential (residue sepa- ration) and 3D structure (angles, spatial distance and secondary structure) of proteins. Enabled by the of GCN, combine the optimization of local and global score for QA, improving over the performance of a global-only scoring method.
Through an extensive set of ablation studies, the significance of different components of the method, including architecture, loss, and features, are carefully analyzed.
PyTorch implementation, datasets and experiments: gethbb..ombblllassseee/eeeeiinnn
2

---

Submitted as conference paper at ICLR 2020
2
METHOD
We start describing method by arguing for representation of protein molecules graphs in learning tasks, then we define the problem of protein quality assessment (QA), and finally we present the proposed GRAPHQA architecture.
2.1 PROTEIN REPRESENTATION AS GRAPIIS
Proteins are large molecular structures that perform vital functions in all living organisms. At the chemical level, a protein consists one or more chains of smaller molecules, which we interchange- ably refer residues for their role in the chain,
amino acids for their chemical composition.
The sequence of residues = { a; } that composes protein represents its primary structure, where a; is one of the 22 amino acid types. The interactions between neighboring residues and with the en- vironment dictate how the chain will fold into complex spatial structures that represent the protein's secondary structure and tertiary structure.
Therefore, for learning tasks involving proteins, a suitable representation should reflect both the identity and sequence of the residues, i.c. the primary structure, and geometric information about the protein's s arrangement in space, i.e. its tertiary structure. Some works use RNN or 1D-CNN model proteins as sequence with the spatial structure potentially embedded in the handcrafted residue features (Hurtado et al. 2018 Conover et al.) 2019). Other recent works explicitly model proteins' spatial structure using 3D-CNN but ignore its sequential nature (Derevyanko et al. 2018 Pages et al.) 2018). We argue that graph-based learning can explicitly model both the sequential and geometric structures of proteins. Moreover, it accommodates proteins of different lengths and spatial extent, and is invariant to rotations and translations.
In the simplest form, protein can be represented as a linear graph, where nodes represent amino acids and edges connect consecutive residues according the primary structure. This set of edges, which represent the covalent bonds that form the protein backbone, can be extended to include the interactions between non-consecutive residues, e.g. through Van der Waals forces or hydrogen bonds, commonly denoted as contacts. By forming an edge between all pairs of residues that are within chemically reasonable distance of each other, the graph becomes rich representation of both the sequential and geometric structure of the protein (figure 2). We refer to this representation, composed of residues, bonds and contacts, as the protein graph:
p= = (1 (fr)).
{ bund -i-Ji 11
City
contact
|i -i- 1- 1, C,ll S dmaxx) dmmx)):
}):
where i,j 1,..., [S] are residue indices, C= { (ar, y: 2); represents the spatial arrangement of the residues, i.c. the protein's conformation, and dmax is cutoff distance for contacts.
With the protein's structure encoded in the graph, additional residue and relationship features can be encoded nodes and edges attributes, v; and ei,j respectively. Section 3.2 describes, in detail, an attribution that preserves the sequence information and 3D geometry while remaining invariant to rotation.
Symet
(a) Sequential
(b) Spatial
(c) Graph
Figure 2: Protein representations for learning. Sequential representations for L.STM or ID-CNN fail to represent spatial proximity of non-consccutive residues. Volumetric representations for 3D- CNN fail instead to capture sequence information and is not rotation invariant. Protein graphs ex- plicitly represent both sequential and spatial structure, and geometrically invariant by design.
3

---

Submitted as a conference paper at ICLR 2020
2.2 PROTEIN QUALITY ASSESSMENT
Experimental identification of a protein's native structure can be time consuming and prohibitively expensive. Alternatively, computational folding methods used to generate decoy conformations for a specific target protein. Since no single method is consistently best, a Quality Assessment (QA) step used identify the decoys that most correctly represent the native structure.
If the native structure native is experimentally determined, the quality of decoy can be measured by comparison, e.g., in the CASP challenge, decoys submitted for target are scored against the unreleased native structure. Some comparative algorithms compute global (per decoy) scores, which can be used for ranking and represent the principal factor for CASP, while others produce local (per residue) scores which help identify incorrect parts of a decoy (Uziela et al., 2018).
In most scenarios, however, the native structure is not available and quality must be estimated based on physical and chemical properties of the decoy, e.g. in drug development, it would be unpractical to synthesize samples of novel proteins and researchers rely on computational folding and quality assessment instead.
Here we introduce GRAPHQA, a graph-based QA neural network that learns to predict local and global scores, with minimal feature and model engineering, using existing datasets of scored pro- teins. In this paper, we train GRAPHIQ on two widely-used scoring algorithms: the Global Distance Test Total Score (Zemla 2003), which is the official CASP score for decoy-level quality assessment, and the Local Distance Difference Test (Mariani et al. 2013)), residue-level score. We denote them LDDT(C" (mmativ)..
as qa := GDT TS(Cd, Cmative) and { 9
With GRAPHQA((P) and GRAPHQA (P) denoting the network's local and global predictions for an input P, the learning objective is to minimize the following Mean Squared Error (MSE) losses:
Cs = Σ GRAPHQAA(P) -
Cg = (PP)PHQA((P)-QQ)
(2)
Note that, for the sole purpose of sorting decoy according to ground truth quality, training with a ranking loss would be sufficient (Derevyanko et al.) 2018). Instead, MSE forces the output to match the quality score, which a harder objective, but results a network can be more easily inspected and possibly used to improve existing folding methods in an end-to-end fashion (section 4.2).
2.3 GRAPHQA ARCHITECTURE
GRAPHQQ is a graph convolutional network that operates on protein graphs using the message- passing algorithm described in BBattgliia et al. (2018). The building block of GRAPHQA, graph layer, takes protein graph as input (with an additional global feature u), and performs the following propagation steps to output a graph with updated node/edge/globall features and unchanged structure:
e'us Φ² (ci.j, U2 Uz.
Update edges
ē' = p (f (e',5) })
Aggregate all edges Aggregate all nodes Update global features
E = pp (f 0f = o² (E,, vú u)
Aggregate edges
5' = p" nu (1 u' 6t (ε', 1' (ε',
Update nodes
where o represent three update functions that transform nodes/edges/global features (e.g. a MLP), and p represent three pooling functions that aggregate features at various levels (e.g. sum or mean).
Similarly CNNs, multiple graph layers are stacked together to allow local information to prop- agate to increasingly larger neighborhoods (i.e. receptive field). This enables the network to learn quality-related features at multiple scales: secondary structures in the first layers, e.g. a-helices and B-sheets, and larger structures in deeper layers e.g. domain structures and arrangements.
The GRAPHQA architecture is conceptually divided in three stages (figure D. At the input, the en- trans- formation and adds a global bias. Then, at its the core, L message-passing layers operate on the en- coded graph, leveraging its structure to propagate and aggregate information. The update functions o consist of Linear- Linear-Dropout- Dropout- R transformations, with the size of the linear layers progressively decreasing. We use average pooling for the aggregation functions p. since preliminary experiments with max/sum pooling performed poorly. Finally, the readout layer outputs local and global quality operation to the latest node and global features, respectively.
coder increases the node and edge features' dimensions through (x(Linerr- Dropout-R
scores by applying a
4

---

Submitted as conference paper at ICLR 2020
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Following the common practice in Quality Assessment, we use the data from past years' editions of CASP, encoppasinn several targets with multiple scored decoys each. Removing all sequences with |S| < 50 from CASP 7-10 results in a dataset of 1000 scored decoys (P, g } q9)) which we randomly split into training set (402 targets) and validation set for nyper-parameter optimization (35 targets). CASP and 12 are set aside for testing against top-scoring methods (table
We evaluate the performances of GRAPHQA on the following standard metrics. At the global level, we compare the predicted and ground-trute truth GDT-TS scores and compute: Root Mean Squared Er- ror (RMSE), Pearson correlation coefficient computed across all decoys of all targets (R), and Pear- son correlation coefficient computed on a per-target basis and then averaged over all targets (Ronge). At the local level, compare the predicted and ground-truth LDDT scores and compute: RMSE, Pearson correlation coefficient computed across all residues of all decoys of all targets (R), and Pearson correlation coefficient computed on a per-decoy basis and then averaged over all decoys of all targets (mmodel). Of these, we focus on Rerrer and Rmodel, which respectively measure the ability rank decoys by quality and distinguish the porectly-predictee parts of model from those that need improvement. A detailed description of these and other metrics can be found in appendix
3.2 FEATURES
Node features The node attributes v, of a protein graph P represent the identity, statistical, and structural features of the i-th residue. We encode the residue identity using one-of-22 encoding of the corresponding amino acid. Following Hurtado et al.] (2018), we also add two residue-level statistics computed using Multiple Sequence Alignment (MSA) (Rost et al., 1994), namely self- information and partial entropy, each described by a 23-dimensional vector. Finally, we add a 14-dimensional vector of 3D spatial information including the dihedral angles, surface accessibility and secondary structure type as determined by DSSP (aassch & Sander 1983).
Edge features An edge represents either a contact or a bond between two residues i and j w.r.t. to the conformation C = { (z. y, 2)z }. An edge always exists between two consecutive residues, while non-consecutive residues are only connected if ||C, - C2 < dmax, with dmax optimized on the vali- dation set. We further enrich this connectivity structure by encoding spatial and sequential distances as an 8D feature vector Gi.j- Spatial distance encoded using a radial basis function exp(-dkj/o) with a determined on the validation set. Sequential distance is defined as the number of amino acids between the two residues the sequence and expressed using a separation encoding, i.e. a one-hot encoding of the separation jl according to the classes {0, 1, 2, 4,5 10,> 10
3.3 OPTIMIZATION AND HYPERPARAMETER SEARCH
The MSE losses in equation 2 are weighted as Liot deCe + XyC, and minimized using Adam Optimizer (Kingma & Ba 2014 4h with La regularization. GRAPIIQ is significantly faster train than LSTM or 3D-CNN methods, e.g. 35 epochs takes ~ hours on one NVIDIA 2080Ti GPU with batches of 200 graphs. This allows to perform extensive byper-paramen search. Table 4 reports the search space, as well as the parameters of the model with highest Runger on the validation set.
4 EVALUATION
We compare GRAPHQA with the following methods, chosen either for their state-of-the-art perfor- mances or because they represent a class of approaches for Quality Assessment. ProQ3D (Uziela et al. 2017) computes fixed-size statistical descriptions of the decoys in CASP 9-10, including Rosetta energy terms, which are then used to train a Multi Layer Perceptron on quality scores. In ProQ4 (Hurtado et al. 2018), DD-NNN is trained predict LDDT scores from a vectorized repre- sentation of protein sequences, a global score is then obtained by averaging over all residues. ProQ4 is pretrained on large dataset of protein secondary structures and then fine tuned on CASP 9-10 using a siamese configuration to improve ranking performances. Their results are reported on both CASP 11, which is used validation set, and CASP 12. 3DCNN (Derevyanko al. 2018 trains
5

---

Submitted as a conference paper at ICLR 2020
Table 1: Comparison of state-of-the-art QA methods. At the residue level we compare LDDT scores and report Pearson correlation and Pearson correlation per model. At the global level we compare GDT.TS scores and report Pearson correlation and Pearson correlation per target.
CASP 11
CASP 12
GDT.TS
LDDT
GDT.TS
LDDT
Rungel
R
Rmode
Rarget
R
Rmodel
ProQ3D
.772
.52
.84
.61
.806
.609
ProQ4
.77
.56
.772
.516
VoroMQA
.651
.457
.605
Rwplus
.206
-.096
.417
AngularQA
.651
.439
3D CNN
.629
.421
.607
Ornate
.637
.386
.670
.491
GRAPHQA
.910
.740
.855
.610
.843
.745
.843
.573
GRAPHQARAW
.836
.609
.799
.529
.816
.673
.796
.507
CNN on three-dimensional representation of atomic densities to rank the decoys in CASP 7-10 according to their GDT _TS scores. Notably, no additional feature is used other than atomic structure and type, however, the fixed-size volumetric representation of this method sensitive to rotations and does not scale well with protein size. Ornate (Pagès et al. 2018) applies similar 3D approach to predict local CAD-scores (Olechnovic Venclovas| 2017) and achieves rotation invariance by specifyyng canonical residue-centered orientation. Although optimized for local scoring, the av- erage of the predicted scores is shown to correlate well with GDT_TS, AngularQA (Conover et al. 2019) feeds sequence-like representation of the protein structure to LSTM to predict GDT-TS scores. The LSTM network is trained on decoys from 3DRobot and CASP 9-11, while CASP 12 is used for model selection and testing. VoroMQA and RWplus (Olechnovič & Venclovas 2017 Zhang & Zhang 2010) are two statistical potential methods that represent an alternative to the other machine-learning based methods.
Table compares the performances of GRAPHQA and other state-of-the-att methods on CASP 11 and 12, while figure a graphical representation of true vs. predicted scores for all target in CASP 12, and an example funnel plot for the decoys of single target. A more in-depth evaluation on the stage and stage splits of CASP 11, 12, 13 and CAMEO can be found in appendix F
Of all methods, only GRAPHQA and ProQ4 co-optimize for local and global predictions, the former thanks to the graph-based architecture, the latter thanks its siamese training configuration (the re- sults reported for ProQ3D refer to two separate models trained for either local or global scores). At the local level, our method proves to on par or better than ProQ3D and ProQ4, demonstrating the ability to evaluate quality at the residue level and distinguishing correctly predicted parts of the pro- tein chain. At the global level, significantly higher It and Riarger metrics indicate than GRAPIIQA is more capable than other state-of-the-an art methods at ranking decoys based on their overall quality. As shown in our ablation studies (section 5). -engineeeee features like MSA and DSSP contribute to the performances of GRAPHQA, yet wish prove that our method can learn directly from raw data. GRAPHQARA RAw is a variant that relies uniquely on the one-hot encoding of amino acid identity, similarly to how 3D-CNN and Ornate employ atomic features only. The results for GRAPHQA RAw show that, even without additional features, our method outperforms purely representation-lcariig methods.
4.1 ABLATION STUDIES
In this section we analyse how various components of our method contribute to the final perfor- mance, ranging from optimization and architectural choices to protein feature selection. Unless stated otherwise, all ablation studies follow the training procedure described in section 3.3 for a lower number of epochs. We report results CASP 11 mean and standard deviation of T0 runs.
Local and global co-optimization We investigate the interplay between local and global predic- tions, specifically whether co-ttmmizinng for both is beneficial or detrimental. At the global level, models trained to predict only global scores achieve a global RMSE of 4.007, whereas models trained predict both local and global scores obtain 0..177 .006, suggesting that local scores can

---

Submitted as a conference paper at ICLR 2020
provide additional information and help the assessment of global quality, At the local level instead, co-optimization does not seem to improve performances: models trained uniquely on local scores achieve local RMSE of 0.121 0002 while models trained to predict both obtain 0.123=.004.
Connectivity and Architecture In this study, we test the combined effects of the depth of the network L and the cutoff value d On the one hand, every additional to aggregate information from a neighborhood that is one hop larger than the previous, effectively extending the receptive field at the readout. On the other hand, the number of contacts included in the graph affects its connectivity and the propagation of messages, e.g. low dmax correspond to a low average degree and long shortest paths between any two residues. and vice versa (section B.22).
layer allows
Thus, an architecture that operates on pparsely-connecee layers to achieve the same holistic view of shallow network operating on denser representations. However, this trade off is only properly exposed if u, o², p² are removed from the architecture. In fact, this global pathway represents a propagation shortcut that connects nodes the graph and sidesteps the limitations of shallow networks. With the global pathway disabled, global predictions are computed in the readout layer by aggregating node features from the last MP layer.
graphs will require more message- passing
reports the RMSE obtained by networks of different depth with global path, operating on protein graphs constructed with different cutoff values. As expected, the shallow 3-layer architecture requires more densely-connecte Surprisingly, local predictions seem to be more affected by these factors than global predictions, suggesting that large receptive field is important even for local scores.
inputs
achieve the same performances of the 9-layer network.
Node and Edge Features We evaluate the impact of node and edge features on the overall pre- minimal rep-
diction performances (figure 5). For the nodes, we use the amino acid identity resentation and combine it with: a) DSSP features, b) partial entropy, c) self information, d) both DSSP and MSA features. All features improve both local and global scoring, with DSSP features being marginally more relevant for LDDT. For the edges, we evaluate the effect of having either: a) a binary indicator of bond/contact, b) geometric features, i.e. the euclidean distance between residues, c) sequential features, i.e. the categorical encoding of the separation between residues, d) both distance and separation encoding. Progressively richer edge features seem to be benefit LDDT predictions, while little improvement can be seen at the global level.
4.2 VISUALIZATIION AND EXPLAINABILITY
The design of GRAPHQA makes suitable not only for scoring, but also to identify refinement opportunities for computationalll-creade native structure of its target for most of the sequence, but one extremity to which both GRAPIIQ and LDDT assign low local scores. Unlike LDDT, however, GRAPIIQA is fully differentiable and the trained model can be used to explain the factors that influenced low score and provide useful feedback for computational structure prediction.
decoys. Figure decoy that correctly models the
A simple approach for explaining predictions of diffcrentiable function f(x) is Sensitivity Anal- ysis (Baehrens et al., 2010, Simonyan et al. [0144, which uses V to measure how variations in the input affect the output. In figure 6 we consider the scores predicted for two different residues and compute the magnitude of the gradients w.r.t. the edges of the graph. Interestingly, GRAPIQA
GDT
LDDT
LDOT
GDT
TDS44
36-
.17
Loyar
Magar 1.45
Prak coo
.44
.2
(a)
(b)
(c)
Figure 3: Histograms of true vs. predicted LDDT
Figure 4: Trade-off between number of message-passing layers and connectivity of the protein graph.
(a) and GDT TS (b) scores on CASP 12, (c) funnel plot of the decoys of target T0944 (PDB 5ko9).
7

---

Submitted as conference paper at ICLR 2020
Amoze (LDDT)
Aerget (GDT-15)
Amedal (LDDT)
Rurget (GDT-TS)
Residues
Edge Type
Sep Erc
0.60 0.55 0.65
6.70
Figure 5: Ablation study of node and edge features. All node features improve both local and global scoring, with DSSP features being marginally more relevant for LDDT (left). Richer edge features benefit LDDT predictions, while little improvement can be seen at the global level (right).
is able to capture quality-related dependencies not only in the neighborhood of the selected residues, but also further apart in the sequence.
Finally, we measure whether the global predictions of GRAPHQA could be used to improve the contact maps used by computational methods build protein models. If the network has learned meaningful scoring function for decoy, then the gradient of the score w.r.t. the contact distances should aim in the direction of the native structure. Considering all decoys of all targets CASP 11, we obtain an average cosine similarity cos (ÖGrAPHQA) /öd, dleecay dnaiive) of 0.14+.08. which suggests that gradients can be used as coarse feedback for end-to-end protein model prediction.
|VV,GraphONoll
||V,GraphOAfull
20
20
40
(a) Predicted
(b) True
(c) Native
(d)
(c)
Figure 6: One decoy of T0773 in CASP1L. Both GRAPHQA (a) andLDDT assign low local scores to segment of the decoy (in red), highlighting a discrepancy w.r.t. the native structure (c). The gradient magnitude w.r.t. the edges of the predicted LDDT score for residues 20 (d) and 60 (e) reveal long range dependencies inside the protein graph.
CONCLUSION
For the first time applied graph convolutional networks to the important problem of protein qual- ity assessment (QA). Since proteins are naturally represented as graphs, GCN allowed us to collect the individual benefits of the previous QA methods including representation learning, geometric in- variance, explicit modeling of sequential and 3D structure, simultaneous local and global scoring, and computational efficiency. Thanks these benefits, and through an extensive set of experiments, we demonstrated significant improvements upon the state-of-the-an results on various metrics and datasets and further analyzed the results via thorough ablation and qualitative studies.
Finally, we wish that Quality Assessment will gain popularity in the machine learning community, that could benefit from several curated datasets and ongoing regular challenges. We believe that richer geometric representations, e.g. including relative rotations, and raw atomic representations could represent an interesting future direction for learning-based Quality Assessment.
8

---

Submitted as conference paper at ICLR 2020
REFERENCES
Mohammed AlQuraishi. End-to-end differentiable learning of protein structure. Cell systems, 8(4): 292-301, 2019.
Namrata Anand and Possu Huang. Generative modeling for protein structures. In Advances in Neural Information Processing Systems, pp. 7494-7505, 2018.
Konstantin Amold, Lorenza Bordoli, Jürgen Kopp, and Torsten Schwede. The swiss-model
workspace: web-based environment for protein structure homology modelling. Bioinformat- ics, 22(2):195-201, 2006.
David Bachrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert Müller. How to explain individual classification decisions. Journal of Machine Learning Research, 1 11(Jun):1803-1833 2010.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanzhez- Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 2018.
J. Cheng, M.H. Choe, A. Elofsson, K.S. Han, J. Hou, A.H.A. Maghrabi, L.J. McGuffin, D. Menendez-Hurtadee K. Olechnovic, T. Schwede, G. Studer, K. Uziela, Venclovas, and B. Wallner. Estimation of model accuracy in CASPI3. Proteins, Jul 2019. doi: 10. 10.1002/prot. 25767.
Matthew Conover, Max Staples, Dong Si, Miao Sun, and Renzhi Cao. Angularqa: protein model quality assessment with Istm networks. Computational and Mathematical Biophysics, 7(1):1-9, 2019.
D. Cozzetto, A. Kryshtafovych, M. Ceriani, and A. Tramontano. Assessment of predictions in the model quality assessment category. Proteins, 69 Suppl 8:175-183, 2007. doi: 10. 10.000o/ 21669.
Derevyanko, S Grudinin, Y Bengio, and G Lamoureux. Deep convolutional networks for quality assessment of protein folds. Bioinformatics (Oxford, England), 34(23):404-40533, 2018.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Eapuru-Guzik, and Ryan P Adams. Convolutional networks on graphs for learning molecular fingerprints. In Advances neural information processing systems, pp. 2224-2232, 2015.
Evans, Jumper, Kirkpatrick, Sifre, T Green, Qin, A Zidek, Nelson, A Bridgland, H Penedones, et al. De novo structure prediction with deeplearning based scoring. Annu Rev Biochem, 77(363-382);6, 2018
Alex Fout, Jonathon Byrd, Basir Shariat, and Asa Ben-Hur, Protein interface prediction using graph convolutional networks. In Advances in Neural Information Processing Systems, pp. 6530-653 6539, 2017.
Kyle Yingkai Gao, Achille Fokoue, Heng Luo, Arun Iyengar, Sanjoy Dey, and Ping Zhang. In- terpretable drug target prediction using deep neural representation. In IJCAI, pp. 3371-3377, 2018.
Justin Gilmer. Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1263-1272. JMLR. org, 2017.
Alvaro Gonzalez, Nicolas Heess, Jost Tobias Springenberg. Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning (ICML), pp. 4467-4476. 2018.
Juergen Haas, Steven Roth, Konstantin Arnold, Florian Kiefer, Tobias Schmidt, Lorenza Bordoli, and Torsten Schwede. The protein model portala comprehensive resource for protein structure and model information. Database, 2013, 2013.

---

Submitted as conference paper at ICLR 2020
David Menéndez Hurtado, Karolis Uziela, and Arne Elofsson. Deep transfer learning in the assess- ment of the quality of protein models. arXiv preprint arXiv:/804.06288. 2018.
John Ingraham, Vikas K Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. NeurfPS, 2019a.
John Ingraham, Adam Riesselman, Chris Sander, and Debora Marks. Learning protein structure with a differentiable simulator. ICLR, 2019b.
Wengong Jin, Kevin Yang. Regina Barzilay, and Tommi Jaakkola. Learning multimodal graph-to- graph translation for molecular optimization. ICLR, 2019.
D.T. Jones and S.M. Kandathil. High precision in protein contact prediction using fully convolu- tional neural networks and minimal sequence features. Bioinformatics, 34(19):3308-3111 3315, Oct 2018. doi: 10. 10..00//ioinfoomatts//tyy.a
Wolfgang Kabsch and Christian Sander. Dictionary of protein secondary structure: pattern recog- nition of hydrogen-bonded and geometrical features. Biopolymers: Original Research on Biomolecules, 2212))2577-26337 1983.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In 5th International Conference on Learning Representations, ICLR, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. URL Forum?id=SSa Sarum?id=SJUAayql
net//
Ruoyu Li, Sheng Wang, Feiyun Zhu, and Junzhou Huang. Adaptive graph convolutional neural networks. In Thirty-Second AAAI Conference Artificial Intelligence, 2018a.
Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning deep generative models of graphs. arXiv preprint arXiv:/803.033244 2018b.
Qi Liu, Miltiadis Allamanis, Mare Brockschmidt. and Alexander Gaunt. Constrained graph varia- tional autoencoders for molecule design. In Advances in Neural Information Processing Systems, pp. 7795-7804, 2018.
J. Lundstrom, L. Rychlewski, J. Bujnicki, and A. Elofsson. Pcons: a neural-network-bbased consen- sus predictor that improves fold recognition. Protein Sci, 10(11):2354-236: 2362222222222,222-2 Nov 2001.
Valerio Mariani, Marco Biasini, Alessandro Barbato, and Torsten Schwede. Iddt: a local
superposition--fee
for comparing protein structures and models using distance difference tests. Bioinformatics, 29(21): 2013.
J. Moult, J.T. Pedersen, R. Judson, and K. Fidelis. A large-scale experiment to assess protein struc- ture prediction methods. Proteins, 23(3):i Nov 1995. doi: 10.1002/pr0..34223000..
Moult, T. Hubbard, K. Fidelis, and J.T. Pedersen. Critical assessment of methods of protein structure predictions (casp): Round iii. Proteins, Suppl 3:2-6, 1999.
Medhini Narasimhan, Svetlana Lazebnik, and Alexander Schwing. Out of the box: Reasoning with graph convolution nets for factual visual question answering. In Advances in Neural Information Processing Systems NIPS, pp. 2654-2665, 2018
Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural net- works for graphs. In International conference on machine learning, pp. 2014-2023, 2016.
K. Olechnovic and C. Venclovas. VoroMQA: Assessment of protein structure quality using inter- atomic contact areas. Proteins, 1316)11 Jun 2017. doi: 10.1002//rot.25278..
Kliment Olechnovië and Česlovas Venclovas. Voromqa: Assessment of protein structure quality using interatomic contact areas. Proteins: Structure, Function, and Bioinformatics, 85(6): 1145, 2017.
10

---

Submitted as a conference paper at ICLR 2020
Guillaume Pagès, Benoit Charmettant, and Sergei Grudinin. Protein model quality assessment using 3d oriented convolutional neural networks. bioRxiv, pp. 432146, 2018.
Christoph Peters, Konstantinos D Tsirigos, Nanjiang Shu, and Arne Elofsson. Improved topology prediction using the terminal hydrophobic helices rule. Bioinformatics, 32(8):1158-1162 2015.
Arjun Ray, Erik Lindahl, and Björn Wallner. Improved model quality assessment using proq2. BMC Bioinformatics, 13(1):224, 2012.
Burkhard Rost, Chris Sander, and Reinhard Schneider. Redefining the goals of protein secondary structure prediction. Journal of molecular biology, 235(1):13-26, 1994.
Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In International Conference Artificial Neural Networks, pp. 412- 422. Springer, 2018.
Karen Simonyan. Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. 2nd International Conference on Learning Representations, ICLR 2014, Bansf, AB, Camada, April 14-16, 2014, Workshop Track
Proceedings, 2014. URL http:tttt///. / /tapii//arvvv..o org/ www.or///////////////o////e///e/ abs//3// 6034
K Uziela, Hurtado Menéndez, N Shu, B Wallner, and A Elofsson. Proq3d: improved model quality assessments using deep learning. Bioinformatics (Oxford, England), 13110):157,,200 2017.
Karolis Uziela, Nanjiang Shu, Bjöm Wallner, and Arne Elofsson. Proq3: Improved model quality assessments using rosetta energy terms. Scientific reports, 6:33509, 2016.
Karolis Uziela, David Menéndez Hurtado, Nanjiang Shu, Bjöm Wallner, and Arne Elofsson. Im- proved protein model quality assessments by changing the target function. Proteins: Structure, Function, and Bioinformatics, 86(6):6546663, 2018.
B. Wallner and A. Elofsson. Can correct protein models be identified? Protein Sei, 12(5):1073- 1086, May 2003. doi: 10. 10/ps.0236803.
B Wallner and A Elofsson. Can correct regions in protein models be identified. Protein Science, 15 (4):900-913, 2006.
Sheng Wang. Siqi Sun, Zhen Li, Renyu Zhang, and Jinbo Xu. Accurate de novo prediction of protein contact map by ultra-deep learning model. PLoS computational biology, 13()):e1003324, 2017.
Jonghun Won, Minkyung Baek, Bohdan Monastyrskyy. Andriy Kryshtafovych, and Chaok Seok. Assessment of protein model structure accuracy estimation in casp Challenges in the era of deep learning. Proteins: Structure, Function, and Bioinformatics, 2019.
Jinbo Xu. Distance-based protein folding powered by deep learning. Proceedings of the National Academy of Sciences, 16863)) 2019.
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. In S. Bengio, H. Wallach, H. Larochelle, and R. Garnett (eds.), Advances in Neural Information Processing
K. Grauman, N. Systems (NeurIPS) 31, 2018
Adam Zemla. Lga: a method for finding 3d similarities in protein structures. Nucleic acids research, 31(13):337--3374 2003.
Jian Zhang and Yang Zhang. novel side-chain orientation dependent potential derived from random-walk reference state for protein fold selection and structure prediction. PloS one, 5(10): e15386, 2010.
Marinka Zitnik, Monica Agrawal, and Jure Leskovec. Modeling polypharmacy side effects with graph convolutional networks. Bioinformatics, 34(13):447-ii66, 2018.

---

Submitted as conference paper at ICLR 2020
A PROTEIN QUALITY ASSESSMENT
For the interested reader, describe here in more detail how the Global Distance Test Total Score (Zemla 2003) and the Local Distance Difference Test (Mariani et al. 2013) are computed. Further- more, provide an intuition
what the benefits and downsides of each method are and motivate why better quality assessment should consider both global measure and a local measure.
Global Distance Test Total Score (GDT TS) Global Distance Test Total Score (GDT global-level score obtained by first superimposing the structure of a decoy to the experimental struc- ture using an alignment heuristic, and then computing the fraction of residues whose position is within certain distance from the corresponding residue in the native structure (figure (7). This percentage computed different thresholds and then averaged produce a score in the range [0, 100), which we rescale between 0 and I (table 2
Figure 7: GDT.TS
Table 2
|CH
Cnnatve
2
<
5
<
10
0.6 A
X
X
1.2 Á
X
X
X
1.9 Á
X
X
X
2.5 Á
X
X
6.3 Á
20%
60%
80%
100%
Local Distance Difference Test (LDDT) Local Distance Difference Test (LDDT), is residue-level score that does not require alignment of the structures and compares instead the local neighborhood of every residue, in the decoy and in the native structure. If we define the neighborhood of residue as the set of its contacts, i.e. the set of other residues that lie within certain distance from it, express the quality that residue as the percentage contacts that shares with the corresponding residue in the native structure.
Figure& 8: Example of LDDT scoring for residue 7: the residues within radius R2 are {,,8, 10
the native structure (left) and { (,, 8 for the decoy (right); at radius R2 we have { (3, 6, 8, 9, 10, 11 the native structure (left) and 33 6, 9, 10 } for the decoy (right).
12

---

Submitted as conference paper at ICLR 2020
B DATASETS AND STATISTICS
B.1 DATASETS
We consider all decoys of all target included in CASP 7-13, excluding proteins whose sequence is shorter than 50 residues and targets that have been canceled by the organizers. In addition to CASP datasets, we test our method on all targets published in CAMEO (Haas and December 2017.
2013) between July
Table 3: Datasets from previous CASP editions
Dataset
Targets
Models
Usage
CASP 7
95
19591
CASP 8
122
34789
Train/Val
CASP 9
117
34946
CASP 10
103
26254
CASP 11
83
16094
CASP 12
40
6924
Test
CASP 13
20
7838
CAMEO
676
20891
B.2 PROTEIN GRAPHS STATISTICS
The cutoff value dmax determines which edges are included in the graph and, consequentially, its connectivity. low cutoff implics a sparsely connected graph, with few edges and long paths between nodes. A higher cutoff yields a denser graph with more edges and shorter paths.
In figure we report some statistics about number edges, average degree and average shortest paths, evaluated at different cutoff values on 1700 decoys from CASP 11.
Number of edges
Avg Degroe
Aug Shartest Path
12000
40-
10000
8000
6000
20
40
12 14
14
14
Cutoff
Cufoff drac
Cutoff dras
Figure 9: Number of edges, average degree, and average shortest paths at different cutoff' values (sample size: 1700 decoys from CASP 11)
13

---

Submitted as conference paper at ICLR 2020
C GRAPHQA ARCHITECTURE
In this section, we illustrate in more detail the structure of the GRAPHQA architecture, as well as the hyperparameter space that was explored optimize performances on the validation set.
C.I MESSAGE-PASSING LAYERS
Within GRAPHQA, protein structure is represented as a graph whose nodes correspond to residues and whose edges connect interacting pairs of amino acids. At the input, the features of the i-th residue are encoded in node feature vector v,. Similarly, the features of the pairwise interaction between residues and j are encoded in an edge feature vector et.j- global bias term u is also added to represent information that not localized to any specific node/edge of the graph.
With this graph representation, one layer of message passing performs the following updates.
1. For every edge j. the edge feature vector updated using function of of adjacent nodes and vy of the edge itself Bi and of the global attribute u.
Φ (ei.j Di Ug. u)
2. For every node i, features from incident edges { (e',s } are aggregated using pooling func- tion p² "0:
-p (f
3. For every node i, the node feature vector is updated using function d of aggregated incident edges E of the node itself u, and of the global attribute u:
(e Vi, u)
4. All edges are aggregated using a pooling function pm
é' p² (( })
5. All nodes aggregated using pooling function n"
5' p'=p"~" ({ (b))))
6. The global feature vector is updated using function o² of the aggregated edges e', of the aggregated nodes e' and of the global attribute u.
o (E',
In GRAPHQA, intermediate updates are implemented as Linear- Sreea--Dopoou-ReL.U functions, and all aggregation functions use average pooling. The encoder and readout layers do not make use of message passing, effectively processing every node/edge in isolation. Message passing is instead enabled for the core layers of the network and enables GRAPHQA to process information within progressively expanding neighborhoods.
The number of neurons in the core message- passsigg layers decreases from the input to the output. Specifically follows linear interpolation between the input and output numbers reported below, rounded to the closest power of two. In preliminary experiments, increase of the number of layers results in convergence issues, which in contrast to the practice of increasing the number of channels in Convolutional Neural Networks.
noticed that progressive
14

---

Submitted as conference paper at ICLR 2020
C.2 HYPERPARAMETER OPTIMIZATION
We perform a guided grid search over the following hyper parameter space. The final model is chosen be the one with the highest Rurget on the validation set. The following considerations made:
The values for dmax are chosen on the base that the typical bond length is 5Å and residue- residue interactions are negligible after 10A.
The values for are chosen so that the RBF encoding of the edge length is approximately linear around 5Å.
The values for I. are chosen approximately match the average length of the shortest paths in the protein graphs at different cutoffs.
In addition to what described in section 2.3] we also tested an architecture with BatchNorm layers between the Dropout and ReLU operations, but apart from significant slowdown we did not notice any improvement.
Table 4: Hyper parameter space and best values
Hyper parameter
Values
Best
MP Layers I.
4, 5, 6, 8,
MP input size
32, 64, 128
128
MP input size v
64, 128, 256, 512
512
MP input size
64, 128, 256, 512
512
MP output size e
8, 12, 16, 32
16
MP output size
8, 16, 32, 64
64
MP output size u
8, 12, 16, 32
32
Cutoff dmax
6, 8, 10, 12
Sigma
10, 25
15
Dropout rate
0. 0.2. 0.3, 0.4
0.2
Learning rate
10-2,10-3
10-3
Weight decay
10-4,10-5
10 10-5
Local weight A
5,
Global weight A
1, 10
15

---

Submitted as conference paper at ICLR 2020
D ADDITIONAL STUDIES
To complement the analysis reported in the main text, we perform additional studies on the effect of feature representation and on the generalization ability of the trained model.
D.1 DISTANCE AND SEPARATION ENCODING
The feature vectors associated to the edge the graph represent two types of distances between residues, namely spatial distance and separation in the sequence. In this study evaluate the effect of different representations on validation performances.
Spatial distance is the physical distance between ammino acids, measured the euclidean distance between their 3 carbon atoms. We consider three possible encodings for this distance:
Absent spatial distance is not provided as input;
Scalar: spatial distance is provided as a raw scalar value (in Angstrom);
RBF: spatial distance is encoded using 32 RBF kernels, with unit variance, equally spaced between and 20.
Figure I0 reports the aggregated performances on CASP 11 of runs for each of the above. The rich representation of the RBF kernels seem to improve both LDDT and GDT TS scoring performances, even though the effect is rather limited.
Russs
Figure 10: Spatial distance: absent, encoded as a scalar, encoded using RBF kernels.
Separation is the number of residues between to amino acids in the sequence, we consider three possible encodings:
Absent: sequential separation is not provided as input;
Scalar: sequential separation is provided a raw scalar value (positive integer);
Categorical: sequential separation is encoded as one-hot categorical variable, according to the classes 00 2, 3, 4, 10, > 10 }. which are based on typical interaction patterns within peptidic chain.
Figure 11 reports the aggregated performances on CASP 11 of ten runs for each of the above. For local scoring, the choice of encoding plays little difference as long separation is present in the input. On the other hand, the choice of categorical encoding over scalar encoding results in higher global scoring performance.
Rewe (IDDT)
Ru (GDT-755)
Oreserc
0.6
Figure 11: Sequential separation: absent, encoded scalar, encoded categorical variable.
16

---

Submitted as conference paper at ICLR 2020
D.2 TRANSMEMBRANE VS. SOLUBLE PROTHINS
In this study, we evaluate how the natural environment of a protein affects the predictive perfor- mances of our method. Targets from CASP 11 and 12 classified transmembrane and soluble according to Peters et al. (2013) and scored separately using GRAPHQA. Transmembrane proteins behave differently from soluble proteins The former expose non-polar residues to the cellular membrane that surrounds their structure. On the contrary, the latter tend to present polar amino acids to the surrounding water-bases baseed solvent.
consequence the environment they are placed in.
Since this information is not explicitly provided to the model, we can compare the predictive perfor- mances between the sets and check that has actually learned flexible protein representation. The outcome of this evaluation is shown in table 5 While is evident that GRAPHQA performs better on soluble proteins, which are more numerous in the training set, also scores transmembrane proteins to acceptable degree.
Table Performances of GRAPHQA on transmembrane and soluble targets from CASP and 12.
GDT.TS _S
LDDT
FRI.
R
Runger
RMSE
0
Piagget
Target
R
Rmndel
RMSE
Transmembrane
0.092
0.714
0.538
0,209
0.714
0,463
0,523
0.328
0.679
0.487
0.170
Soluble
0.076
0.790
0.627
0.177
0.796
0.500
0.594
0.361
0.770
0.532
0.155
17

---

Submitted as conference paper at ICLR 2020
E ADDITIONAL METRICS
The Quality Assessment literature is rich of metrics to measure the performances of scoring method. In the main text we tried to keep the exposition uncluttered by only reporting figures for the most important metries. Here present more extensive set of metrics, that further describe our method and can serve as future benchmark.
In the following, use:
......
Target proteins
1, ...... D
Decoys of a target
i, 1, ,......|S |54|
Residue indexes target
g. GDT.TS( (Cttdd C.t.mative)
Global quality score (true)
qi f, LDDT(C (t..alive
Local quality scores (true)
GRAPUQA $ (pt.a)
Global quality score (predicted)
Local quality scores (predicted)
(PP.))
Root Mean Squared Error (RMSE) We compute RMSE between all true and predicted scores, for both LDDT and GDT TS.
For LDDT, it it the square root of:
18t
MSE 1
(RAPHQA((D(.d))
qi
ST
For GDT TS, it it the square root of:
D'
1 MSE
Σ 1=1
gordd GRAPHQA
T D
d=1
Correlation coefficients We compute the Pearson (R). Spearman (p) and Kendall (7) correlation coefficients between all true and predicted scores. Since all scores are treated equally, with no dis- tinction between different decoys or different targets, a high value for these scores can misleading. Thus, their per-model and per-target versions should be also checked.
For LDDT:
PEARSON (Eq.t.d) ), (RAPHQA((DL.))) ( 1)
P SPEARMAN (Eq. L.dd), }, GRAPHQA((PA-))
T KENDALL (() (Atd), (RAPHQA(((LL)))
For GDT.TS:
R PEARSON ((gaitd) (RAPHQA
P = SPEARMAN (1 (Lqott) }, GRAPHIQA(PP..) })
T KENDALL (LgO }, GRAPPIQA (PPt) (ptid)
Correlation coefficients per-model For every decoy of every target, we compute the Pearson (Rmodel). Spearman (Pmode)) and Kendall (Tmodel) correlation coefficients between true and predicted residue-level scores (LDDT). We then report the average correlation coefficients across all decoys of all targets. The per-model correlation coefficients estimate the performance of the network to rank individual residues by their quality and distinguish correctly vs. incorrectly folded segments.
18

---

Submitted as conference paper at ICLR 2020
Per-model correlation coefficients are computed only for LDDT:
T
D'
PEARSON (Ld. (id J, [PA.PIQA
Renodel
T
D
1=1
T
D
SPEARMAN L. [GRAPHQA((PP..))
Pinodel
T
D
1=1
d=1
T
1
KENDALL (fg.t.)) (GRAPHQA((P!)))
Tmodel
Correlation coefficients per-target For every target, we compute the Pearson (Rurget), Spearman (Prarget) and Kendall (Target) correlation coefficients between true and predicted decoy-level scores (GDT_TS). We then report the average correlation coefficients across all targets. With reference to the funnel plots, this would be the correlation between the markers in every plot, averaged across all plots. The per-target correlation coefficients estimate the performance of the network to rank the decoys of target by their quality and select the ones with highest global quality.
Per-target correlation coefficients are computed only for GDT TS:
Ruarget
PEARSON ) (PPGAAPHQAP(Ph.)) })
SPEARMAN (L (RAPHQA'(PA.))) 1)
Purrget
T
1
KENDALL (GAAAAA
Turget
T
First Rank Loss (FRL) For every target, we compute the difference in GDT TS between the best decoy according to ground-truth scores and best decoy according the predicted scores. We then report the average FRL across all targets. This represents the loss (true) quality we would suffer if we were to choose decoy according to rankings and can is represented in the funnel plots by the gap between the two vertical lines indicating the true best (green) and predicted best (red).
FRL measures the ability to select single best decoy for given target. In our experiments, however, we noticed that FRL is extremely subject to noise, as only considers top-1 decoys. Therefore, we consider NDCG to be superior metric for this purpose, though have not seen it used in the QA literature.
FRL is only computed for GDT TS:
1 FRL T
Σ max {
(q g.
where d+ mag, GRAPHIQA((P..) } for every target t.
Recall at (REC@K) We can describe Quality Assessment as an information retrieval task, where every target represents query and its decoys are the documents available for retrieval. If we con- sider the best decoy to have a score of and all others to have zero score, we can compute the average R as the percentage of queries for which the best decoy is retrieved among the top-k results.
This metric, however, is subject to the pitfalls of FRL, since it only considers the best decoy of every target and ignores the relevance of the others. As described below, NDCG offers a better perspective over the decoys retieved by QA method.
Normalized Discounted Cumulative Gain at (NDCG@N) For a given query we consider the top-k decoys ranked according to their predicted global scores. Discounted Cumulative Gain
19

---

Submitted as conference paper at ICLR 2020
(DCG@K) is computed as the cumulative of their ground- -tuth GDT-TS scores (gain), dis- counted however according to the position in the list. A high DCG@k is obtained therefore by selecting the k-best decoys to be part of the top-k predictions, and b) sorting them in order of decreasing quality (the higher in the list, the lower the discount).
Dividing DCG@k by DCG (obtained by ranking according to the ground scores), yields the Normalized Discounted Cumulative Gain NDCG@k [0, 1]. which can be compared and aver- aged across targets.
F ADDITIONAL RESULTS
In this section we present additional results for datasets, dataset splits, methods, and metrics that are excluded from the main text for sake of brevity. The datasets considered in the following pages are: CASP 11, CASP 12, CASP 13, CAMEO.
The CASP 11 and 12 datasets conventionally divided into: stage 1, containing 20 randomly- selected decoys per target, and stage 2, containing the top-150 decoys of each target. In the QA literature, some papers report results either on the dataset as a whole, splits. Furthermore, some papers report performances of other methods that differ from the original papers for reasons that are left unspecified. In the main text, we adhere the following rules to summarize the metrics we collected:
the stage and stage
Metrics computed on stage are considered noisy and ignored, since stage splits contain
only 20 randomly-selected decoys per target
Metrics computed on stage 2 and on the whole dataset are considered equally valid, allow- ing to "merge" results from papers with different scoring strategics
If multiple values are reported from multiple sources for the same (method, dataset) pair, only the best one is reported
20

---

Submitted as conference paper at ICLR 2020
F.1 CASP 11
GDTTS
LDDT
Mathed
Rirget
Paupet
Pagget
Runde
Pruq3D
ProQu
0.84
0.61
0.125
CASP 11
PrnQ4
GRAPHOAR.x
Ours
Progs
0.082
0.836
0.609
0,146
0.837
0.49
0.637
0.354
0.77
0.799
0.56
0,516
0.147
GRAPHQA
0.74
0.461
0.122
Prag3D
Omatto
3D CNN
0.046
0.066
0.795
0.691
0.755
0.782
0.673
0.606
0.58
0.462
0.529
VoruMQA
CNN
0.085
0.682
04822
0.361
3D CNN
0.122
0.512
0,402
0.305
Omatt
0.128
0.08
0:467
0.003
0.371
-0.016
0.274
CNN
0.457
Omase
Omate
0.077
0.635
0,465
0.634
0.372
0.44
0.275
GRAMRQAAAo
Ours
0.09
0.829
0.63
0.155
0.81
0.514
0,409
0.393
0.79
0.475
0.131
GRAPHQA
0,515
0.108
PruQID
Omate
CNN
0.053
0006
0,772
0.464
0,452
0.796
004335
00432
0.594
0.307
0.304
VooMQA
CNN
0.63
0.291
RWplus
CNN
0.089
0.206
0.248
0.176
stage
Oreate
0.888
0.056
0.167
0.033
0.199
0.111
0.157
CNN
0,288
Omuaz
Oreate
0.555
0.637
0.386
0.673
0:377
0.475
0.259
GRAzuQAg.ov
Ours
0.071
0.82
0,379
0.149
0.82
0.355
0.618
0.251
0.787
0.529
0.142
GRAMIQA
Curr
0.123
0.126
GDT-TS
LDDT
1
R: 0.910
R
0.852
Rtarget: 0.740
Rodel 0.602
Predicted
Predicted
0
True GDT-TS
0
True GDT-TS
1
50%
0.87
40%
Average Racalia
Average nDCG
20%
Figure 12: CASP 11: Histograms of true predicted LDDT and GDT TS scores, average recall @ k, average NDCG
21

---

Submitted as conference paper at ICLR 2020
V163
19206
18295
16720
00313
70333
72773
792744
13220
12227
70003
70264
7000
T3226
71230
T1200
7036
70.94
10200
+8.
1004
1904
True
Figure 13: CASP 11: funnels (continues)
22

---

Submitted as conference paper at ICLR 2020
12
1811
15472
T1000
7011
1M0
168
1284
7202
s65, decip Predicted Tine
Trua
Figure 14: CASP 11: funnels (continued)
23

---

Submitted as conference paper at ICLR 2020
F.2
CASP 12
GDTLTS
TestSet
Method
Source
FRL
Razes
RMSE
P
Purmet
Tonns
Randd
RMSE
Posdd
CNN
0.164
0.009
0,602
0.451
ProQ4
0.772
0.516
0.776
0.408
VarMQQ
CASP 12
KWplus
3D CNN
3D CNN
3D CNN
0.146
0.192
0.407
0.313
0.521
0.355
0.257
0.381
AngularQA
AngularQA
0138
0.651
0092
0.673
0.796
0.501
GRAPIQQ
Ours
0889
0.845
0.745
0.137
0.834
0.66
0.664
0.503
0.848
0.565
0.124
PruQUD
Ornale
0.08
0.671
0.705
0448
0.836
0.335
0,482
VoroMQA
RWplus
Ornate
0.132
-.272
0.479
0,538
0.465
0,381
0.364
stage
Omaie
Orsate
0.113
0,551
0.566
0.481
0,504
0.339
0.374
AngularUA
AngularQA
0.148
0,502
GRAPHQAALo
GRAMIQQ
Curs
0.043
0.814
0.799
0.085
0:755
0.684
(1.599
0.541
0.718
0.474
0.105
ProgtD
Oreate
0.06
0.806
0.6
0.5
0.54
0.601
0.388
VarMQ4
RWplus
Ornate
0.103
-0.066
0.417
-0.096
0.378
-0.067
0.265
stage
Omoi
Oristi:
0,072
0.67
0.491
0.657
0.458
0.472
0.322
AngularQA
AngularQA
0.128
0.377
GRAPHIQA
0.614
GRAFIIQA
0.707
0.679
0.456
0.842
0.573
0.125
GDT-TS
LDDT
1
R: 0.843
R
0.843
Rtarget: 0.745
model 0.565
Predicted
Predicted
0
True GDT-TS
0
True GDT-TS
1
50%
0.83
Avatrage Racal K
nDCG
30%
0.80
10%
Figure 15: CASP 12: Histograms of true predicted LDDT and GDT.T TS scores, average recall @ k, average NDCG
24

---

Submitted as conference paper at ICLR 2020
1845
10677
T102
T0000
J064 18AA.
1691
565, decop Predicted Tine
Trxz
Figure 16: CASP 12: funnels
At close analysis, it appears that the model completely fails to decoys of target T060 and defaults to predicting a small constant value. To help pinpointing the problem, we compare the predictions made by GRAPHQA and GRApHQARA (available in the repository). It turns that the model trained on amino acid identity only does not output the same degenerate predictions as its fully-featured counterpart (the predictions are not perfect, but definitely better than a constant), We suspect that an error in the preprocessing pipeline might have produced misleading features for T060, e.g. the multiple sequence alignment program that extracts self information and partial entropy. or the DSSP program that computes secondary structure features.
25

---

Submitted as conference paper at ICLR 2020
F.3 CASP 13
CASP 13 is the most recent edition of CASP and, at the moment, only few targets are available for public evaluation, i.e. their decoy structures fully characterized, submitted predictions and ground-truth GDT-TS scores can be downloaded from the website. Here we present an evaluation on publicly available targets, while waiting to update these results as more data is released.
Also, it is important to note that GRAPHQA is only trained on CASP 7-10, while other participants have likely (re)trained their models on all previous CASP datasets as well as other datasets. How- ever, even without retraining, we achieve performances that are in line with the results presented for CASP 11 and 12.
GDT,TS
TesiSet
Meihod
Source
FRL
Rurpet
RMSE
R
Raadd
RMSE
Prog3D
Durs
0.160
0.349
0.671
0.129
0.619
0,025
0,458
VarMMQA
Ours
0.024
0.767
0.665
0.177
0.665
0.666
0.571
0.443
CASP 13
3D CNN
Ours
0.135
0.061
0.753
0.192
0.622
0.662
0.457
0.457
0.535
0.642
0.360
GRAMKQARaw
0.767
0.727
0.701
0.475
GRAPHQA
Ours
0.093
0.846
0.834
0.123
0.340
0.799
0.647
0.616
0.764
0.539
0.142
GDT-TS
LDDT
1
R: 0.846
R
0.764
Rtarget: 0.834
model 0.539
Predicted
Predicted
True
0
True
1
GDT-TS
GOT-TS
10%
0.835
Average Recall k
Average k
0.830
0.825
Figure 17: CASP 13: Histograms of vs. predicted LDDT and GDT.TS scores, average recall @ k. average NDCG k
26

---

Submitted as conference paper at ICLR 2020
TEK4
T000s
1000
T000
70000L
Predicted Irue
Fredtted
Figure 18: CASP 13: funnels
27

---

Submitted as conference paper at ICLR 2020
F.4 CAMEO
LDDT
TestSet
Meethod
Source
FRL
R
Riyze
RMSE
2
Pieast
Tueast
R
Rasdd
RMSE
Prod]aD
Pro14
0.79
0.64
0.137
Pru
VoroMQA
CAMBO
plas
3NC
0.162
0.122
0,085
0.068
SD CNN
3NN
0.06
0.586
0.533
0.426
GRAPEQQ
GDT-TS
LDDT
1
1
R: 0.747
R
0.748
Rtarget:
0.598
model 0.616
Predicted
Predicted
True
0
True
1
GDT-T5
GOT-TS
100%
0.945
Average Recill g
0.940
0.935
0.930
0.925
Figure 19: CAMEO: Histograms of vs. predicted LDDT and GDT scores, average recall @
k, average NDCG @ k
28

---

Submitted as conference paper at ICLR 2020
2811
2036
2500
122
24077
Age
25.0
280
21322
2017.
2900
2005.0
2600C6
JAPLA
100YA
X00LP
JNGAL
262082
Semt dacty
Prokted Trii
M.K
Figure 20: CAMEO: funnels (continues)
29

---

Submitted as conference paper at ICLR 2020
M
A0
20071AA
2005
2009
2000A
:
2070
12200
103 0
464
6330
4011
4509.8
400CC
40NE.S
4LAA
432A
44TCA
40046
4.0.
Sem Prodittdd Thue
24 4
Figure 21: CAMEO: funnels (continues)
30

---

Submitted as conference paper at ICLR 2020
Fat
16.4
4.5
4,5.0
498
4.100
24
M116
0008
48400
.9
47
1.4
4775
4842.12
2.7
01.24
1-" 7
14
.06
2.8
Mm.A
1092.8
504LD
4282.
4080.8
Semt Prodittdd Thue
..6
Figure 22: CAMEO: funnels (continues)
31

---

Submitted as conference paper at ICLR 2020
M
103
1000
400
4054.0
467
11 7
4000
4095 A
100
4070
4:3
2010
4524
4077
:
RP.S
AQULB
9044.4
40000
108J
Sem Prodkted
Thus
Figure 23: CAMEO: funnels (continues)
32

---

Submitted as conference paper at ICLR 2020
405
40200
40213
4075.
4026
4036
2000
4207
4000
4,
J040
42801.2
1229
4007.4
4020
4362.5
1:a
A
S24IDD Proditted Thue
40M2
5010A
1044.
406.2
10.1
Sem
Figure 24: CAMEO: funnels (continues)
33

---

Submitted as conference paper at ICLR 2020
7p
10010
40004
1052
49914
1021
10.000
2022
4001
1919
201P
18122
<
10000
4201A
24
.6..A
ARGMA
MMALA
1827.
Sem Prodittdd Thus
Figure 25: CAMEO: funnels (continues)
34

---

Submitted as conference paper at ICLR 2020
486.
46/LL
4500
18220
1
6372
4822
413.00
4070
20.03
4024
.98
MAGL
492A
490A
4100.
Semt Prodittdd Thue
Figure 26: CAMEO: funnels (continues)
35

---

Submitted as conference paper at ICLR 2020
490.3
1A4500
04-6
18420
4865A
1817
80:3.
2017
1010
420.
1807.
4100.3
4892.
>//
AMAAA
4W/LF
4M0LA
4MALS
Sem Prodited Thue
Figure 27: CAMEO: funnels (continues)
36

---

Submitted as conference paper at ICLR 2020
:
472.2
100.0
4W/2.
48/2.4
16.50
601 m
4014
4037
E7NLA
223
-00X7
4600CC
48810
4
JMMLA
AMALA
SMKLA
4182.4
1.2
Sem Prodittdd Thue
Figure 28: CAMEO: funnels (continues)
37

---

Submitted as conference paper at ICLR 2020
40226
4001 0
1,22.
5122
622m
80000
4004
4028
4024
W
4022
48G2.6
g/.
1350.1
MBBA
.BOW.
10.
4401.
Semt Proditedd Thue
Figure 29: CAMEO: funnels (continues)
38

---

Submitted as conference paper at ICLR 2020
14
401
400
1930
402
405.0
4525
4800.6
000.0
184
A090.C
45ALA
415L8
498
400AA
Ser Proditted Thue
Figure 30: CAMEO: funnels (continues)
39

---

Submitted as conference paper at ICLR 2020
.6b
4023
408.5 5
653.2
190
2097-
0120
1180
400
4076
015
29
201111
6.74
-07.8
407-3
4190
wid
.5
3.4
106.
K00A
412
4182A
Semt Prodkted Thus
Figure 31: CAMEO: funnels (continues)
40

---

Submitted as conference paper at ICLR 2020
4712A
4706
e07
100
0005
010
ga
2003
4709-
4TAM
402.
105.8
470L6
42028
428
428AA
42449
Sem Prodkted Thue
Figure 32: CAMEO: funnels (continues)
41

---

Submitted as conference paper at ICLR 2020
401.3
1036/
6d5
201.0
670
405.
201-
4705
0000
6226.
1000
54:78
S4FT.C
$
41
SMH0
$88.5
300
5UAA
$88.5
Sem Prodkted Thue
Figure 33: CAMEO: funnels (continues)
42

---

Submitted as a conference paper at ICLR 2020
Prpdlteer
Trok
Figure 34: CAMEO: funnels (continued)
43


Reviews:
[{"review_id": "rJeIpEYCKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Proteins are sequences of Amino Acids. Identifying the 1-D sequence of a protein is straightforward. Each protein folds to a 3D structure. Determining the 3D structure of a protein (the protein folding problem) is expensive and hard.  It is well known that the function of a protein is determined by its 3D structure. Several computational methods have been proposed for protein structure prediction, but none of these models perform well in all circumstances. Different models perform well for different kinds of proteins. The current work deals with evaluating the different models to determine which model is likely to perform better on which protein. \n\nThe authors use Graph Convolutional networks with messaging to solve this problem of evaluating protein quality. Their method is evaluated using the Global Distance Test Total Score and Local Distance Difference Test (which is done at a residue level). They use several node and edge level features like DSSP, Partial Entropy and Self Intro. \n\nPros: \n\nTheir methods perform better than comparable methods using 1D and 3D CNNs. An ablation study is conducted to show the importance of various features. The paper is well written and the source code is provided for reproducibility. Overall, this is a good application paper showing the application of a known technique to solve a problem in a new domain. \n\nCons: \n\nThe novelty is minimal and the problem is of interest only to specialists in this domain. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SyeIDP2AKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes use of graph convolutional networks (GCN) for quality assessments (QA) of protein structure predictions. In particular, protein structure prediction is a very active area of research witnessing steady progress during the previous decade or so. To estimate the quality of the prediction, many experimentally resolved structures are needed. However, experimental structure determination is expensive and many protein families are notoriously hard to experiment with. Thus, current estimates of the quality of the protein structure prediction models are incomplete and biased. As a result, there is an interest in guessing quality of protein structure predictions on protein families that are not characterized experimentally. Previous papers already proposed several neural network architectures for the QA task. This paper shows that GCN applied on a graph of a predicted protein structure achieves higher accuracy than the previously proposed neural networks\n\nStrengths:\n+ The proposed GCN outperforms other neural network baselines.\n+ The protein representation is reasonable.\n+ The paper is reasonably well written with a nice overview of the related work.\n+ Ablation studies are well done, in clean graphs. This can be useful for other authors who work with GCNs.\n\nWeaknesses:\n- Methodological novelty is low -- this is a straightforward application of GCN \n- The objective of QA is a bit suspect for the sole reason that the training and testing is performed using experimentally resolved protein structures. This data set is biased and there are no guarantees that the reported accuracy will hold over a vast range of protein families that are not structurally characterized.   \n- Separation encoding is done as a one-hot vector. This could probably be passed as a scalar value. Would be nice to have comparison between 1-hot vs scalar in the experimental results\n- Formulas in section 2.3 are cryptic for audience unfamiliar with GCN and it is not specific to this application.\n- Figure 3b) shows that there is a cluster of predicting 0 where the ground truth is bigger than 0.6.\n\nOverall, this is a borderline paper. There is little methodological novelty and the QA application is a bit of a niche problem in bioinformatics. However, the results show a decent improvement over the state of the art in this particular application, so this paper might be of importance for a limited audience interested in this problem. Giving this work a benefit of the doubt the entered rating is a weak accept.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJli5s0ycr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript describes a new deep learning method for the prediction of the quality of a protein 3D model in the absence of the experimental 3D structure of the protein under study. The major idea is to model a protein 3D model using a graph. That is, each residue in a protein is modeled as a node and one edge is added to connect two residues if they are spatially close to each other.  Based upon this graph representation, the manuscript describes a graph convolutional neural network (GCN) to predict both local (i.e., per residue) and global quality. The authors showed that this GCN method works well on the CASP11 and CASP12 data. Unfortunately, there is no experimental result on CASP13 models, which significantly reduce my interest on this paper.\n\nMinor concerns:\n\nReferences are missing or misplaced at some places. For example, in the 1st sentence of the 4th graph, \"While computational protein folding has recently received attention...\", only protein design papers are cited. Some representative protein structure prediction papers shall be cited here.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an approach for estimating the quality of protein models. The proposed method consists in using graph convolutional networks (GCNs) to learn a representation of protein models and predict both a local and a global quality score. Experiments show that the proposed approach performs better than methods based on 1D and 3D CNNs.

Overall, this is a borderline paper. The improvement over state of the art for this specific application is noticeable. However, a major drawback is the lack of methodological novelty, the proposed solution being a direct application of GCNs. It does not bring new insights in representation learning. The contribution would therefore be of interest to a limited audience, in light of which I recommend to reject this paper.
Decision: Reject