Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This manuscript describes a new deep learning method for the prediction of the quality of a protein 3D model in the absence of the experimental 3D structure of the protein under study. The major idea is to model a protein 3D model using a graph. That is, each residue in a protein is modeled as a node and one edge is added to connect two residues if they are spatially close to each other.  Based upon this graph representation, the manuscript describes a graph convolutional neural network (GCN) to predict both local (i.e., per residue) and global quality. The authors showed that this GCN method works well on the CASP11 and CASP12 data. Unfortunately, there is no experimental result on CASP13 models, which significantly reduce my interest on this paper.

Minor concerns:

References are missing or misplaced at some places. For example, in the 1st sentence of the 4th graph, "While computational protein folding has recently received attention...", only protein design papers are cited. Some representative protein structure prediction papers shall be cited here.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper proposes use of graph convolutional networks (GCN) for quality assessments (QA) of protein structure predictions. In particular, protein structure prediction is a very active area of research witnessing steady progress during the previous decade or so. To estimate the quality of the prediction, many experimentally resolved structures are needed. However, experimental structure determination is expensive and many protein families are notoriously hard to experiment with. Thus, current estimates of the quality of the protein structure prediction models are incomplete and biased. As a result, there is an interest in guessing quality of protein structure predictions on protein families that are not characterized experimentally. Previous papers already proposed several neural network architectures for the QA task. This paper shows that GCN applied on a graph of a predicted protein structure achieves higher accuracy than the previously proposed neural networks

Strengths:
+ The proposed GCN outperforms other neural network baselines.
+ The protein representation is reasonable.
+ The paper is reasonably well written with a nice overview of the related work.
+ Ablation studies are well done, in clean graphs. This can be useful for other authors who work with GCNs.

Weaknesses:
- Methodological novelty is low -- this is a straightforward application of GCN 
- The objective of QA is a bit suspect for the sole reason that the training and testing is performed using experimentally resolved protein structures. This data set is biased and there are no guarantees that the reported accuracy will hold over a vast range of protein families that are not structurally characterized.   
- Separation encoding is done as a one-hot vector. This could probably be passed as a scalar value. Would be nice to have comparison between 1-hot vs scalar in the experimental results
- Formulas in section 2.3 are cryptic for audience unfamiliar with GCN and it is not specific to this application.
- Figure 3b) shows that there is a cluster of predicting 0 where the ground truth is bigger than 0.6.

Overall, this is a borderline paper. There is little methodological novelty and the QA application is a bit of a niche problem in bioinformatics. However, the results show a decent improvement over the state of the art in this particular application, so this paper might be of importance for a limited audience interested in this problem. Giving this work a benefit of the doubt the entered rating is a weak accept.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Proteins are sequences of Amino Acids. Identifying the 1-D sequence of a protein is straightforward. Each protein folds to a 3D structure. Determining the 3D structure of a protein (the protein folding problem) is expensive and hard.  It is well known that the function of a protein is determined by its 3D structure. Several computational methods have been proposed for protein structure prediction, but none of these models perform well in all circumstances. Different models perform well for different kinds of proteins. The current work deals with evaluating the different models to determine which model is likely to perform better on which protein. 

The authors use Graph Convolutional networks with messaging to solve this problem of evaluating protein quality. Their method is evaluated using the Global Distance Test Total Score and Local Distance Difference Test (which is done at a residue level). They use several node and edge level features like DSSP, Partial Entropy and Self Intro. 

Pros: 

Their methods perform better than comparable methods using 1D and 3D CNNs. An ablation study is conducted to show the importance of various features. The paper is well written and the source code is provided for reproducibility. Overall, this is a good application paper showing the application of a known technique to solve a problem in a new domain. 

Cons: 

The novelty is minimal and the problem is of interest only to specialists in this domain. 
review_title: Official Blind Review #1

Meta-review: This paper introduces an approach for estimating the quality of protein models. The proposed method consists in using graph convolutional networks (GCNs) to learn a representation of protein models and predict both a local and a global quality score. Experiments show that the proposed approach performs better than methods based on 1D and 3D CNNs.

Overall, this is a borderline paper. The improvement over state of the art for this specific application is noticeable. However, a major drawback is the lack of methodological novelty, the proposed solution being a direct application of GCNs. It does not bring new insights in representation learning. The contribution would therefore be of interest to a limited audience, in light of which I recommend to reject this paper.
Decision: Reject