Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the task of predicting the effects of mutations in protein sequences. To that end it leverages a graph neural network with : 1) Biochemical (eg., AA type, SAS, B-factor) and geometric (eg., 3D coordinates of alpha-carbons, dihedral angles) features of amino acids (nodes in the graphs) and edge features (eg., interatomic distances, local N-C positions and position encoding) 2) A stack of Graph Equivariant Convolution (EGC) layers to learn node and edge embeddings 3) A multi-task pre-training framework that combines AA type classification, SAS and B-factor prediction and 3D-coordinates denoising objectives (positions and angles). The architecture is trained on CATH 4.3.0 and mutation effect prediction is assessed in the zero-shot setting against several baselines.
strength_and_weaknesses: **Strengths**
- The GCN layers help bypass data augmentations / be more data efficient
- The multi-task pre-training is a very sensible idea to learn protein embeddings that combine the different modalities characterizing protein sequences (ie., primary, secondary, tertiary structure) and in turn improve downstream task performance (eg., mutation effects prediction)

**Weaknesses**
- As evidenced by the results in section 3.4 and figure 5, there seems to be severe task interference when pre-training simultaneously on all tasks. Including the AA recovery and SAS prediction tasks captures the bulk of the performance lift, while other tasks are either not adding substantial value beyond that, or seem to actually be destructive (eg., dihedral prediction). It is thus not clear whether one should keep all these objectives when pre-training? What is the final training objective / set of pre-training tasks that is recommended to use by the authors?
- The current empirical evaluation lacks several important baselines with respect to mutation effects prediction (eg., DeepSequence, EVE, Tranception) without which claims about “SOTA performance” in abstract/conclusion seem unsubstantiated.
- The method introduced in this work does seem to achieve relatively high fitness prediction performance Vs inference time tradeoff. To drive this point home though would require comparing with Potts models which also achieve a very favorable tradeoff as well.

clarity,_quality,_novelty_and_reproducibility: **Clarity**
- “More importantly, when predicting the fitness of the higher-order mutants, most of these models made a crude assumption that the multiple-site mutation effect is a linear summation of the effect of each individual mutation, which is incorrect in most cases” (Section 1) → this is a bit misleading in the context of the prior sentences in that section. Auto-regressive models provide exact likelihood estimation and therefore do not need to resort to these simplifying assumptions with respect to multiple mutants. This is not the case however for architectures relying on masked-language-modeling objectives (eg., ESM-1b / ESM-1v) which do make the individual mutations summation assumption.
- Furthermore, the language in the last paragraph of section 1 would lead one to believe that the method introduced in this paper is not subject to ignoring epistatic effects / do not sum the effects of single mutants independently. However equation 5 in supplementary B.5 seems to indicate the opposite. Could you please clarify?
- The MSA Transformer is not a “zero-shot” but rather a “few shot” method since it does require MSAs at inference to make predictions -- please adjust the language accordingly in section 3.2 & Figure 3
- Figure 3 is a bit difficult to read. Would suggest to separate ablations from comparison with baselines (it is also very hard to tell apart the different ablation points with the current color scheme / symbols). It would also be helpful to add a small table that summarizes average performance comparing your method Vs baselines
- Could you please clarify why a very fast inference time is important in practice? Since the time and cost of in silico experiments is much lower than actual wet labs experiments, should that be a primary concern?

**Quality**
- Benchmark: the set of 15 DMS assays selected in experiments is a subset of the broader ProteinGym benchmark [1] -- why are you not reporting the performance on the full set of assays?
- Baselines: as discussed in the weaknesses section above, to substantiate the claims regarding SOTA made in abstract and conclusion, you should include additional baselines (eg., DeepSequence, EVE, Tranception) and Potts models for the speed of inference claim
- Regarding ESM-1v, are you using a single network or the ensemble of 5 networks as per [2]? The performance of a single non fine-tuned ESM-1v model is relatively low (see Tables 1 and 2 of [2])

**Novelty**
- The multi-task pre-training framework that covers different protein modalities is novel

**Reproducibility**
- Could you please confirm whether the code and data used would be released upon acceptance?

----------------------------------------------------------------------------------------------------------------------------------------

[1] Notin, P., Dias, M., Frazer, J., Marchena-Hurtado, J., Gomez, A.N., Marks, D.S., & Gal, Y. (2022). Tranception: Protein Fitness Prediction with Autoregressive Transformers and Inference-time Retrieval. ICML.

[2] Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., & Rives, A. (2021). Language models enable zero-shot prediction of the effects of mutations on protein function. NeurIPS.

summary_of_the_review: The modeling approach introduced in this paper is very promising. There are a couple residual issues -- in particular regarding task interference during the multi-task pre-training and evaluation / baselines chosen in experiments -- that would greatly benefit the work if discussed further / corrected. Willing to increase my score if these are addressed during rebuttal.

-------------------------------------------------------------------------------------------------------------------------------------
[Update post rebuttal]
Based on my discussion with the authors and the additional results provided during rebuttal, I believe that the paper in its current form is still not ready for publication and hence maintain my original score (ie., below the acceptance threshold). I provided clear and actionable recommendations to improve the paper in my last response to the authors.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a ‘lightweight graph neural network’ (LGN) for fast zero-shot fitness prediction of mutational effect from protein structures. Noisy perturbations of node and edge features computed from the protein structure are fed into Equivariant Graph Convolution (EGC) layers to extract hidden representations. These representations are used for multitask learning of amino acid type, microenvironment properties (solvent accessible surface area and crystallographic B-factor) and denoising of residue coordinates. Using the learned representations, the authors demonstrate highly efficient SOTA performance in mutation effect prediction.
strength_and_weaknesses: Strengths:

The general aim of leveraging structure information to improve fitness prediction performance in the context of protein design is important. 

Weaknesses: 

I am not sure if the overall premise of treating mutants as perturbations to node signals and using a denoising autoencoding approach works for their goals. The premise of denoising, is that one brings the perturbed data back to the original data manifold, so likely perturbations will simply be denoised to the training distribution (consisting primarily of wild type proteins). This will not allow for novel protein generation. Indeed there is no quantification in the manuscript for the ability to generate fitter more novel mutants.  The authors state “Altering AA types of a protein in nature can be viewed as adding corruptions to the node features of the protein graph, and denoising the graph makes a remedy to search for mutants with the best fitness”  The connection between denoising and fitness is assumed but never fleshed out. 


Additional comments for improvement are below:

1. Sato & Ishida (2019) are cited twice in the introduction. I believe the first citation in the following sentence is incorrect: “Sato & Ishida (2019) applied 3DCNN to identify a new polymerase with advantageous single-site mutation and enhanced the speed of degrading PET by 7-8 times at 50”. The correct citation should be Lu et al., Nature, vol. 604, pgs 662–667, 2022. Note that the temperature of 50 degrees is measured in Celsius, and some background information (i.e. PEG is a waste product that can recycled by degradation) would be helpful to contextualize this work. The second citation of this paper, regarding autoregressive inference for scoring single-site mutations, is also irrelevant. Please re-check all citations, taking care to only include works that are relevant to this paper.

2. What does “capture the micro-frame property in the protein graph geometry” mean? Does it refer to local geometric properties (e.g., alpha helix, beta sheets) of a contiguous subset of residues in the protein? The term ‘micro-frame’ is never clearly defined.

3. The usage of terms ‘invariance’ and ‘equivariance’ is inconsistent throughout.

4. The 93-dimensional edge attributes include a 66-dimensional encoding of the ‘relative position in the protein sequence’. This comprises of a 65-dimensional 1-hot encoding of sequence distance between pairs of nodes, and a scalar contact signal describing ‘if the two residues are in contact in space’. Here, the threshold value of 65 (possible typo in Appendix A.3 states 64) is chosen according to the sequence distance distribution. Is this threshold fixed for all proteins in your dataset?

5. In the introduction, the authors state that their model predicts mutation scores without assuming independence of individual mutations. Specifically, the authors point to the shortcomings of summing up log-odd ratios. However, in Appendix B.2, Eqn 5, the fitness score for multi-site mutations is computed by summing log-odd ratios for single- site mutations. Can you please clarify?

6. “The p-value for both coefficients is 0.000.” This is likely due to rounding or truncation. Perhaps say that p-values for both coefficients is < 0.001?

7. I cannot find any discussion on the limitations of this work. The proposed model is ‘lightweight’ and efficient to train because it relies on the availability of prior knowledge, i.e., tertiary protein structures. This is an important distinction as the models argued against in the introduction are large due to their attempt to model evolutionary-scale information about protein sequence composition in the absence of structure information.
 						
8. Can you comment on the application of this model towards predicting protein stability changes (i.e., changes in the Gibbs free energy of unfolding) between wild type and mutant proteins?

9. Consider citing related work on fitness prediction using regularized latent space optimization (Castro et al., Nature Machine Intelligence, vol. 4, pgs 840–851, 2022), and structure-informed sequence embedding (Wang et al., Scientific Reports, vol. 12, 6832, 2022). 

10. “The latter protein language models derived from natural language processing (NLP) encode sequence semantics and often need hundreds of GPU cards to train on billions of protein sequences” This claim needs a citation as most protein datasets are upper bounded at 100s of Millions. ESM-1 (Rives et al 2021) was trained on 250M sequences and AlphaFold 2 (Jumper et al 2021) trained on 350M.

11. Figure 2 seems to be an important result however it is never mentioned in the manuscript and therefore is difficult to interpret.

12. The ablations in Figure 3 are obscured by their same coloring, especially for datasets where their y-positions overlap.

13. I would also suggest adding quantifications of the ability to perform directed protein evolution (i.e. generate newer fitter molecules). 



After reading the rebuttal: 
I raised the score from "reject, not good enough" to "marginally below the acceptance threshold":
They have made all the corrections that we and the other reviewers pointed out.
They have included additional comparisons to DeepSequence, Tranception, ProGen2 in Figure 3, Figure 6, and Table 1. In cases where a comparison could not be performed (EVE and Potts model) due to technical issues, the authors provide a reasonable explanation.
The wording around "independence of single mutations" (Appendix B.2) that was confusing for us and the other reviewers has been fixed.
I hesitate to recommend acceptance because:
VAE type models can generate high fitness protein sequences, and they do not suffer from the independence of single mutations assumption. Our main critique, i.e. "This will not allow for novel protein generation. Indeed there is no quantification in the manuscript for the ability to generate fitter more novel mutants." was not addressed in the rebuttal. Ultimately their method is good for predicting fitness and other properties efficiently, but is not practically useful for generating new sequences.
The authors admit that computing MSA for pre-training is computationally expensive. However they assume that this info is given ahead of time and they do not include it in their runtime. At the same time they argue: "we doubt the speed of inference time for Potts Model, as the efforts in preparing the MSA information should also be included."

clarity,_quality,_novelty_and_reproducibility: There is a lack of clarity in the writing.  The results and presentation could be highly improved in quality. 

The premise of the work is novel but not sure that it will yield good results in practice. Most comparisons are to work that does not use structure information. 
summary_of_the_review: The general aim of leveraging structure information to improve fitness prediction performance in the context of protein design is important. However the manuscript and its approach are not well motivated or clearly presented. Moreover, the manuscript contains errors that impact both the strength and the readability of the work presented. The captions of Figures are sparse and make it difficult to interpret results and one figure is not mentioned anywhere in the main manuscript.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors studied the problem of amino-acid mutation prediction in protein by regarding this problem as solving a denoising problem with lightweight GNN. Several large-scale pre-trained models have previously been proposed and applied to this problem, such as ESM-1v and MSA transformer. The authors proposed a new lightweight graph neural network approach for this problem. Three major contributions are claimed.
1. Improvement in generalization ability.
2. Improvement in efficiency.
3. Improvement in performance on high-order mutations.
strength_and_weaknesses: # Strength
1. The authors provided a clear formulation of the problem of interest.
2. The authors provided a clear statement of their claimed major contributions.
3. The authors provided a very detailed description in the method section, including intuition and mathematical formulation of the loss function.
# Comments
1. The authors did not offer a very strong intuition on applying zero-shot or multi-task learning to the mutation fitness prediction problem.
2. The authors claimed that the model improved the generalization ability, but no clear evidence is given to support how the generalization ability has been improved. There are also no explicit explanations for this supposedly most significant advantage of the model.
3. There is not any reference or explanation to Figure 2 in the main text, which puts the reader in a quite confusing situation of wondering about its purpose. 
4. The authors tried various LGN configurations and presented the results in Figure 3. However, there isn’t a conclusion or hypothesis on which configuration may result in better performance under certain conditions. The authors could have conducted more experiments and provided more insights into these results. 
5. Figure 3 is not very illustrative, e.g., ESM-1v is missing/hidden for F7YBW8, and the labels for the LGN models are not distinguishable.
6. The comparison in section 3.5 is somehow not clear enough. First, the authors only tested on one protein sample, which does not offer sufficient evidence to prove the claimed advantage. Second, the training time usage comparison is ambiguous since different models are trained on datasets of various sizes, as indicated in Table 1 of the paper. 
7. The authors did not mention how the model might be used for downstream tasks, particularly mutation prediction. I would personally be interested to know whether there would be a web tool or a pre-trained model for users to try.
# Minor
1. Some grammar mistakes (although without affecting the understanding of the text), such as “approach … sum up” and “results are conducted”. 
2. Since Table 2 is only included in the supplementary material, the “Table 2” reference in the main text’s section 3.5 might need be changed.

clarity,_quality,_novelty_and_reproducibility: The authors introduced a new approach to the protein design problem but did not present sufficient evidence to support the claimed advantages. The results also lack further explanation and interpretation in detail. As a result, it is unclear to judge whether the contributions are valid. In addition, no clear explanation of how the model could be used is provided.
summary_of_the_review: The authors introduced a new approach to the protein design problem but did not present sufficient evidence to support the claimed advantages. The results also lack further explanation and interpretation in detail. As a result, it is unclear to judge whether the contributions are valid. In addition, no clear explanation of how the model could be used is provided. In my opinion, the paper does not meet the standard of ICLR at this moment and may need further major revisions with a focus on the results part.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper focuses on the task of predicting the effects of mutations in protein sequences. A graph neural network is employed, and the model is trained on CATH 4.3.0 and mutation effect prediction is assessed in the zero-shot setting against several baselines.

While this paper works on an interesting topic, the reviewers have raised quite a few concerns, including the justification of the technical proposal and the appropriateness of the used metrics, the comprehensiveness of the experimental evaluation, etc. The authors have provided detailed responses to the review comments, and we have asked the reviewers to reconsider their comments based on the author rebuttal. However, the reviewers end up believing that the author rebuttal did not address their concerns, and would like to keep their original scores. In this case, we do not think it is a good idea to accept the paper.

justification_for_why_not_higher_score: The reviewers have raised quite a few concerns, including the justification of the technical proposal and the appropriateness of the used metrics, the comprehensiveness of the experimental evaluation, etc. The authors rebuttal failed in convincing the reviewers to change their mind.
Decision: Reject