Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes using protein structure and sequence information as a pretraining task for representation learning of proteins. They then apply these methods to two downstream applications: (1) zero-shot prediction and (2) OOD prediction.

strength_and_weaknesses: Strengths:
- Combining sequence and structural information and pretraining hasn't been done before which is definitely a natural progression for the progress in the protein + ML space in recent years.
- Finetuning of structural/inverse folding models hasn't been studied as much compared to sequence-based finetuning.

Weaknesses:
- The framing and experiments of the paper could be improved to highlight the core contributions. Hsu et al (2022) had demonstrated the impact of structure on zero-shot fitness prediction. The takeaway from the section on zero-shot prediction largely recapitulated the results from Hsu et al while demonstrating that adding sequence information atop structure did not seem to impact performance. While this is somewhat interesting, it seems like this should not be such a dominant section of the paper. Rather the pretraining and finetuning with structure and sequence for OOD prediction seems much more novel. However, the experiments in this section are much less explored and don't benchmark against simple baselines (CNNs etc) or other methods. Experiments focused on finetuning and pretraining strategies for structural data in combination with sequence data would be interesting but seemed less prioritized than the zero-shot section. Overall, the exposition and experiments seem to highlight the less novel application while providing less thorough experiments that would push the field forward for the OOD prediction task. Getting fine-tuning to work for these LLMs can be challenging so guidelines on how to do this for various representations(sequence-only, structure-only, combined) would be extremely valuable in my opinion.
- The point about the lack of structure data causing the need for sequence data for pretraining has been somewhat obviated given AlphaFoldDB.

Minor Comments:
- Citing ProteinMPNN (Daupras et al 2022) seems relevant here even if its not benchmarked against.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clear.

Quality: The quality is overall good but the experimeents in the OOD section could be improved.

Novelty: The novelty in the paper is not properly highlighted. The zero-shot section is not really novel while the experiments in the OOD section are a bit underbaked from a point that would help practitioners as well as methods development.

Originality: There is some originality but largely I suspect some of the core ideas of this work was scooped in recent months.
summary_of_the_review: I think the paper has good work that pushes the research field forward for protein engineering applications. However, I think the focus on zero-shot prediction is mostly a distraction from the more useful aspects of combining sequence and structure information (as opposed to just structure and just sequence). More thorough experimentation and benchmarking along with a rework of the exposition to highlight this fact would greatly improve the quality of this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: BERT-style pretraining has become popular for protein modeling, since there are very large databases of natural protein sequences available. A separate body of work on inverse folding has explored structure-conditioned generative models of sequences.This paper explores structure-conditioned BERT, where at both train and test time the structure is assumed to be known for any query sequence. The authors demonstrate that conditioning on the structure improves performance on a variety of zero-shot and few-shot protein function prediction tasks.

strength_and_weaknesses: =strengths=
Well written
Experiments compare to recent work
Experiments demonstrate that conditioning on structure can provide good performance improvements (e.g., on zero-shot modeling of the covid RBD).

=weakenesses=
The experiments don't clearly demonstrate a regime where the paper's proposed method improves over existing work. The experiments demonstrate that structure-conditioned models are good at zero-shot prediction, but existing structure-conditioned models can do this well too.

Experiments don't compare to any non-neural-network baselines

The paper is frank about the fact that there are lots of opportunities for follow-up work, such as by pretraining at scale on alphafold-predicted structures. The paper is good enough for acceptance in its current form, but it wouldn't have been a terrible idea to slow things down, run some of the proposed experiments, and submit when the full setup is available.

clarity,_quality,_novelty_and_reproducibility: Overall, the paper is well written and placed in the current research dialog. I found the structure of the body of the paper a bit confusing, though. There wasn't a clear separation between Methods and Results sections. 

The proposed modeling idea combines a few core ideas already existing in the literature. It is a natural, well-motivated idea, but not ground-breaking. 
summary_of_the_review: I work in this exact research area and am in general supportive of this kind of work at ICLR. It can have large real-world impact and these protein modeling problems are also a great place for cutting-edge methods to be applied. My primary hesitation about this paper, however, is the nature of the experimental results.

I didn't understand why there were few results for ESM in Table 2. Without these, how can you make the claim "For all tasks except DeepSequence, MIF is better than sequence-only methods?" There is a good github package for ESM. It shouldn't be too hard to run on these datasets. Can you add these?

Along these lines, to me the take-away point from table 2 is that structure-conditioned models are good at zero-shot prediction, not that the particular model proposed by this paper provides a breakthrough. GVP, for example, is quite strong. Am I missing something? Where is there convincing evidence that MIF-ST improves over existing work?

I was disappointed by the complete lack of baselines that are not based on large neural networks. What about, for example, profile HMMs or Potts models for the zero-shot experiments? What about a simple linear model or lightweight non-linear model (like a shallow MLP) for the few-shot experiments? Adding all of these to the paper would be a lot of work. Can you provide baselines for at least some of the experiments, though?

The paper's propose model combines two orthogonal ideas: (1) using a CNN instead of a transformer and (2) conditioning on the structure. (1) appears in current work under review. It's not required for acceptance, but the paper would have been considerably improved if there were experiments that isolate (2) by using a transformer.

I was really surprised by how strong the RBD results are for structured-conditioned models. Do you have any intuitions for why conditioning on the structure makes such a big difference here?
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: For the first time, this study proposes to apply masked language modeling (MLM) to the inverse folding problem of proteins. The protein language model is parameterized as a bidirectional version of the structured GNN (Ingraham et al, 2019). The authors also showed that using the outputs from a pretrained sequence-only protein MLM as input can further improve pretraining perplexity. The proposed models are evaluated on downstream protein engineering tasks, including zero-shot mutation effect prediction and out-of-domain generalization.
strength_and_weaknesses: $Strengths$:

Applying MLM to the protein inverse folding problem sounds interesting. It is valid to use graph neural networks to incorporate the structure priors from folded proteins. Besides, using the learned amino acid representations from a pretrained sequence-only protein MLM as inputs to the graph neural networks is a resonable and meaningful trial.

$Weaknesses$:

1. Performance. 

Since this paper presents a straightforward pre-training methodology that uses masked inverse folding as a pretext task, it is important that the pre-trained model shows superior performance on downstream tasks. However, in Table 2, performance of the proposed MIF-ST is lower than GVP+AF2 on three selected tasks, i.e., MSP, RBD, and stability. Moreover, when compared to GVP only, MIF-SF still produces much worse results on MSP and RBD. Also, explanations behind these failures are missing. 

Besides, when evaluated on the out-of-domain generalization tasks, CARP-640M is employed as the only baseline. However, MIF-ST already includes CARP-640M as the feature extractor for amino acids. Thus, the reported comparisons are kind of unfair because MIF-ST is expected to achieve better results than CARP-640M. The authors should at least include another recent baseline in Table 2. If there are no baselines available, the authors should clarify the reasons in the manuscript.

2. Some statements are either confusing or inappropriate. 

In the abstract, the author claimed "inverse folding methods do not take advantage of sequences that do not have known structures." After reading the paper, I failed to find how the authors addressed this point besides incorporating the predicted structures from PDB and AF2 (as shown in Table 3). If this is the truth, the above statement should be revised because GVP already includes structures from AF2 as training data. 

In the last paragraph of sec. Introduction, the sentence "Figure 1 compares our previous sequence-only dilated convolutional protein MLM (CARP), MIF, and MIF-ST" is not appropriate (from my perspective). The citation of CARP refers to the anonymous submission but the authors did not provide the anonymous draft. As a result, how to implement the CNN model in MIF-ST is missing (well, this is another problem that needs to be addressed in the rebuttal), and I have to search CARP online and find the draft on bioRxiv. I'm not sure if the use of words like "our previous xxx" violates author guidelines of ICLR'23 but personally, I recommend using other words instead.  


Minor points:

1. Does formula 4 lack a concatenation notation? Otherwise, it should be $\mathcal{E}_{i,j}$.

2. In Figure 2, it seems that $w$ stands for the angle instead of coordinates.

3. Why do the authors take the best reported value instead of mean value for each task across several tested model variants?

4. Based on Table 3, I assume the authors already tried to use the predicted structures from AlphaFold2? If this is the truth, it is necessary to explain why GVP can utilize the predicted results from AF2 in a better way.
clarity,_quality,_novelty_and_reproducibility: $Clarify\ and\ Quality$:

Some statements are either confusing or inappropriate. Some background knowledge is missing, e.g., dihedral and planar angles.

$Novelty$:

The idea of applying masked inverse folding is interesting. However, the technical novelty is limited as both Structure GNN and CARP have been proposed before. Overall, the novelty is okay to me because, in my opinion, performance matters more in pre-training methodologies. 

$Reproducibility$:

The authors provided source code but the readme file is missing. Thus, it is unclear how to run the code.
summary_of_the_review: I lean upon a weak reject due to the unsatisfactory performance compared to GVP on MSP, RBD, and stability. Moreover,
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use masked inverse folding to pre-train (1) a protein structure-based encoder (MIF) and (2) a structure-based encoder enhanced by protein language model representations (MIF-ST). (1) MIF encodes the information of protein backbone structure and masked protein sequence into node and edge features, and the model is trained to recover the masked residues based on the encoded information. (2) MIF-ST further improves the performance of sequence recovery by feeding protein language model representations into the structure-based encoder. On zero-shot and out-of-domain protein landscape prediction benchmarks, MIF and MIF-ST achieve competitive empirical performance against previous methods. 
strength_and_weaknesses: Strength:
1. The proposed methods are well motivated. MIF is proposed to improve sequence-only masked language modeling with backbone structural information, and MIF-ST is proposed to further improve MIF by pre-trained protein sequence representations. 
2. The advantages of pre-trained structure-based encoders are demonstrated on a comprehensive set of protein engineering related tasks, and the ablation study analyzes the effect of using experimental and predicted structures for downstream evaluation. 


Weakness:
1. For performance comparison, some important protein structure pre-training baselines are lacked. In a closely related work [a], several structure-based encoders pre-trained by contrastive learning and self-prediction methods are proposed, and the source codes are released. It will be nice to compare with these baseline models on protein engineering tasks. 
2. Some ablation studies are lacked. According to Table 2, ESM-1b and ESM-1v are superior protein language models against CARP-640M. Why not using their representations as inputs of MIF? In this way, we can expect stronger MIF-ST models than the current one based on CARP-640M representations. 
3. Some notations are not consistent across figures and texts. In Figure 2, authors use $\varphi$ and $\theta$, while $\phi$ and $\psi$ are used in the equations (3) and (4).


[a] Zhang, Zuobai, et al. "Protein representation learning by geometric structure pretraining." arXiv preprint arXiv:2203.06125 (2022).

clarity,_quality,_novelty_and_reproducibility: Quality:

In general, this paper is well-written and well-motivated, except for some notation issues.


Novelty:

The technical novelty of this work is plain. The structure-based encoder is proposed by a previous work, and the masked inverse folding pre-training scheme share great similarity with the "structure-based residue type prediction" objective proposed by [a].


Reproducibility:

Authors submitted source codes for reproducing the results. The codes look well-organized according to my brief check. 

summary_of_the_review: To summarize, this paper shows some merits of boosting future research on structure-based pre-training for protein engineering, while more comprehensive performance comparisons and ablation studies are required to make it a more solid contribution. I think it is currently on the border and expect authors' efforts during the response period.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes two protein masked language models leveraging protein sequence and structure information for pre-training and subsequent fine-tuning on downstream tasks. 

The reviewers and AC all agree that the paper makes some interesting contributions and the proposed methods are promising, but that further work is needed for the paper to be truly significant and impactful.

We strongly urge the authors to conduct additional experiments mentioned in the reviewer and discussion phase, which unfortunately could not be performed due to the time limitation.
- As noted by Reviewer vWP7, important ablation studies are missing and we strongly encourage the authors to perform them.
- It would also be important to compare against the additional pre-training baselines which could not be evaluated in time during the discussion phase.
justification_for_why_not_higher_score: While the method is promising, all reviewers and AC agree that important experiments and comparison are lacking.
Decision: Reject