Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a prompt-based multi-task framework for pre-training and fine-tuning protein sequence representations. From a methodological standpoint, the paper adapts the prompt fine-tuning idea (ie., a differentiable continuous prompt token is pre-trained instead of using a discrete prompt) from the NLP literature to protein modeling with large transformer networks. It then makes two contributions over the standard transformer architecture 1) A specific masking scheme for the prompt token is used to keep only the effect of the prompt on the input sequence and eliminate the opposite effect (ie., the prompt should be task-dependent and not sample-dependent) 2) Learned task-specific layer-specific skip connection linear projections to let the network learn different weights for each task at each layer. The multi-task pre-training involves three kinds of pre-training: a) masked-language modeling with sequence-based information only (MLM) b) Prediction of the alpha-carbons positions (CRD) c) prediction of protein-protein interactions (PPI). Experiments on function annotation and protein engineering (FLIP benchmark) demonstrate the benefits of the different ideas introduced in this work.
strength_and_weaknesses: **Strengths**
- The idea of combining different pre-training tasks seems very sensible for proteins given the diversity of modalities that characterize them (eg., via their primary, secondary, tertiary and quaternary structure). As noted by the authors, there is however a high risk of task interference when one wants to obtain pre-trained representations that combine these different modalities / tasks. The approach suggested by authors appears to be doing a fine job at efficiently combining these different modalities given the performance reported in sections 4.2. and 4.3.
- The introduced prompt masking and skip connections both appear to be critical to strong empirical performance — the latter seems to be particularly important to mitigate task interference as evidenced by the ablations in section 4.4.
- The paper is very clear overall (in particular the methodology section) with nice visuals facilitating understanding and additional analyses in section 4.5 to help investigate the source of the performance lift.

**Weaknesses**
- The ablations for the different pre-training tasks in section 4.5 / Figure 6 are a bit puzzling. It does seem that the CRD task has destructive value on that particular binding affinity prediction task since: a) the performance of CRD + MLM or CRD + PPI leads to both lower performance Vs MLM or PPI alone respectively b) the performance of CRD + MLM + PPI is also lower vs just using MLM + PPI. This seems particularly important from a practical standpoint, and additional experiments are needed to confirm whether: 1) that problem applies to other downstream tasks or is just specific to binding affinity prediction — and if so, why?  2) there is something fundamentally wrong with the CRD pre-training as currently implemented? 3) there is a way to anticipate ex ante (or post fine tuning) which tokens should be used to ensure optimal task performance ?
- The ablation in section in Table 3 is a bit puzzling as well: it appears that the performance of PromptProtein without layer skip is lower than the performance from the conventional MTL. Could you please explain why that might be the case? (I would have assumed intermediate performance between conventional MTL and full PromptProtein as I presume the attention masks are still used in that ablation?)
- Several points (in section 4 primarily) were not fully clear (see clarity paragraph below).
- The following claim in conclusion does not seem fully substantiated: “PromptProtein beats state-of-the-art baselines by significant margins”. Authors do report the relevant baselines listed in the FLIP paper [1]. But since that paper was released, several methods have shown markedly superior performance for protein modeling & achieving high spearman with deep mutational scanning assays — see for example, [2] and [3]. I would suggest adding these two baselines to the analysis or tone done the SOTA claims.


------------------------------------------------------------------------------------------------------------------------
[1] Dallago, C., Mou, J., Johnston, K.E., Wittmann, B.J., Bhattacharya, N., Goldman, S., Madani, A., & Yang, K.K. (2022). FLIP: Benchmark tasks in fitness landscape inference for proteins. bioRxiv.

[2] Hsu, C., Verkuil, R., Liu, J., Lin, Z., Hie, B.L., Sercu, T., Lerer, A., & Rives, A. (2022). Learning inverse folding from millions of predicted structures. bioRxiv.

[3] Notin, P., Dias, M., Frazer, J., Marchena-Hurtado, J., Gomez, A.N., Marks, D.S., & Gal, Y. (2022). Tranception: Protein Fitness Prediction with Autoregressive Transformers and Inference-time Retrieval. ICML.

clarity,_quality,_novelty_and_reproducibility: **Clarity**
- Could you please clarify how you handle the instances where not all tasks are available for the same proteins in a given mini-batch (eg., instances where primary structure is available in Uniref50, but structure is not available in the PDB)?
- I would clarify in section 3.1 that $g_p$ (the linear projection of the prompt p) is a scalar. In my first lecture, I had (wrongly) assumed this was a vector of the same size as $h_p$ but that was not really making sense anymore when reading the analysis in section 4.
- “We observe that one amino acid in the 15-th layer can only attend to the local neighbors in the sequence, whereas the amino acid in the 33-th layer can attend to those amino acids that are more distant along the sequence but potentially closer in the 3D space” (in section B.3) —> I found this particularly insightful and would recommend to move this section from supplementary to the main text if space allows.
- What is the nature of the operator $\tau_\theta$ used for fine tuning? Linear projection? Any non-linearity applied?
- What is ESM-unstrained (Table 2)?
- Fig5 analysis — it is not clear what is being done here. Is this analysis conducted for a particular protein sequence (if yes, which one)? Or some aggregation over Uniref50 sequences? The embeddings from which layer(s) are being used here?
- I find the “skip connection” terminology a bit confusing as it seems that the term $g_p$ is used as a multiplicative factor for the attention-based transform but not the skip connection term. Also on Figure 3, the arrow labeled “skip connection” is in fact not the skip connection as it supports the computation of the  $g_p$ term which should not be present in the skip connection as per equation 2.
- In the conclusion: “In the future, we are interested in examining the effectiveness of the proposed prompt-guided multi-task pre-training and fine-tuning framework in domains where hierarchical task information is required in the pre-training stage.” —> Could you please provide examples for such domains?

**Quality**
- Very sound approach overall. Authors provided some very compelling empirical results, yet there are a few concerns with some of the ablation results as detailed above. Also the SOTA claim in the is not fully substantiated as discussed above as well.

**Novelty**
- The prompt masking and skip-connection weights are novel to my knowledge and appear to be both critical to the strong reported performance. Could the authors please confirm these two ideas are indeed introduced for the first time in this paper and not borrowed from the NLP literature?

**Reproducibility**
- Authors confirm that the code will be open sourced upon acceptance (section 4.1)

summary_of_the_review: This paper is aiming to address a very important area in protein modeling: learning rich sequence embeddings by leveraging multiple pre-training tasks jointly. The approach is sound and the methodological section is overall very clearly presented. There are a few concerns with respect to some of the results and claims as detailed above. Given the several strengths of the work, I would be willing to increase my score if authors address these points during rebuttal.

------------------------------------------------------------------------------------------------------------------------------------------------------------
[UPDATE POST REBUTTAL]

My most important concerns have been alleviated during rebuttal and I have increased my score accordingly.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces PromptProtein, a prompt tuning-based protein language model that combines pre-training tasks at the sequence, structure and protein-protein interaction levels. The model is then fine-tuned on function annotation and protein engineering downstream tasks.

In a nutshell, PromptProtein introduces learnable prompt embeddings for each of the three pre-training tasks. During fine-tuning, a small network learns to combine the three task-specific prompt embeddings into a prompt embedding that is specific to the downstream task at hand. The authors also propose two minor changes to the transformer architecture. Namely:
+ Attention from prompt tokens to sequence tokens is masked.
+ A learnable gate that is a linear function of the prompt token is applied to the skip connections.

Results on both function annotation and protein engineering downstream tasks are favourable relative to the baselines.

strength_and_weaknesses: Strengths:
+ Good results on function annotation and protein engineering tasks.

Weaknesses:
+ Results shown only for a subset of sub-tasks in the FLIP benchmark.
+ Ablation experiments are insufficient to disentangle the importance of the proposed architectural and task-related changes.
+ Insufficient details for full reproducibility in the manuscript.

Other weaknesses:
+ Limited methodological novelty (not critical for an application-focused paper).
clarity,_quality,_novelty_and_reproducibility: While the paper is mostly well written and easy to follow, there are a few low-level details that seem to be lacking from the manuscript. In brief, to the best of my knowledge:

+ Precise architectural details are lacking.
+ The structure prediction module $\kappa$ is not described in detail.
+ The prompt-tuning module $\tau_{\theta}$ is not described in detail.

Other questions:

1. Why does the performance of PromptProtein in Tables 2 and 3 disagree by a factor larger than the reported standard error?
2. Why do some visualisations use t-SNE and others MDS? It would be best to stick to one approach for consistency and avoid risk of cherry picking.
3. Are all prompts of length one? Have you experimented with longer prompts?
4. During fine-tuning, are only the prompt-tuning module parameters learnt, or are the main model parameter's also fine-tuned? (Algorithm 5 would suggest the former, but this could be more clearly stated in the main manuscript).
6. Did you experiment with saturating gates (e.g. sigmoid) for the skip-connection? Does it even occur that $g_P > 1$, implying that the skip-connection contribution changes sign? If so, what's the interpretation?

# Typos:

equivalence -> equivariance
FILP -> FLIP
summary_of_the_review: Within the extremely active field of protein language models, this paper introduces the following changes over the baselines:

1. While many baselines (e.g. ESM) pre-train only on sequences, the proposed approach pre-trains on three different tasks: masked language modelling on sequences, structure prediction and protein-protein interaction prediction.
2. The proposed approach relies on prompt tuning for (1) fine-tuning and (2) to allow for a degree of task specialisation at pre-training time.

While these changes are minor from a purely methodological viewpoint, I believe they are of interest to the protein language modelling literature. Moreover, these innovations appear to translate into improved downstream performance in function annotation and protein engineering tasks.

However, this work is not without limitations. Most notably, in my opinion, the experiments currently have two main shortcomings:

1. For most of the FLIP tasks (Thermostability, AAV, GB1), the authors show results only for a small subset of task variants. Instead, performance should be shown for *all* of the variants in the FLIP benchmark to prevent any potential cherry-picking.
2. The ablations do not provide sufficiently compelling evidence that the performance improvements are due to the specifics of PromptProtein and not e.g. the use of prompt tuning in general.

To this end, I recommend:
1. Providing experimental results for all subtasks in the FLIP benchmark.
2. Extend the ablation study to all tasks instead of GB1 (2-vs-rest) only.
3. To study the importance of pre-training with different prompt tokens for each task, compare to an ablation baseline that has a single learnable prompt of length three that is shared across the three pre-trained tasks, as opposed to three task-specific prompts of length one. This would help understand to which extent the performance improvement may be simply due to the extra expressivity in the skip-connections or, most importantly, using prompt tuning for fine-tuning to the downstream tasks as opposed to e.g. full model fine-tuning.
4. To study the importance of multi-task pre-training, results should be shown for ablation baselines trained without each of the three tasks, one at a time.

Because of this, I slightly lean towards weak rejection of the manuscript for now, but strongly encourage the authors to address these issues in the rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors propose a novel prompt-guided multi-task pre-training and fine-tuning framework for protein structure pre-training. Multi-level supervised information, including masked language modeling (MLM), CA coordinate prediction (CRD), and protein-protein interaction (PPI), are integrated into one unified learning framework without interfering with each other. Extensive empirical evaluation and embedding visualization validate the effectiveness of the proposed method.
strength_and_weaknesses: Pros:
1. The overall motivation is well founded and executed. Combining different level of supervised information for protein structure pre-training indeed leads to more informative embeddings, and the prompt-based multi-task learning avoids possible interference between different tasks. The visualization in Figure 4 and 5 provides clear evidence to support the proposed learning framework.
2. Extensive experiments are conducted on two types of down-stream tasks, function annotation and protein engineering. Multiple benchmark datasets are used, and many closely-related baselines are included for comparison, which makes the empirical evaluation hightly convincing.

Cons:
1. It seems that some details are missing (please correct me if I failed to find them in the manuscript). This includes the design of structure predictor (a MLP network?), and the prompt tuning module (linear combination of MLM/CRD/PPI prompts with learnable coefficients?).
2. In Equation (2), it looks like that the linear projection of prompt is layer independent, i.e., the weight is constant in all the layers, which conflicts with Figure 4. Please clarify.
3. Currently, the CRD task corresponds to the second and third levels of protein structures. Since the second level of protein structures refers to secondary structures (alpha-helix and beta-sheet), it may be more natural to introduce secondary structure prediction as learning task for this level. The training data can be the same as the CRD task, but with different supervised information.
4. Authors mentioned that the average error of coordinate prediction task on a single residue is 5A, which may indicate that the model is well trained to minimize this loss. This could be caused from the relatively simple design of structure predictor (if it is a MLP network), which I assume may not work well if the amino-acid sequence is long (e.g. more than 100 amino-acid residues). It should be helpful to use more sophisticated structure predictors (e.g. AlphaFold2’s structure module), but this may be out of the scope of this paper.
clarity,_quality,_novelty_and_reproducibility: Clarity: Overall satisfying, with only a few technical details missing.

Quality: Good.

Novelty: The prompt-based multi-task learning framework is novel for protein structure pre-training.

Reproducibility: Good if detailed design of structure predictor and prompt tuning modules can be further explained.
summary_of_the_review: The proposed prompt-based learning framework for protein structure pre-training is novel, and indeed resolves the possible interference between different learning tasks. A few minor issues need to be resolved in the author feedback to further improve the overall quality.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a prompt-guided multi-task pre-training and fine-tuning framework for protein function prediction.  The authors borrow the techniques from NLP and successfully applied to two tasks and show promising performance. 
strength_and_weaknesses: Strengths

1. The paper is well organized and written.
2. Some NLP techniques including pre training and prompt tuning are applied to protein structure study.
3. Empirical study shows good results.

Weaknesses

I would not say this to be a weakness.  The authors lend key techniques from NLP, but protein structure has four distinct levels.
Which levels are being helped by NLP techniques, and which levels are not?
clarity,_quality,_novelty_and_reproducibility: The paper is well structured. If there are points I missed, I believe it is my inadequacy in biological knowledge.
summary_of_the_review: The work successfully transfers NLP techniques to protein study.  The choice of the techniques including prompt-guided multi-task pre-training and fine-tuning are tailored to apply onto function prediction and protein engineering.  The work is thorough and thought provoking.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a prompt-guided multi-task pre-training framework and counterpart fine-tuning module to efficiently leverage the various structure levels of a protein.  Experiments on function annotation and protein engineering (FLIP benchmark) demonstrate the value of the approach. 

The reviewers and AC all agree that the paper makes some novel and sensible contributions, including the attention mask and skip connection components of the approach, which show very promising results.

The authors did a great job at addressing the reviewers comments. In particular the additional ablation studies and extended experimental results for all subtasks in the FLIP benchmark are great additions to demonstrate the value of the approach.

We strongly encourage the authors to tackle the future work direction on hierarchical task information. 
justification_for_why_not_higher_score: The paper translate the prompt related ideas from NLP to the study of proteins. Although the paper proposes some novel modifications to the transformer architecture and presents strong empirical results, the contributions are not ground breaking enough for a spotlight or oral presentation. 
justification_for_why_not_lower_score: The paper makes some very useful contributions to the study of proteins and their framework is likely to be adopted by or inspire researchers in the field. Although some of the reviewers did not revise their reviews after rebuttal, I checked thoroughly that their concerns were satisfactorily addressed. The authors did a great job at address them. In particular the new ablation studies and additional experiments are very convincing.
Decision: Accept: poster