Title: Prot2Text-V2: Protein Function Prediction with Multimodal Contrastive Alignment
Venue: NeurIPS 2025
Accepted: yes
Keywords: protein function prediction

multimodal large language model

contrastive learning
Abstract: Predicting protein function from sequence is a central challenge in computational biology. While existing methods rely heavily on structured ontologies or similarity-based techniques, they often lack the flexibility to express structure-free functional descriptions and novel biological functions. In this work, we introduce Prot2Text-V2, a novel multimodal sequence-to-text model that generates free-form natural language descriptions of protein function directly from amino acid sequences. Our method combines a protein language model as a sequence encoder (ESM-3B) and a decoder-only language model (LLaMA-3.1-8B-Instruct) through a lightweight nonlinear modality projector.
A key innovation is our Hybrid Sequence-level Contrastive Alignment Learning (H-SCALE), which improves cross-modal learning by matching mean- and std-pooled protein embeddings with text representations via contrastive loss. After the alignment phase, we apply instruction-based fine-tuning using LoRA on the decoder to teach the model how to generate accurate protein function descriptions conditioned on the protein sequence. We train Prot2Text-V2 on about 250K curated entries from SwissProt and evaluate it under low-homology conditions, where test sequences have low similarity with training samples. Prot2Text-V2 consistently outperforms traditional and LLM-based baselines across various metrics.


Peer reviews:
comments: SUMMARY:
This paper presents Prot2Text-V2, a multimodal sequence-to-text model for generating free-form protein function descriptions from amino acid sequences. It integrates a protein encoder (ESM2) with a decoder (LLaMA) via a lightweight projector and introduces a contrastive alignment strategy to improve semantic consistency. The method is evaluated under low-homology settings and shows strong performance compared to prior baselines.

STRENGTHS AND WEAKNESSES:
Strength:
The paper addresses the underexplored task of generating free-text protein function descriptions directly from sequences, which has practical value in annotating novel proteins beyond fixed ontology terms.

Weaknesses
1. The core modeling choices, including the use of a nonlinear projector and decoder-only architecture, are more of engineering simplifications than methodological innovations. While effective, these components largely build on existing paradigms rather than introducing fundamentally new ideas.

2. Although the paper emphasizes generalization under weak-homology settings, it lacks direct comparison to other function prediction methods that also claim to improve generalization. In addition, ProtT3 is only evaluated under random splits and not under the same low-homology CD-HIT split, limiting the fairness of the comparison.

QUESTIONS:
The appendix shows that including protein names significantly boosts performance. Could the authors clarify to what extent the model relies on such semantic cues, and whether this impacts its generalization to unnamed or novel proteins?

LIMITATIONS:
The authors clearly acknowledge the lack of wet-lab validation and discuss potential extensions to structural modalities and reasoning capabilities.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Prot2Text-V2, a model that aligns protein embeddings with an LLM's feature space using a contrastive strategy, H-SCALE. Following this alignment, the model fine-tunes a LLaMA-3.1-8B-Instruct decoder with LoRA to generate functional protein descriptions. Experiments on a low-homology dataset show the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.\tThe paper solves a meaningful problem, protein-to-text generation.\n2.\tThis paper is easy to understand.\n3.\tThe paper uses a contrastive strategy to better align protein representations with an LLM's feature space, which enhances the model's capability for protein function prediction.\n\nWeaknesses\n1.\tThe novelty is incremental. Most components in this work have existed before, such as ESM, finetuning the LLM with Lora, and contrastive learning. The main contribution of this paper lies in H-SCALE. However, the paper fails to provide a clear theoretical motivation for why aligning both mean and standard deviation is the optimal strategy. Furthermore, it lacks the necessary experimental ablation study (e.g., comparing mean-only, std-only, and the hybrid approaches) to validate this specific design.\n2.\tThe baseline comparisons could be further strengthened, particularly with respect to the state-of-the-art method, ProtT3. Although the paper includes an evaluation against ProtT3, this is conducted on ProtT3's original dataset, which follows a random split. However, one of the core contributions in this work emphasizes generalization under low-homology settings. Therefore, a direct and fair comparison on the more challenging low-homology dataset would be essential to convincingly demonstrate state-of-the-art generalization performance.\n3.\tP2T-GPT [1] also conducted experiments on proteins from a similar database (SwissProt). It trains a protein encoder and GPT from scratch and demonstrates promising results. It would be better to include an analysis and comparison with P2T-GPT.\n[1] Protein Captioning: Bridging the Gap between Protein Sequences and Natural Languages. In TOMM, 2024\n\nQUESTIONS:\nSee the Weakness part\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper introduces Prot2Text-V2, a multimodal framework for generating natural language descriptions of protein functions directly from amino acid sequences. Existing methods rely on structured ontologies or homology, limiting flexibility and generalization to novel proteins. Prot2Text-V2 addresses this by combining the ESM-3B protein language model encoder with an LLaMA-3.1-8B-Instruct decoder, connected via a nonlinear modality projector. A key innovation is Hybrid Sequence-level Contrastive Alignment Learning (H-SCALE), which aligns protein and text embeddings using mean and standard deviation pooling to enhance cross-modal consistency. After contrastive alignment, the model is fine-tuned with instruction-based LoRA to generate functional descriptions. Experiments on 250K SwissProt entries show Prot2Text-V2 outperforms baselines in low-homology scenarios, with improvements in lexical (e.g., BLEU-4: 43.34%) and semantic metrics (e.g., GPT score: 64.58).

STRENGTHS AND WEAKNESSES:
Strengths:

H-SCALE Contrastive Learning: Aligns protein and text embeddings using global and local features, improving semantic consistency.

Multimodal Architecture: Bridges protein sequences and natural language via ESM-3B and LLaMA-3.1, enabling free-form functional annotation.

Weakeness:

Training requires 8x A100 GPUs and 50 hours, restricting accessibility for small labs. LLM decoder (8B parameters) increases inference latency compared to lighter models.

QUESTIONS:
what kind of wet-lab validation experiments of selected proteins  do you want to do?

LIMITATIONS:
Yes. They have.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the task of protein captioning - given an amino acid sequence, predict a textual description with the hypothesized protein function. The paper proposes a new model which leverages a protein encoder and an LLM decoder linked via a modality projector. The model is trained via a two step training process: a contrastive learning step aimed to align the latent representations of protein sequences and of textual descriptions (H-SCALE), and a fine-tuning step aimed to guide the LLM decoder to produce protein textual descriptions. The approach is evaluated via multiple evaluation metrics on held out protein subsets of various sequence similarity to the training set, and compared to baselines to show improved results across held out sets and metrics. The claimed contributions are 1) the overall Prot2Text-V2 model, 2) the H-SCALE contrastive alignment training procedure, 3) the approach’s strong performance compared to baselines in the low homology regime.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper tackles the task of annotating protein sequences with textual descriptions which is an interesting problem in protein modeling. The proposed approach (pretrained protein encoder, pretrained llm decoder, modality projector, two-step training approach, H-SCALE) is clearly described and the authors add intuitions throughout the text for various architectural choices\n- The overall approach shows improved performance across several evaluation settings including held out protein subsets of various sequence similarity to the training set and multiple evaluation metrics\n\nWeaknesses:\n\nOverall, the originality and significance seems modest and I recommend a borderline accept due to the improved results. In particular, the task of protein captioning and the train-test splits used are established in cited prior work (Prot2Text, ProtT3) and the paper proposes a new approach that improves performance across metrics:\n- Approach: The paper notes the similarity to ProtT3 and highlights the H-SCALE contrastive learning approach as the notable difference. This seems like an incremental improvement to me, but I asked a few questions below that could help me understand the significance of this contribution more and I will reassess this after rebuttal\n- Experimental setting: I think the paper's originality could be improved by describing whether the analysis/metrics are newly introduced or mimic Prot2Text.\n- Results: I think the paper's significance could be improved by providing more insights into the results, e.g. discussing the discrepancy between metrics in Figure 5, whether the metrics highlighted in the main text correlate to the human expert evaluation described in the appendix, etc.\n\nQUESTIONS:\nH-SCALE\n* From what I understand, the ProtT3 baseline the paper mentions is also using an ESM encoder with an LLM decoder, a two step training process with a contrastive learning step followed with a protein captioning training step. The major difference flagged in the paper is Q-Former vs. H-SCALE. Can the differences in performance reported in Table 2 be attributed primarily to Q-Former vs. H-SCALE or are there other differences between the two approaches as well?\n* H-SCALE is posed as a new contrastive learning scheme that also leverages standard deviation and intermediate representations. Have you considered quantifying how much these choices impact alignment quality (in the style of Figure 4) or downstream performance when using sequence embeddings only?\n* Figure 6 highlights how the model that was trained with H-SCALE converges faster (and to a higher final performance) as a function of the number of training steps performed. Does this account for the steps performed during contrastive learning?\n\nOverall performance\n* Can you please explain the difference between the BLEU-2 (B-2) score in Table 1 and the BLEU-2 F1 Score (%) in Figure 5? What made you choose different metrics in Figure 5? Also, what made you exclude the exact match performance in Figure 6?\n* Are all the baselines included in the main text included in the human expert evaluation? I did not see BLAST and ProtT3. Did you find the human expert assessments to be correlated with the metrics used in the main text?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-23"}