Prot2Text-V2: Protein Function Prediction with Multimodal Contrastive Alignment
Authors: Xiao Fei
Michail Chatzianastasis
Sarah Almeida Carneiro
Hadi Abdine
Lawrence Paul Petalidis
Michalis Vazirgiannis
Venue: NeurIPS 2025 poster ()
Keywords: protein function prediction
multimodal large language model
contrastive learning
Primary area: machine_learning_for_sciences
TL;DR: Predict protein function description from amino-acid sequence with multimodal LLM and contrastive learning

Abstract: Predicting protein function from sequence is a central challenge in computational biology. While existing methods rely heavily on structured ontologies or similarity-based techniques, they often lack the flexibility to express structure-free functional descriptions and novel biological functions. In this work, we introduce Prot2Text-V2, a novel multimodal sequence-to-text model that generates free-form natural language descriptions of protein function directly from amino acid sequences. Our method combines a protein language model as a sequence encoder (ESM-3B) and a decoder-only language model (LLaMA-3.1-8B-Instruct) through a lightweight nonlinear modality projector.
A key innovation is our Hybrid Sequence-level Contrastive Alignment Learning (H-SCALE), which improves cross-modal learning by matching mean- and std-pooled protein embeddings with text representations via contrastive loss. After the alignment phase, we apply instruction-based fine-tuning using LoRA on the decoder to teach the model how to generate accurate protein function descriptions conditioned on the protein sequence. We train Prot2Text-V2 on about 250K curated entries from SwissProt and evaluate it under low-homology conditions, where test sequences have low similarity with training samples. Prot2Text-V2 consistently outperforms traditional and LLM-based baselines across various metrics.

Prot2Text-V2 V2: Protein Function Prediction with Multimodal Contrastive Alignment
Xiao Fei', Michail Chatziannastasis! Sarah Almeida Carneiro', Hadi Abdine", Lawrence P. Petalidis', Michalis Vazirgiannis
'École Polytechnique, Institut Polytechnique de Paris, France
Mohaaed bin Zayed University of Artificial Intelligence, United Arab Emirates
M44 Health, United Arab Emirates
xiao fei @polytechniqquee edu
(almeidacarneiro. mvazirg)@lix, wra.rrggllx..poyttccniieeeee. fr
mixalisxx700mail.ccmm com, hadi abdiiaedddenebbaat ac. 1petalidistm42
Abstract
Predicting protein function from sequence is a central challenge in computational biology. While existing methods rely heavily on structured ontologies similarity- based techniques, they often lack the flexibility to express structure-free functional descriptions and novel biological functions. In this work, we introduce Prot2Text- V2, a novel multimodal sequence-to- to-text model that generates free-form natural language descriptions of protein function directly from amino acid sequences. Our method combines a protein language model as a sequence encoder (ESM-3B) and decoder-only language model (LLaMA-3.1-8B-- ----rutsttt through lightweight nonlinear modality projector. A key innovation our Hybrid Sequence-level Contrastive Alignment Learning (H-SCALE), which improves cross- cross-modal learn- ing by matching mean- and std-pooled protein embeddings with text representa- tions via contrastive loss. After the alignment phase, apply instruction-base fine-tuning using LoRA on the decoder to teach the model how to generate accu- rate protein function descriptions conditioned on the protein sequence. We train Prot2Teext V2 on about 250K curated entries from SwissProt and evaluate it under howolooyggy conditions, where test sequences have low similarity with train- ing samples. Prot2Text-V2 consistently outperforms traditional and LLM-based baselines across various metrics.
Introduction
As sequencing technologies uncover millions of new proteins, most remain uncharacterized, under- scoring the urgent need for accurate, scalable function prediction. Traditional tools like BLAST [Attchhl et al. 1990) and InterPro [Hunter et al. 2009). though effective, rely on manual curation and homology- based inference, making them slow and especilyy for proteins from poorly studied organisms lacking close homologues. These approaches are also constrained by predefined labels or controlled vocabularies, restricting their ability express the full complexity of protein function. Protein function is complex, spanning molecular mechanisms, pathways, and evolutionary roles. Free-text predictions, enabled by natural language generation, offer greater flexibility and expressiveness than rigid labels like Gene Ontology (GO) terms. This has spurred growing interest in models that can generate rich, compositional natural language annotations of protein function.
Although deep learning models for protein function prediction have achieved promising performance, usually they have been thoroughly tested on protein samples with low sequence similarity
'The source code for this project is available at http:://ghhub.commmmccci//tttttt
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Protein Sequence Encoder
(b) Language Model Decoder
о Embedding Vectors E
-
Protein Captioning sutback
ESM2-38 Encoder fo
Modality Projector Bo
LLaMA-3.1 100t--3.--8---- 1-nt-rnttt Decoder
Amino-adi Sequence Sy <OUS> MLP POALLL LL... oN <808>
AM
p
Amyyo
Drosddingg
HD
Ertedding
Homo sppaos (Human))
UniProt
Visu
Doncciio
Figure I: Illustration of the multimodal architecture and primary data flow of Prot2Text- V2. (a) In the protein encoder, the amino-acio sequence is first encoded by the protein language model and then projected to the hidden dimension of the text decoder with a two-layer non-linear modality projector. (b) In the language model decoder, the projected sequence of protein embeddings is inserted into the sequence of word embeddings of the user message, forming the prompt to the language model. Finally, the decoder generates the description of the protein auto-regressively.
to training samples, which is critical scenario for real-world generalization. This gap raises concerns that existing models may not truly understand protein function, but instead rely heavily on sequence similarity. As result, they often fail to generalize beyond their training data, limiting their effectiveness for novel or poorly characterized proteins. To address this, we propose Prot2Text- V2, a multimodal sequence-to-ter to-text framework that generates rich, textual functional descriptions directly from amino acid sequences. Our approach combines a powerful protein encoder (ESM-3B [Lin et al. 2023) with a large language model decoder (LLaMA-3.J 1--..------------- -8B--Inttuttt (Grattatiori et al. 2024]
key innovation of our approach is the Hybrid Sequence-level Contrastive Aligmment Learning (H-SCALE) framework, which establishes direct and parameter-efficient alignment between protein and text representations. Unlike prior models such as ProtT3 that rely on additional attention modules (e.g., Q-Formers) and redundant text encoders to achieve multimodal alignment, H-SCALE performs explicit contrastive alignment between projected protein embeddings and intermediate text embed- dings within the decoder itself. This design preserves the full expressivity of the pretrained protein encoder, removes the learned bottleneck introduced by query-based alignment, and significantly reduces latency and memory overhead. Following this aligmment phase, the decoder is fine-tur tunet using instruction bastruction-based LoRA (Hu et al., 2022), enabling controlled natural language generation conditioned protein features and prompts. To comprehensively assess the model, we evaluate Prot2Text- V2 on low-homology proteins, where accurate generalization beyond sequence similarity most biologically meaningful. Our evaluation includes automated metrics, L.LM-as-a-judge scoring for semantic fidelity, and expert human assessment, providing multi-faceted understanding of model quality and robustness.
Our contributions can be summarized as follows:
We introduce Prot2Text-V2, a model that generates Humaa--readalee descriptions of protein function directly from amino acid sequences. combines pretrained ESM-3B protein encoder with instruction-tuned LLaMA-33118B decoder. We combine them via two- layer nonlinear modality projector, which simplifies training, accelerates inference, and avoids the need for complex cross-attention mechanisms used in prior work.
We introduce H-SCALE, a new contrastive learning scheme based on both mean and stan- dard deviation-pool
sequence embeddings and intermediate decoder text representations. This alignment phase enables more effective downstream instruction tuning.
We highlight Prot2Text-V22 strong performance in the reak-homology regime, where generalization is most critical and biologically relevant, We achieve state-of- state-of-the-ane of-thh-art art results across lexical and semantic metrics, outperforming both traditional methods and recent LLM baselines.
The rest of the paper is structured as follows. Section 2 reviews related work on protein function prediction, multimodal learning, and contrastive learning. Section details our proposed method, Prot2Text-V V2, including its architecture, the contrastive alignment strategy, and instruction-bart based fine-

---

tuning, Section covers the experimental setup, datasets, baselines, and results. Section 5 discusses limitations and future directions, and Section Section Additional materials are provided in the Appendix.
Related Work
Traditional Protein Function Prediction: Protein function annotation has traditionally relied on sequence homology and structured ontologies like Gene Ontology (GO). Early tools such as BLAST [Altschul et al. 1990] and Foldseek [van Kempen et al. 2022] identify similar proteins to infer function, but they struggle with novel sequences lacking close homologs. To address this, recent deep learning approaches predict GO terms from sequence and/or structure using convolutional networks. graph- based models, and protein language models [Kulmanov and Hochndorí] 2020 Yuan ct al. [224 2023 Fan et al. [2000] Li et al. 2023]. More recent efforts apply autoregressive modeling, prompt luning, and hybrid architectures to refine label prediction [Wang et al. 2025 Rong 2024]. While these methods improve scalability and coverage, they still reduce complex protein functions predefined labels, limiting the granularity and expressiveness of the annotations.
Recent approaches have shifted toward generating natural lan- guage descriptions directly from protein sequences, offering
Protein-to-Text Generative Models:
flexible alternative to predefined labels. Models like Prot2Text (Abdine et al. (024)) pioneered this direction but still face challenges in effectively aligning sequence encoders with language decoders. Subsequent work has explored
range of strategies: some focus on sequence- or structure-based generation [Liu et al. 2024b Xiao
al. 2024). others leverage multimodal or iterative training across molecular representations [Liu
et al. 2024a Zhou et al. 2025 Kim et al. 2025 Wang et al., 2023]. and recent models incorporate
instruction tuning and large language models for functional synthesis [Lv al. 2025 Wang et al.
2024a].
Contrastive Protein-Tess Text Alignment: Contrastive learning has emerged powerful strategy for aligning protein sequences with textual annotations, enabling richer multimodal representations. Early work adapted CLII--tyle frameworks protein embeddings with separate encoders (Wu ct al. 2024 Radford et al.) 2021]. while subsequent models incorporated structural and functional modalities to enhance alignment [Su al. 2024 Liu et al. 2024b Ünsal et al. 2024 Wang et al. methods with contrastive objectives [Zheng and Li [224 Xu et al. [023 further improve alignment by leveraging sequence-aware architectures. Despite these advances, many models still depend on pooled or special-toker embeddings, which can miss fine-grained details.
2024b). Transformer-be
Methodology
The major objective of Prot2Text-V V2 to generate accurate and informative free-text descriptions of protein function directly from amino acid sequences. This task requires model capable of understanding and aligning protein representations with natural language. However, LLMs operate in different semantic space that is not natively aligned with protein representations. To address this challenge, our method introduces: (1) a lightweight and flexible decoder-only architecture with a nonlinear modality projector and (2) brief yet effective contrastive alignment phase before standard instruction tuning.
3.1 Decoder-Only Model Architecture
Protein language models (PLMs) like Evolutionary Scale Modeling-2 (ESM2) Lin et al. 2023 have demonstrated a strong ability to capture evolutionary signals, structural features, and functional characteristics by leveraging large-scale pretraining millions of protein sequences. Therefore, choose ESM2 as the protein amino-acid sequence encoder f-)-) with pre-trained parameters 0. For an input protein amino-acid sequence S, E N' composed of a series of amino-acid tokens, we first embed it with the ESM2-3B encoder, which returns a sequence of updated embedding vectors as sequence features: Ep = (op))) E R!xdy We then employ LLAMA-3.. 1-1.A....-B--.... --Stt-nstttutt |Grattafiori et al. 2024 as our foundation large language model gs to tackle with the protein captioning task. This LLM has been extensively pretrained on diverse textual corpora and further tuned for instrucioonfollowiii tasks, making it well-suited for our task.
3

---

The protein encoder and language decoder differ in both hidden dimensions and semantic spaces, making them incompatible. Directly injecting raw protein embeddings into the decoder can result in semantic misalignment, degrading generation quality. Prior models [Brandes et al. 2022 Rao et al,, 2019) address this with cross-attention or multimodal transformers, but require extensive data and training. To bridge this gap, we introduce two-layer nonlinear projection module hg. as in Equation I, that maps protein embeddings into the decoder's semantic space. It uses GELU activation [Hendrycks and Gimpel 2016) for nonlinearity and layer normalization (LN) to stabilize training, producing decoder-compation embeddings for effective cross-modal conditioning.
H = hg(Ep)) LN(W2 o(W1 Ep)) E Rxd, with W1 Rd W2 E R
(1)
Finally, we incorporate the projected embedding sequence into the language decoder's s chat dialogue using the following structured template with flexible user message. The assistant's response (colored in red) is left for the model to regreseegrys generate:
System: You are scientific assistant specialized in protein function predictions. Given the sequence embeddings of a protein, describe its function clearly and concisely in professional
language.
User: (Protein name:
<NAME>?)
(taxonomy:
<TAXON>;)*
sequence embeddings:
11 pp1 11 H
Assistant: <PROTEIN DESCRIPTION
Optional metadata text fields
Figure 2:
A flexible chat template for language decoders, where user message metadata fields subject to dropout during training. During evaluation, these fields are either fully retained or completely removed, enabling two distinct testing scenarios.
Our integration approach specifically inserts placeholder tokens into the chat dialogue, which the language decoder initially converts to word embeddings Hp We then perform a soft-prompt [Lester al., 202T] replacement of the protein sequence segment with the encoder-adapter's output embeddings. This direct integration strategy unlike methods like [van den Oord et al.] 2017] requiring additional discretization steps provides two key advantages: (I) it preserves the full semantic richness of the encoder's continuous representations, and (2) enables precise multimodal alignment by eliminating information bottlenecks that typically occur codebook-basee approaches.
3.2 Two-Stage Training Process
propose a two-stage alignment and fine-tuning approach first align and project the protein sequence encoder's outputs Ep to the same hidden semantic space the language decoder, then fine-tune the decoder to produce accurate functional descriptions by conditioning on the aligned protein representations. The additional alignment stage ensures stable coos- understanding before the standard end-to-end fine-tuning, thus producing reliable predictions by leveraging the full context of encoder hidden states.
3.2.1 Hybrid Sequence-Level Contrastive Alignment Learning
In this alignment stage, we connect our proposed modality adapter the protein sequence encoder and train the system using sequence-description pairs (Sp, Tp.) from our training set (illustrated in Figure 3.. The amino-acid sequence will be processed following a standard pipeline, first by the pre-trained sequence encoder and then the randomly initialized modality adapter (Fquation 2 In parallel, we process the protein's 's text description through the frozen language decoder, which comprises 32 self-attention blocks, and extract the intermediate representations from the 16th block (Equation 3.. Where g = 9head 932 o 931 o... o 91 Jembed denote the composition of the sequential self-attention blocks in the LLaMA-3.1 architecture, including the language model head and the input token embedding layer. The sequence-lewe protein embedding is represented as H E Rd, where d, = dg by projection. Moreover, for most protein-text pairs, the sequence length (eeg not equal to the text length lxt. i.c., looq Ixx.

---

(a) Hybrid Sequence-level Contrastive Alignment Learning
(b) Supervised Fine Tuning
Lh-scale d LInfONCH (1 H4) ) A2 LIffONCH (o,, o.)
L ErrssEnnrrppy Cup
Next-token Probability
avg
std
avg
std
LLaMA Decoder ly
Embedding Vectors H
[16 Hidden States p
Prompt Text T
You
Modality Projector
LLaMA Blocks x16
Modality Projector
ESM Encoder fo
LLaMA Blocks x16
ESM Encoder fa
Amino-acid Sequence Sp LLP PLLLLP L..... QN-EDS-
Labeled Description Tp Functions cell surface
Amino-acid Sequence Sp <CLS> MLP GLALLL QN <EDS>
Illustration of the two-stage training process of Prot2Text-V2. (a) During the first stage, Hybrid Sequence-level Contrastive Alignment Learning (H-SCALE) aligns the protein encoder outputs with the semantic space of the pretrained language decoder, enabling effective cross-modal learning (b) In the second stage, instruction -assed based supervised fine-tuning performed to teach the decoder how to generate protein function descriptions based on the aligned embeddings. Red blocks denote trainable parameters, while blue blocks indicate frozen components.
Figure 3:
Hong M = (fe(()))) = =H p. pulsoy where H E Rdd,
(2)
H = 916 (on(1gs(... (Dembed (CmmmeaaTTp))-aaeee (Tp)))...))
HH p.
Hizt p.lcus
(3)
This strategy overcomes two key issues in aligning language decoder with a different-modality encoder: (1) raw input token embeddings lack the self-attention context needed to capture linguistic relationships, and (2) final decoder outputs are overspecialized for next-token prediction, losing broader semantic information through extensive autoregressive optimization. By using the 16th decoder block, we capture rich contextualized representations that better encode protein-relevant semantics. The extracted representations are aligned with the protein sequence embeddings via combined InfoNCE contrastive loss [Oord et al.) 2018). applied to both mean- and standard-deviation- pooled features:
Latign = X1 LinfoNCE(Us: mt) + A2 LinfoNCE(O,, ot): exp(1 x y/T)
(4)
where LinfoNCE(2, v) log
(5)
ΣB exp(e "Tpp)
where μ4 McanPool H))))) and μ4 McanPool (H (100) denote the global mean-pooled
embeddings of the protein sequence and the labeled description, respectively. Similarly, the textual StdPool (10 (1666, capturing local variation through mean and standard deviation pooling. The temperature hyperparameter is set to 0.07, and the batch size is B 1024. Finally, the loss terms are balanced using weights X1 = 0.7 and A2 = 0.3, which control the contributions of global versus local alignment.
features pooled to obtain a, = MeanPool (H779) and
By minimizing Lalign, we enforce both global consistency and local variation alignment between the original and transformed embeddings, resulting in more robust cross-modal representations. This alignment phase, performed before standard end-to-end fine-tuning, plays critical role: it projects the protein embeddings into the same semantic space as the decoder's token representations, result, the decoder can more effectively integrate information from both the input prompt and
5

---

Without Alignment
With H H-SCALE Alignment
Protein Embedding Sequerccss
Protein Embedeing Sequences
304
10-
10
Nomm 1000 Log
15-
15
20
20
-65
25-
25
Text Embeddiing Sequences
Text Embedding Sequences
Figure 4: Illustration on similarity matrices before (left) and after (right) alignment on a group of test protein-description pairs. After alignment, protein embeddings show strong one-to-one correspondence with their matching text embeddings, as indicated by the prominent diagonal. This reflects significantly higher similarity between each protein sequence embedding and its own text description compared to mismatched pairs, demonstrating successful cross modal alignment.
the protein features. This reduces the risk of representational mismatch, where the decoder might otherwise ignore or overemphasize one modality due incompatible embedding distributions.
We visualize the effect of the contrastive learning on pairwise similarity between protein embeddings and their corresponding text embeddings in Figure Before applying H-SCALE, the similarity matrix is diffuse and lacks a clear structure, suggesting that protein and text embeddings are poorly aligned, as they show high similarity with mismatched pairs. After our contrastive alignment, the matrix exhibits strong diagonal dominance, where each protein sequence embedding is most similar its true text description. Further details also explored Appendix A
3.2.2 End-to-End Fine-Tuning
In contrast to the alignment stage where we fix the language decoder then train the modality adapter and partially the protein encoder, in this stage, focus fine-tuning the pretrained language decoder with Low Rank Approximation (LoRA) [Hu et al. 2022 with an end-to-end teacher forcing cross entropy regulate the conditional auto-regressive generation of the decoder.
The model is fine-tuned on instruction-formann (1) protein sequence encoded as multimodal embeddings H1ç00 (2) optional metadata including the name the protein and taxonomic classification, and (3) task instruction T guiding the model to generate the description. The training objective maximizes the likelihood of the target caption while masking loss computation over the input prompt tokens. Specifically, given an input sequence embeddings H and target tokens Tp, the model minimizes the supervised fine tuning loss:
protein-caption pairs, where each input consists of
T
Caft = ErrossEntrop (Tp,P) =
Σ
log (t
(6)
where represents the predicted next-token probability distribution using the language model head of the decoder. The gradients propagated only the caption tokens Tp, ensuring the model learns to generate accurate descriptions without overfitting to the prompt structure.
To standardize the input, the structured chat template is applied to integrate system instructions, user messages and the response of the model. This ensures consistent formatting across diverse prompts, improving the generalization ability of the model. To improve robustness, we apply dropout to each metadata field (name and taxonomy) with an 80% probability during training. This approach addresses incomplete inputs, reduces over-reliance on potentially noisy annotations, and enhances generalization by simulating real-world scenarios where metadata may missing or inconsistent. This supervised fine-tuning stage further aligns the decoder's generative behavior with protein

---

Table Evaluation results for Prot2Text- V2 and baselines. Lexical metrics include exact match ratio (Exct), BLEU-2 F1 Score (B-2), BLEU-4 FI Score (B-4), ROUGE-1 FI Score (R-1), ROUGE-2 F1 Score (R-2), and ROUGE-L Fl Score (R-L). Semantic metrics comprise BERTScore using RoBERTa (RBT) and BioBERT (BBT) embeddings, plus GPT-4o-based L.LM judge score (GPT). Prot2TText- V2 with H-SCALE outperforms general-purpose L.M,s, as well as various baselines and contrastive
learning variants.
Model
Exct
B-2
B-4
R-1
R-2
R-L
RBT
BET
GPT
GPT-4o-mini OpenAl 2023
(0-shol)
2.65
0.31
10.81
1.25
7.73
81.44
71.36
GPT-4o-mini( I-shot
5.86
1.24
15.38
2.66
11.58
83.21
72.39
Souuee3.S-Sonnee Anthrupic
2024
(0-shol)
5.47
0.90
13.91
1.92
10.22
82,98
72.39
Claude-3 3.5- -shoot)
6.62
1.54
16.02
3.09
12.14
83,58
71.79
P2T-GPT
Zhang el
2024
17.24
33.30
26.06
38.14 14
23.72
35.47
88.23
78.66
Prot2Text
Abdine
2024
32.05
36.36
32.45
50.49
42.60
48.33
90.56
84,26
56.95
Transformer Vaswani ct
2017
17.48
15.75
27.80
19.44
26.07
82.31
75.58
ESM2GPT
35,49
32.11 1
47.46
39.18
45.31
89,97
83.21
RGCN2Transfomerr
31.64
27.97
42.43
34.91
40.72
90.58
84.30
RGCN2GPT
24.05
21.63
36.20
28.01
34.40
85,12
78.91
RGCN&ESM2GPT
33.01
30.39
45.75
37.38
43.63
89.04
82.51
SEQ2LLaMA
20.85
24.97
21.44
33,99
25.86
32,18
87.13
77.32
30.99
RGCN2LLaMA
22.79
25.96
22.46
39.36
31.34
37.60
88.31
79.58
30.69
RGCNNESM2LLaMA
30.07
35.84
31.84
50.87
43.28
48 98
90,92
84.75
58.48
Prot2Text-V2 (wl H-SCALE)
36.52
42.70
38.98
54.71
47.62
52,90
91.33
85.58
60.93
Proi2Text-V2 (w/ H-SCALE)
39.16
46.67
43.34
57.24
50.17
$5.39
91.95
86.81
64.58
captioning through structured instruction tuning. Finally, we discuss the computational complexity of the whole pipeline in Appendix B
4 Experiments and Results
4.1 Experimental Setup
The dataset used during our experiments was the one proposed by Abdine et al. [024]] This dataset is a multimodal dataset with 256,690 proteins, filtered originally from SwissProt (see Appendix C |Bairoch and Apweiler 2000). with their respective corresponding sequence, the predicted structure from AlphaFold (Jumper et al. 202T), and the textual deseription. This curated dataset provides aligned sequence, structure, and text modalities, with rigorous filtering and low redundancy across splits, making it better suited for our multimodal learning tasks. Notably, to the best of our knowledge, this is the only publicly available dataset that minimizes protein sequence similarity between training and test sets. We further discuss our data splits in Appendix C.I
Dataset:
Hyperparameters: Our model is implemented using PyTorch and trained on single node with 8 NVIDIA 1 80GB GPUs We use the AdamW optimizer [Loshchilov and Hutter [017]] with c= 1 10-6 B1 0.9, and B2 0.999. The learning rate starts at 0.0002 and decays to zero following a cosine scheduler, with warm-up period covering 6% of the total training steps. For the LoR A adapter, we apply it to the self-attention modules in both the ESM encoder and LLaMAA decoder, using a rank of 32 and an a value of 64. Training lasts for 12 epochs in contrastive learning and 24 epochs for supervised fine-tuning. During the contrastive learning stage, the batch size per device is 1024, which is further divided into chunks to accommodate memory constraints. The batch size is set to 4 per GPU, and gradient accumulation is applied every 8 forward passes, resulting in an effective batch size of 256.
Evaluation Metrics: For quantitative evaluation, we assess Prot2Text- V2 using standard sequence
generation metries: BLEU (Papineni et al. 2002), ROUGE (1, 2, L) Lin 2004), BERTScore [Zhag
et al. 2019 and contextual metrics based on RoBERTa [Liu et al. 2019 and BioBERT [Lee et al.
2020 - BLEU and ROUGE capture lexical overlap- BL.EU emphasizing precision and ROUGE
while BERT-based metrics evaluate semantic similarity using contextual embeddings. As
7

---

BLEU-2 F1 Score (%)
BLEU-4 F1 Score (%)
RougeL F1 Score (9)
45
40
25
40-
25
25
30-
15-
15
Prot2Text-V2
ProtZText-V2
Prot2Text-V2
BLAST
Prot2Text
Prot2Text
Prot7Text
[0.30]
(3035]
(35.40)
(40-45)
(48,58)
(50.55)
($5.00)
(6085)
(63.70)
10.301
G0.351
(3.40)
(40.49)
(45,50)
(0.55)
(5.60)
68885)
[0.30]
(00,55)
05.401
1404551
(4550)
(50.55)
801
(60
(68 (9.70)
(85,
(40
RoBERTa Score (%)
BioBERT Score (%)
GPT-4o Score (/10)
84
90
78
74
54-
Prot2Text-V2
Prot2Text-V2
Prot2Text-V2
BLAST
BLAST
BLAST
Prot2Text
70
Prot2Text
[0.30]
(30.35)
(25.40)
(40. 10-45|
(45:50)
(50 $0.55|
(60.85)
(e5.70)
10.30]
(00.51
(35.40)
(40 451
(50.55) (45.50)
(98.60)
(09:70)
(0.30]
(20.35)
(35 40)
(40 451
(50.55) (45.50)
(BS 601
(60 e5]
(65.70)
(59
(80
BitScore BLAST Similarity
Figure 5: Evaluation across BLAST sequence bitscore bins (%) shows that Prot2Text-V V2 outperform
baselines (Prot2Text and BLAST). The evaluations show consistent improvements, particularly the [0-50] bins, highlighting better generalization to remote homologs.
BLBU and ROUGE are non-decomposable corpus-level scores, we estimate their variance and con- fidence intervals using the method described in Appendix E. Additionally, we include a semantic evaluation metric using GPT-4-mini as an LL-M-based judge. Unlike traditional metrics, it assesses prediction correctness by reasoning about semantic content, accommodating functional variations and omissions due to token truncation. This allows more flcxible evaluation, particularly when predictions partially align with generalize beyond the ground truth. Full details are in Appendix H Baseline models and ablations We evaluate our model against methods from two distinct paradigms: tradi- tional requence-alignmee we describe the baselines in more detail.
based methods and deep learning generative methods. In the following,
BLAST: BLAST [Altschul al. 1990] is classic alignment-based tool that predicts protein function by finding homologs with known annotations. It works well for high sequence similarity (>70%) but struggles below 40%, limiting its effectiveness for novel distant proteins. Despite this, BLAST is often omitted from recent M-bsed function prediction comparisons, making it difficult to assess whether neural models genuinely generalize mainly retrieve similar sequences.
Prot2Texx: We also compare Prot2Text-V V2 against Prot2Text and a range of unimodal and multi- modal baselines based on the cross-attention framework of Abdine et al. [2024]. Unimodal mod- els RGCN [Schlichtkrull et al. 2018). ESM, and a Transformer [Vaswani et al. [2017] trained from scratch- process either structural graphs or sequences independently. Multimodal baselines include RGCN+ESM, which concatenates graph and sequence embeddings without explicit amino-acid level alignment, and RGCN combined with scratch- trainee Transformer. We further evaluate multiple sizes of Prot2Text, all using GPT-2 [Radford et al. 2018 as the language decoder.
cratch-ppoceess
ProtT3: We compare our method to ProtT3 [Liu et al. 2024b). state-of-the-an approach that also employs language model with ESM-2 and a cross modal projector for protein encoding. While

---

there are some similarities with ProtT3 architecture, our model departs in the following essential respects: ProtT3 relies on a Q-Former, an additional attention module with learnable queries of a fixed number and a redundant text encoder, to align the protein and text modalities. Which introduces computational overhead, potential information loss from query transformation, and lacks thorough evaluation of weak homology proteins. In contrast, Pror2Text--V? V2 follows a decoder-only design with a lightweight projector and with a parameter- alignment method. Rather than learning alignment implicitly through a query-attention module, H-SCALE directly aligns the projected protein represen- tations to intermediate text embeddings with a new contrastive learning objective. This eliminates the learned bottleneck, preserves the full expressivity of the pretrained protein encoder, and avoids added parameters, latency, and memory, and also allows us to use a more powerful LLM decoder. P2T-GPT: We also compare our model to P2T-GPT [Zhang et al. 2024 a relatively lightweight model (160M parameters) trained entirely from scratch without leveraging large-scale pretrained components or proper multimodal alignment.
Generic LLMs: Despite not being specifically designed for protein analysis, seneral-purpose LLMs such Galactica [Taylor al., 2022], Claude 3.5 Sonnet [Anthropic] 2024]. LLaMA 3.1 [Grattafiori iet al. 2024], and GPT-4o mini [OpenAT] [023) have been trained on massive and diverse datasets and are widely adopted in practice. We include these models in our comparison to assess whether our specialized pretraining approach offers superior performance compared to zero-shot and few-shot predictions from LLMs.
Ablation Studies: To validate the effectiveness of our architecture, we conduct ablation studies across five variants: (i) SEQ2LLaMA, where raw amino acid sequences (in text format) are inserted directly into the user prompt for the model; (ii) RGCN2LI LaMA, using an RGCN encoder to encode protein structure predicted with AlphaFold (Varadi et al. 2022]; (iii) RGCN&ESM2L
which fuses outputs from both RGCN and ESM encoders via gated cross-attention fusion module to integrate structural and sequence information; and (iv) Prot2Text- V2 (w/o CL), which skips con- trastive pretraining and begins directly from instruction tuning. These variations allow us to isolate the contributions of individual components.
4.2 Results
We present our evaluation results in protein function prediction in Table I Prot2Text-V V2 (w/ H- SCALE) outperforms all baselines across lexical and semantic metries. It achieves top BLEU-2 (48.32) and BLEU-4 (36.07) scores, reflecting strong n-gram precision, along with leading ROUGE-1 (63.51), ROUGE-2 (45.06), and ROUGE-L (59.66) scores, indicating improved fluency and recall. For semantic alignment, it surpasses general-purpose LLMs (RBT: 83.58). Prot2Text variants (RBT: 90.58), and ablations (RBT: 91.33), reaching a top RoBERTa score of 91.95. On the domain-specifer BioBERT metric, it leads with 86.81, outperforming general LLMS (72.39), prior variants (84.30), and ablations (85.58). Prot2Text- V2 (w/ H-SCALE) also excels in semantic alignment with reference texts. Its GPT score of 64.58 further confirms superior semantic coherence and generation quality, demonstrating strong lexical accuracy and contextual relevance for robust text generation.
Furthermore, we evaluate performance focusing on the challenging low-similarity cases where traditional methods often fail. Figure 5 shows average BLEU-2, BLEU-4, ROUGE, RoBERTa, BioBERT, and GPT-40 judge metrics across BLAST bitscore bins ranging from low (0-30) to medium-high (65-70). While BLAST outperforms multimodal methods at higher similarity levels (above 55%), it performs poorly in low-similarity ranges (0-50%) where sequence copying is ineffective. Notably, T2 achieves the highest scores in these low-similarity scenarios. Additional details on BLAST score computation are provided in Appendix [D
We further compare our main model, Prot2Text-V2 (w/ H-SCALE), against ProiT3 Liu et al. 2024b) and its variants in Table We retrained Prot2Text-V2 V2 using the same dataset split provided by ProtT3, which follows a random partitioning strategy and does not explicitly focus on the low sequence similarity regime. Additionally, to evaluate the models under the same low-homology settings. in Tables 3 and 4 we filtered test proteins with the same bitscore ranges as shown in Figure and made direct comparison between different approaches on the same ranges to evaluate BLEU-4 F1, and BioBert F1 scores. Prot2Text- V2 consistently outperforms ProtT3 across the reported metrics, highlighting the superior performance of our method under their evaluation protocol. From the comparison with P2T-GPT, we also conclude that Prot2Text-V V2 benefits from integrating powerful pretrained models (FSM and ..MMM) and an H-SCALE a.CALeaadapter,, enabling it to capture richer biological and linguistic priors. This architectural difference is reflected in the substantial performance

---

gains across all evaluation metrics, especially in semantic fidelity and functional relevance, Finally, we report extra qualitative evaluations along with human expert evaluations Appendix [F] and G
Table 2:
Evaluation results of our proposed model against ProtT3 and Galactica. For a fair comparison, we report results on the original data split from ProtT3 paper.
Model
Exct
B-2
B-4
R-1
R-2
R-L
M-R
Galactica (Taylor et al.
2022
13.49
42.48
37.79
44,42
34.73
42,46
48.27
ProfT3 (w/ MLP)
20.60
48.96
44.59
57.28
50.17
56.89
57.30
ProfT3 (w/o stage
22.88
50.21
46.76
58.64
51.63
57.17
58.62
ProtT3 [Luu al.
2024b
25.74
55.03
51.47
63.67
56.59
62.16
63.63
Prot2Text-V2
49.04
60.16
$7.29
67.33
61.39
65.61
66.05
Table BLEU-4 (B-4) Fl scores of Prot2Text, ProtT3, and Prot2Text-V2 across BLAST sequence
bitscore bins (%).
Model/BitScore [0,30]
(30,35)
(35,40)
(40,45)
(45,50)
(50,55)
(55,60)
(60,65)
(65,70]
Prot2Text
6.58
6.41
7.51
8.63
12.58
16.03
20.76
21.68
27.04
ProtT3
6.19
5.26
7.12
9.76
13.10
10.87
18.97
30.95
40.82
Pr012Text-V2
11.89
15.21
14.78
14.00
16.05
22.20
28.65
29.26
31.08
Table 4: BioBERT (BBT) F1 scores of Prot2Text, ProtT3, and Prot2 V2 across BLAST sequence
bitscore bins (%).
Model/BitScore [0,30]
(30,35)
(35,40)
(40,45)
(45,50)
(50,55)
(55,60)
(60,65)
(65,70)
Prot2Text
73.38
73.92
74.30
74.11
75.04
76.64
78.24
78.31
78.98
ProfT3
67.57
66.44
69.19
74.37
79.73
78.11
76.44
81.06
83.99
Pro12Text-V2
76.74
77.23
77.52
77.68
78.99
80.61
81.07
81.95
83.49
5
Limitations and Future Work
While our evaluation shows promising results using lexical and semantic metrics, these may not fully reflect funetional correctness. Therefore, limitation of the work is the lack of experimental (wet-lab) validation to confirm biological accuracy. For future work, we plan to establish collaborations with experimental biologists carry out wet-lab validation of selected proteins, enabling more biologically grounded evaluation. Furthermore, future work will focus on integrating models with stronger reasoning capabilities that can better capture the protein function. We also aim improve our model by incorporating additional molecular modalities, such as structural and surface features.
6 Conclusion
In this work, we proposed Prot2Text- V2, a novel multimodal framework for the prediction of protein function in free text format. First, we introduce contrastive learning pretraining strategy that aligns the protein sequence representation with the text embeddings. Then, we leveraged instruction-basee fine-tuning to teach the decoder how condition on the aligned protein embeddings and generate accurate functional descriptions in natural language. Through extensive evaluation under challenging homology conditions, Prot2Text- V2 consistently outperformed traditional alignment methods, prior multimodal approaches, and general purpose LLMs on both lexical and semantic metrics. Our ablation studies further confirmed the importance of the contrastive alignment phase and the design choices in our architecture. We believe that Prot2Text-V V2 represents significant step toward scalable and flexible protein annotation, with applications in functional genomics, drug discovery, and synthetic biology.
general-purpose
10

---

7 Acknowledgments
We would like to thank the anonymous reviewers of the 39th NeurIPS conference for their valuable comments and constructive suggestions. We are also grateful to Dr. Johannes Lutzeyer for his assistance with the mathematical aspects during the rebuttal process. Our sincere thanks go to Dr. Lawrence P. Petalidis, Dr. Maria Dimitriadi, and Aron Vukatana for providing professional guidance on the subject and expert evaluation the results. This work was granted access to the HPC resources of IDRIS made available by GENCI, and the computations were further supported by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.
References
Hadi Abdine, Michail Chatzianastasis, Costas Bouyioukos, and Michalis Vazirgiannis. Prot2text: Multimodal protein's 's function generation with gnns and transformers. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 10757-10765, 2024.
Stephen Altschul, Warren Gish, Webb Miller, Eugene W Myers, and David Lipman. Basic local alignment search tool. Journal of molecular biology, 215(3):403-410, 1990.
Anthropic.
Introducing claude 3.5 sonnet.
claude- 5- sonnet, June 2024. Accessed: 2025-05-15.
Amos Bairoch and Rolf Apweiler. The swiss-pros pros protein sequence database and its supplement trembl in 2000. Nucleic acids research, (6(1):45-44 2000.
Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. Proteinbert: universal deep-learning model of protein sequence and function. Bioinformatics, 38(8):2102-211 2022.
Kunjie Fan, Yuanfang Guan, and Yan Zhang. Graph2go: a multi-modal attributed network embedding method for inferring protein functions. GigaScience, 98))giaa081), 2020.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415. 2016.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Sarah Hunter, Rolf Apweiler, Teresa Attwood, Amos Bairoch, Alex Bateman, David Binns, Peer Bork, Ujjwal Das, Louise Daugherty, Lauranne Duquenne, et al. Interpro: the integrative protein signature database. Nucleic acids research, ()(suppL[):D22111 D211-221,, 2009.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873);53--58) 2021.
Dongki Kim, Wonbin Lee, and Sung Ju Hwang. Mol-Ilama: Towards general understanding of molecules in large molecular language model. arXiv preprint arXiv:2502.13449, 2025.
Maxat Kulmanov and Robert Hochndorf. Deepgoplus: improved protein function prediction from sequence. Bioinformatics, 36(2):422-429, 2020.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 1244-:1234-22000 2020.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt tuning, arXiv preprint (tfh:200400899) 2021.

---

Zihao Li, Changkun Jiang, and Jianqiang Li. Deepgatgo: A hierarchical pretraining- graph- attention model for automatic protein function prediction. arXiv preprint arXiv:2307.13004 2023.
Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123- 1130, 2023.
Pengfei Liu, Yiming Ren, Jun Tao, and Zhixiang Ren. Git-mol: A multi-modal large language model for molecular science with graph, image, and text. Computers in biology and medicine, 171: 108073, 2024a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 16692, 2019.
Zhiyuan Liu, An Zhang, Hao Fei, Enzhi Zhang, Xiang Wang, Kenji Kawaguchi, and Tat-Seng Chua. ProtT3: Protein-to-text generation for text-based protein understanding. In Lun-Wet Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), Bangkok, Thailand, August 2024b. Association for Computational Linguistics.
Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Liuzhenghao Lv, Zongying Lin, Hao Li, Yuyang Liu, Jiaxi Cui, Calvin Yu-Chian Chen, Li Yuan, and Yonghong Tian. Prollama: A protein large language model for multi-task protein language processing. IEEE Transactions on Artificial Intelligence, 2025.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprini arXiv: 1807.03 03748, 2018
OpenAl. Gpt-4 technical report, 2023. URL https: http://////ppaaa..oooeeeeeeeeaee gpt-
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 31 3-1-118 2002.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8766 8763. PmLR, 2021.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. Advances in neural information processing systems, 32, 2019.
Dingyi Rong, Wenzhuo Zheng, Bozitao Zhong, Zhouhan Lin, Liang Hong, and Ning Liu. Au- toregressive enzyme function prediction with multi-scale multi-modality fusion. arXiv preprint (rXiv:2408.06391 2024.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. The semantic web: 15th international conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, proceedings 15, pages 593-607. Springer, 2018
Jin Su, Xibin Zhou, Xuting Zhang, and Fajie Yuan. Protrek: Navigating the protein universe through tri-modal contrastive learning. bioRxiv, pages 2024-05, 2024.
12

---

Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint
1.00085, 2022.
Serbülent Ünsal, Sinem Özdemir, Bünyamin Kasap. M Erşan Kalayeı, Kemal Turhan, Tunca Dogan, and Aybar C Acar. Multi-modal representation learning enables accurate protein function prediction in low-data setting. arXiv preprint arXiv:2412.08644 2024.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In Proceedings of the st International Conference on Neural Information Processing Systems. NIPS' 17, page 6309-6318 6318, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Michel van Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger. Foldseek: fast and accurate protein structure search. Biorxiv, pages 2022-02, 2022.
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, al. Alphafold protein structure datahase: massively expanding the structural coverage of protein-sequence space with high-aceuracy models. Nucleic acids research, 50(D1):D43--D D444, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Chao Wang, Hehe Fan, Ruijie Quan, and Yi Yang. Protchatgpt: Towards understanding proteins with large language models. arXiv preprint 2404:44002.09449 2024a.
Wenkang Wang, Yunyan Shuai, Min Zeng, Wei Fan, and Min Li. Dpfunc: accurately predicting pro- tein function via deep learning with domain-guided structure information. Nature Communications, 16(1):70, 2025.
Xinyou Wang, Zaixiang Zheng, Fei Ye. Dongyu Xue, Shujian Huang, and Quanquan Gu. Diffusion language models are versatile protein learners. arXiv preprint arXiv:2402. 18567, 2024b.
Zhuoyuan Wang, Jiacong Mi, Shan Lu, and Jieyue He, Multimodal-learinng for predicting molecular properties: framework based on image and graph structures. arXiv preprint arXiv:2311.16666 2023.
Kevin E Wu, Howard Chang, and James Zou. Proteinclip: enhancing protein language models with natural language. bioRxiv, pages 2024 2024.
Yijia Xiao, Edward Sun, Yiqiao Jin, Qifan Wang, and Wei Wang. Proteingpt: Multimodal Ilm for protein property prediction and structure understanding. arXiv preprint arXiv:2408. /163,, 2024.
Minghao Xu, Xinyu Yuan, Santiago Miret, and Jian Tang. Protst: Multi-modality learning of protein sequences and biomedical texts. In International Conference on Machine Learning, pages 38749-38767. PMLR, 2023.
Qianmu Yuan, Junjie Xie, Jiancong Xie, Huiying Zhao, and Yuedong Yang. Fast and accurate protein function prediction from sequence through pretrained language model and homology- label diffusion. Briefings in bioinformatics, 24(3):bbad 17, 2023.
Qianmu Yuan, Chong Tian, Yidong Song, Peihua Ou, Mingming Zhu, Huiying Zhao, and Yuedong Yang. Gpsfun: geometry-aware protein sequence function predictions with language models. Nucleic Acids Research, 52(W (WWWI):244-WW)
2024.
Jianrong Zhang, Hehe Fan, and Yi Yang. Protein captioning: Bridging the gap between protein sequences and natural languages. 2024.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinherger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09611 2019.
13

---

Jiangbin Zheng and Stan Z Li. Cepl: Conference on Pattern Recognition, pages 22-38. Springer. 2024.
contrastive protein learning. In International
Xibin Zhou, Chenchen Han, Yinggi Zhang, Jin Su, Kai Zhuang, Shiyu Jiang, Zichen Yuan, Wei Zheng, Fengyuan Dai, Yuyang Zhou, et al. Decoding the molecular language of proteins with evolla. bioRxiv, pages 2025-01, 2025.
NeurlPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. In both sections, we clearly introduce our core contribution: Prot2Text-V2. a novel multimodal sequence-to-tex model designed to generate free-form language descriptions of protein function directly from amino acid sequences. This central claim is reiterated and expanded in the summary of contributions within the introduction.
We support this claim throughout the paper with comprehensive empirical evidence, in- cluding multiple experimental setups, extensive ablation studies, and additional results in the appendix. Specifically, we demonstrate the effectiveness of our key innovation, Hybrid Sequence-level Contrastive Alignment Learning (H-SCALE). This technique im- proves cross cross-modal learning by aligning mean- and std-pooled protein embeddings with text representations using a contrastive loss framework.
The thcoretical formulation, experimental validation, and generalizability of our methods are all consistent with the claims made at the outset of the paper, thereby ensuring that the abstract and introduction faithfully represent the scope and contributions of the work.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We clearly document the main limitations of the proposed work in Section 5 In we point out that even though our model, through the experiments, is able generate protein function it is a limitation the lack of wet-lab experiments to be 100% certain of the predicted functions. In this sections we also address scalability and dataset concerns.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
Yes, the paper provides the full set of assumptions and complete and correct proof for the theoretical results it presents. Specifically, we further introduce the mathe- matical setting and formal justification for our key methodological Hontibutioon-Hybrid Sequence-level Contrastive Alignment Learning (H-SCALE)- in Appendix.
All relevant assumptions are clearly stated alongside our formal propositions, and the ac- companying derivations are included either in the main text (particularly in the Methodology section) or in the Appendix for completeness. The structure of our theoretical content fol- lows best practices: formulas are clearly numbered and referenced, and all novel components are derived with supporting rationale.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
14

---

Justification: To the best of our ability, we have fully disclosed all necessary details required reproduce the main experimental results. This includes comprehensive documentation of data set preprocessing steps, data splits, filtering criteria, model architecture, training protocols, hyperparameters, computational resources, and evaluation metrics and criteria. To ensure clarity and reproducibility, these details are presented in dedicated and clearly structured sections of the paper.
Additionally, we are providing our code via an anonymous link as part of the submission, and it will be made publicly available through GitHub repository upon publication. These efforts are intended to ensure that other researchers can faithfully reproduce our results and verify the main claims of the paper.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, we provide open access to both the code and data necessary to reproduce our main experimental results. The code currently available through GitHub repository at: htttps//giithb.. com/ CoolinoX/rrrotexxtttt contaainig a comprehensive README.mo file that outlines setup instructions, environment configuration, and step-by-step commands for running the experiments and reproducing the results.
The datasets used in work open-source and accessible. Scripts provided to reproduce all experiments related to our proposed method as well as the baselines.
6. Experimental setting/details
Question: Does the paper specify all the training and details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: the paper specifies all relevant training and testing details necessary to understand and reproduce the experimental results. We include comprehensive information data preprocessing, filtering, and splitting strategies, well as detailed descriptions all model hyperparameters, including learning rates, batch sizes, training epochs, and optimization algorithms. The rationale behind hyperparameter selection explained, and we state the type of optimizer used along with other relevant training configurations (e.g.. early stopping, learning rate schedules, etc.). These details are provided both in the main paper- primmrily in the Experimental Setup and Methodology sections and supplemented further in the Appendix for completeness.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We measure the statistical uncertainty of our evaluation metries using 2-sigma interval, representing 95.4% confidence. However, because many lexical and linguistic metrics are non-decomposable, standard methods based on per-sample variance do not apply here. Instead, we use resampling approach over sample groups-deeeee details are provided Appendix F
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
the paper provides detailed information on the compute resources used for all reported experiments. Specifically, we disclose the type hardware used seecldinn GPU type, number of GPUs, CPU specifications, memory capacity, and storage requirements. We
Justification:
15

---

also report the average training time per experiment and provide an estimate of the total compute required across all runs. We clarify whether experiments were run on an internal compute cluster or a cloud-based environment, and include the relevant configuration details needed for reproducibility. Additionally, we note that the total compute includes not only the experiments reported in the paper, but also accounts for preliminary runs and ablation studies used to refine our approach.
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
The research presented in this paper fully conforms to the NeurIPS Code of Ethics. We have carefully reviewed the ethical guidelines and ensured that work upholds principles of transparency, reproducibility, and fairness. Our study does not involve human subjects, personally identifiable information, sensitive data, and we use only publicly available and properly licensed datasets.
Justification:
We have made all efforts to ensure that our models and methodologies are presented with appropriate caveats regarding their limitations and potential misuse. Furthermore, we provide open access to code and data to promote reproducibility and responsible research practices.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, the paper discusses both the potential positive and negative socictal impacts of the work. In the main text, we highlight the positive impact of our model in accelerating the functional annotation of proteins, which can significantly aid biological research and drug discovery by leveraging tightthroughput sequence analysis.
We also address the limitations of our approach and acknowledge potential risks, such as incorrect or biased functional predictions, which could lead to downstream scientific or medical misinterpretatioos.. These considerations, along with important safeguards and mitigation strategies, are discussed in detail in the Appendix.
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: The paper describes safeguards put in place for the responsible release of our models and data. Given the potential risks associated with incorrect or biased protein function predictions, we emphasize the importance of cautious deployment and use. We include usage guidelines in the supplementary materials that outline appropriate contexts and limitations for applying our model.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
All existing assets used in this work-includee code, datasets, and pretrained models not developed by are properly credited to their original creators with appropriate citations included in the paper. We specify the versions used whenever applicable and provide URLs the original sources. All datasets leveraged are open-source and publicly available under licenses that permit their use in research, and these licenses and terms of use are explicitly acknowledged. We have ensured compliance with all relevant licensing requirements and terms of service.
Justification:
16

---

13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All new assets introduced in this paper- such as our Prot2Texx--V V2 model implementation and any associated datasets or scripts- are thoroughly documented and provided alongside the release. Documentation includes detailed descriptions of the model architecture, training procedures, hyperparameters, dataset preprocessing steps, and usage instructions. We include licensing information and discuss limitations to ensure responsible use. Since our assets do not involve data from human subjects, issues related consent do not apply. The assets are currently anonymized for submission and will be fully accessible upon publication via public repository with clear, structured documentation to facilitate reuse and reproducibility.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
This paper does involve any crowdsourcing experiments or research with human subjects. Therefore, instructions to participants, screenshots, and compensation details are not applicable.
Justification:
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve research with human subjects or crowdsourcing: therefore, Institutional Review Board (IRB) approval or equivalent was required or obtained.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMSs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Our work does not involve the use of LLMs in any aspect of the research. We did not employ LLMs for editing, understanding technical concepts, data processing, visualizing results, conducting experiments, proving theorems, or implementing methods. All contributions and analyses in this study conducted independently without the assistance of generative Al tools.
Justification:
17

---

A Experimental Details on H-SCALE
30
Ses Set BLEU-4
20
Prof2Teex-VV2 (w/ H-SCALE) Text2 (wlo CL)
12
16
Training Steps (k)
Figure 6: BLEU-4 scores the test plotted training steps plot for two Prot2Text-V V2 variants: with H-SCALE and without contrastive learning (w/o CL). These results are from supervised fine- tuning over approximately 48 hours. However, our H-SCALE contrastive learning converges just 10 hours on the same computing platform. The accelerated convergence in the second training stage offsets the cost of the initial alignment phase. Thus, the primary advantage of H-SCALE lies not just in training efficiency, but in its improved final performance. The comparison highlights the impact of H-SCALE on convergence speed, and its higher final performance.
For our experimental results, we begin by plotting the BLEU-4 score on the test set as a function of the number of training steps (Figure 6). In this figure, compare two variants: Prot2Text-V with H-SCALE, and Prot2Text-V V2 without contrastive learning (w/o CL). The graph illustrates that Prot2Text-V V2 with H-SCALE achieves higher BLEU-4 scores more rapidly, reaching mear-maximum performance by training step 4-well ahead of its counterpart. Despite the presence of a variance band, indicated by the shaded region, the H-SCALE model consistently outperforms the other. This demonstrates that aligning the protein encoder output with the semantic space of the pretrained language decoder significantly enhances the model's ability to generate accurate and fluent text, thereby validating the effectiveness of the H-SCALE mechanism.
Compared with the CLIP-based contrastive learning scheme, H-SCALE approach additionally aligns the standard deviation of sequences of embeddings from two different modalities. Our theoretical motivation is grounded in modeling the sequence of amino-acid token embeddings for given protein from an underlying multivariate distribution, and the description text token embeddings from another. Under the assumption that these distributions can be reasonably approximated as multivariate Gaussian, they are fully characterized by their first two moments: the mean vector and the covariance matrix. Thus, the source (protein) and target (text) distributions. Specifically, we align: (i) The first moment (the mean embedding vector); (ii) A diagonal approximation of the second moment (the element-wise standard deviation vector). This method serves as a computationally efficient proxy for matching the full covariance matrices, effectively aligning the two distributions via the method of moments. Even when the true distributions deviate from perfect Gaussian, aligning the second moment provides richer, more robust alignment than matching the mean alone, as also captures the feature dispersion. While aligning higher-order moments could offer further refinement, our approach represents principled trade-off between fine-tuning flexibility, computational tractability and alignment fidelity. It provides a lightweight yet effective mechanism for distribution-level. cross-modal bridging. To empirically validate this design, Table 3 we conduct ablation studies that isolate the contribution of mean and std pooling:
approach operationalizes this by aligning the empirical moments of
18

---

Table Ablation study of Prot2Text- V2 with different aligmment strategies. RBT and BBT denote Re-ranking BLEU and BioBERT F1 scores, respectively.
Ablations
Exet
B-2
B-4
R-1
R-2
R-L
RBT
BBT
w/o alignment
36.52
42.70
38.98
54.71
47.62
52.90
91.33
85.58
w/ pnoling alignment
37.01
44.39
41.23
55.30
48.27
53.62
91.61
86.19
w/ std-pooling alignment
33.42
37.28
33.78
50.49
43.22
48.75
90.93
84.70
w/ H-SCALE
39.16
46.67
43.34
57.24
50.17
55,39
91.95
86.81
B Computational Complexity
The computational complexity of Pro12Tex1-V2 arises primarily from its two-stage training process and its multimodal architecture, combining a protein sequence encoder (ESM2-3B), a modality projector, and decoder-only language model ----MA--..----I---. has a standard transformer architecture, therefore, it scales quadratically with the protein sequence length. The decoder-only LLaMA-3.188D architecture has 8.2 billion parameters and similarly scales quadratically with the sequence length. The computational complexity of the contrastive learning part similar, as the model performs forward and backward passes across the encoder and the decoder.
The ESM2-3B encoder
C Technical Details on SwissProt Dataset
The original human-curated UniProt-S Proosst-Swisrrrot dataset (Release 2022.02) contains more than 573,230 protein sequences and their corresponding function annotations, likely with many highly similar or redundant sequences, including isoforms, fragments, and repeated entries.
C.1 CD-HIT Clustering
To ensure robust generalization in protein function generation, created non-redundant train/vali- dation/test split based on amino acid sequence similarity using CD-HIT, a fast clustering algorithm for biological sequences. CD-HIT groups sequences into non-overlapping clusters based on user- defined sequence identity threshold. uses a greedy incremental clustering strategy, selecting the longest sequence cluster representative and assigning other sequences it if they pass the word-count-based similarity threshold, otherwise initializing a new cluster.
In the dataset used for training and evaluation in this study, protein sequences are clustered using CD-HIT to enforce maximum of 40% sequence identity (the minimum CD-HIT identity threshold for protein sequences) between training and validation/test sets. This design choice reflects real-world scenarios where functional models must generalize to novel sequences that differ substantially from those seen during training. The resulting split includes 248,312 proteins for training, 4,172 for validation and 4,203 for testing. We use this established split to allow direct comparison with prior work and to asses our model's ability to transfer knowledge beyond close homologs.
C.2 Dataset Statistics
During training and inference of Prot2Text-Ve V2, we truncate protein amino acid sequences to 1,024 residues, adhering to the maximum sequence length supported by the pretrained ESM2-3B module. Statistical analysis from Figure of our dataset confirms that the majority of protein sequences fall below this limit, ensuring that truncation has minimal impact model reliability.
For functional descriptions, we truncate labels 256 LLaMA-3.1 tokens to balance computational constraints and training efficiency. Since most human-curated UniProt labels are shorter than this threshold as visualized in Figure g we omit the end-of-sequence token for truncated cases, encouraging the model to generate longer responses when appropriate. During inference, we permit up to 512 new tokens to enable synthesis of multi-source descriptions, yielding more comprehensive functional predictions.
19

---

33.54% Pseudomonadota
Bacillati
8.72% Other Bacteria
SwissProt
18.21% Metazoa (Animala)
7.62% Viridiplantae (Green Plants)
6.02% Fungi
1.50% Eukaryota
3.86% Archaea
Figure 7: Taxonomic composition of the 248,312 non-redundant protein sequences from SwissProt after CD-HIT clustering. The dataset dominated by well-studied bacterial groups such as Pseu- domonadota (33.5%) and Bacillati (20.5%), which are common in experimental research. It also includes substantial representation from animals (18.2%), green plants (7.6%), and fungi (6.0%), ensuring coverage of key eukaryotic lineages. A smaller fraction comprises Archaea (3.9%) and other taxa, maintaining broad phylogenetic diversity while minimizing redundancy for efficient downstream analysis.
Length of Protein Sequences in Dataset
Length Text Descriptions Dataset
truncation=1024
truncation=255 256
12
Number of Proteins (k)
Number Proteins (k)
10
20
6
10
2
200
400
600
800
1000 1200 1400 1600
50
100 150
200
250
300
350
400
Number Residuals
Number Tokens
Figure 8: Left: Distribution of protein sequence lengths (in residues) in the dataset. Most sequences fall well below the 1,024-residue truncation limit used for ESM2-3B compatibility. Right: Distribu- tion of functional description lengths (in tokens). The vertical line marks the 256-token truncation threshold- while most labels are shorter, some require truncation (handled by omitting the EOS token encourage longer generations).
C.3 Homology Analysis
The CD-HIT clustering algorithm plays a critical role in our preprocessing pipeline by eliminating redundant protein sequences that could otherwise bias model evaluation. Without such filtering, an overrepresentation of highly similar proteins would artificially inflate performance metrics, as traditional methods can easily achieve high accuracy on strongly homologous sequences. To quantify clustering versus
this effect, we compared two data partitioning strategies: one using complete random shuffling. This comparison revealed substantial differences in homology distribu- tions between training and test sets. As visually supported by Figure 9 the clustered approach yields more meaningful evaluation conditions by ensuring test proteins maintain sufficient evolutionary distance from training examples. These results confirm that careful sequence clustering is essen- tial for obtaining reliable, generalizable assessments of model performance across diverse protein families. Further details these metrics can be found in Appendix D
20

---

BLAST BotScore
BLAST Identity (%)
0.7
Random Shuffle Clustering
Random Shuffie
0.6
5
CD-HIT Clustering
0.5
Test Split Density
Test Split Density
0.4
3
0.3
2
0.2
0.1
0.0
100
200
300
400
500
20
30
40
50
60
70
80
90
100
BLAST Alignment Coverage (%)
BLAST Alignment Length (Residuals)
16
0.25
Random Shuffle
Random Shuffle
14
CD-HIT Clustering
CD-HIT Clustering
Test Split Density (96
Test Split Density (%6
0.20
10
0.15
8
0.10
0.05
2-
0
0.00
20
40 60
60
100
200
400
600
800
1000
Figure 9: Comparison of homology distributions between CD-HIT clustering and random shuffligg approaches. Evolutionary distances between test samples and the training set quantified using four metries: BitScore. sequence identity, alignment coverage, and alignment length. The significant leftward shift distributions demonstrates CD-HIT's effectiveness in rebalancing the dataset toward lower-homology proteins, reducing evaluation bias.
D Protein Homology Assessment Using BLAST
To rigorously evaluate the evolutionary distance between test proteins and training set, employ the BLAST (Basic Local Alignment Search Tool) algorithm to compute pairwise alignment metrics for each test sequence against all training sequences. For each test protein, record several key metrics based on its highest-scoring match in the training set: BitScore, Expectation Value (E- (E-Value), Alignment Coverage, Alignment Length, and Sequence Identity.
D.1 Smith-Watermal Algorithm and Bit Score
The theoretical foundation for sequence alignment scoring is the Smith- Saith-Waterman (S-W) algorithm, which computes the optimal local alignment between two sequences using dynamic programming. The S-W alignment score is computed using a substitution matrix (e.g.. BLOSUM62) and affine gap penalties. However, the exact S-W algorithm computationally expensive.
BLAST approximates the S-W score using a fast heuristic, and reports a normalized score called the bit score. The bit score B transforms the raw alignment score S into standardized unit:
XS In K B
In 2
where:
21

---

S is the raw alignment score.
À and K are statistical parameters dependent on the scoring matrix and gap penalties, measured in bits.
The bit score allows for comparison across different BLAST runs and parameter settings, providing consistent metric for alignment quality.
D.2 E-value (Expectation Value)
The E-value represents the number of alignments with bit score B expected occur by chance in database search. It quantifies the statistical significance of an alignment:
= mn 2 2-
where:
m is the effective length of the query,
71 is the effective length of the database,
the bit score of the alignment.
Lower E-values indicate more statistically significant alignments. An E-value close 0 suggests the match is unlikely to have occurred by random chance.
D.3 Alignment Coverage
The coverage (or alignment length fraction) measures the fraction of the query sequence that is aligned to the subject sequence. If an alignment spans L residues of a query of length Q, then:
Coverage Q
Coverage reflects how much of the query is involved in the alignment and is important for distin- guishing between partial and full-length hits.
D.4 Sequence Identity
The identity score quantifies the fraction aligned positions in which the query and subject residues are exactly the same. If an alignment of length L contains / identical amino acids, the identity score is:
1 L
Identity
Identity provides simple measure of sequence similarity, but does not account for conservative substitutions, which are captured by the substitution matrix in the bit score.
E Uncertainty Estimation of Non-Decomposable Evaluation Metrics
In many machine learning evaluation scenarios caaunioo-ppatiuullaryy in natural language inneration-mettie such BLEU, ROUGE, and BERTScore are widely used to assess the quality of predictions. Unlike simple metrics such as accuracy or mean squared error, these scores are not decomposable: they depend aggregate statistics across the entire set of predictions. As result, computing the metric on individual samples and averaging the results does not yield the same value as computing the metric over the corpus whole.
This poses challenge when attempting to estimate statistical uncertainty, such as constructing confidence intervals. Standard techniques that rely per-sample variance invalid in this setting.
22

---

Instead, a resampling- based approach over groups of samples must be used. The method presented below estimates the variance and confidence interval of such non-decomposable scores using group- level evaluation, enabling statistically valid uncertainty estimation with minimal computational overhead.
Let (2,, J be a set of 2n samples with corresponding system outputs and references. Assume we are able to compute the evaluation metric (e.g., BLEU) only at the group level due its non-decomposable nature or computational cost.
We partition the 2n samples into disjoint groups, each of size computed over group be denoted by:
Let the evaluation metric
$;
Metrie(Group,). for i = 1, ..... k.
Let the average of the group-level scores be:
34-
The sample variance of the metric across groups serves as estimate of the uncertainty in the overall evaluation:
0²_
8
8)²
i=1
We use the grand mean E as a point estimate of the population mean μc:
T.
Assuming approximate normality of the group-mean estimator (justified by the Central Limit Theorem for sufficiently large k),, 2-sigma confidence interval for the metric is given by:
-2.0 8 2.
This interval quantifies the uncertainty of the evaluation score under random resampling of the data and accounts for the non-deoomposability of the metric.
F Protein Function Prediction with Additional Text Fields
During training, we allow users to optionally include two additional text fields: the protein name and its taxonomy. This enables the model support generation in cases where such information is available. To accommodate this, we incorporate these fields into the training process with an 80% dropout rate, ensuring the model remains robust in this scenario. In main text although this training is conducted our test set had 100% dropout to guarantee that we will remain fair in all the previously presented scenarios.
Table/Blad Figure 10 we present additional experiments in which the protein name is included. We hypothesize that protein names offer valuable semantic cues that can improve functional prediction, especially when aligned with the LLaMA decoder's 's latent representations. In Table we evaluate Roo2TTxtt-V2+ (w/ (-SCALE)-wheet several general-purpor LLMs: LLaMA 3.1, Claude 3.5 Sonnet, and GPT-4o-mini. Each these models is prompted with both the amino acid sequence and protein name, under zero-shot and five-shot settings, without access to external retrieval. As shown in Figure 10, the inclusion of these additional fields significantly enhance function prediction performance. Nevertheless, in semantic context, the original Prot2Text- V2 metrics remain comparable, demonstrating competitive performance without the added fields.
the '+' indicates inclusion of the protein name name-against
possible to see in T that even the scenario which these extra fields are available our model surpasses all models among all of the presented metrics.
23

---

BLEV-2 F1 Score (%)
BLEU-4 F1 Score (9)
Rougel FL Score (N)
50
45
16
23-
V2
Protz No0
Protz NoV
Protz Nost
10,80)
(9:25)
as)
(4044))
(4,30)
(B540)
(60
(8.70)
16,00)
mes4g)
(40-00/
(0530)
09,30)
(Bcm))
10,00
(25-0)
451
(45.00)
RoBERT Score (%)
BioBERT Score (%)
GPT-4o Score (100)
02
Prot2 N24
Pral No4-
15-
Protz MAXE
Prat2
Protz 2ra
(R.00)
geaet
(5445)
(4,00)
massy
(3
(6009)
(P.55)
(S545))
(9.29)
(8025)
Inasi
(2.39)
(45
BiScore BLAST Similantt
Figure 10: Evaluation of protein function descriptions across BLAST sequence bitscore bins (%).
MoM2Text-V2+ which includes the protein name input, achieves the best performance.
Table 6: Comparative evaluation of Prot2Text-V24 (w/ H-SCALE), which incorporates the protein name input, against general-pue pprrpse LLMs including LLaMA 3.1, Claude 3.5 Sonnet, and GPT-4o- mini. Results highlight the advantage of domain-specific modeling when protein context is explicitly provided.
Model
Exct
B-2
B-4
R-1
R-2
R-L
RBT
BBT
GPT
LLaMA-3.1 (0-shot)
7.63
2,80
23.83
7.76
18.89
85.84
77.22
45.03
LLaMA-3.1 (5-shot)
7.99
3.18
25.60
8.97
20.55
86.40
77.74
44.71
Claude-3.5-Sonnel (0-shot)
9.78
4.22
25.98
9.28
20.04
86.02
79.23
Claude-3.5-Sonnee (1-shot)
15.72
7.79
28.24
11.76
21.70
86.56
79.71
GPT-4o-mini (0-shot)
4.79
1.18
2.79
10.32
82.69
73.46
GPT-4o-mini (1-shot)
8.53
2.46
19.04
4.39
14.41
84.24
73,96
Prot2Text-Ve V2 (w/ H-SCALE)
39.16
46.67
43.34
57.24
50.17
55.39
91.95
86.81
64.58
Prot2Text-V22 (w/ H-SCALE)
43.85
55.64
52.46
63.46
57.03
61.70
93.17
89.01
71.29
24

---

G Qualitative Evaluation
Table 7 presents qualitative examples of predictions generated by Prot2Text- V2. The table includes the original UniProt description for each selected protein, serving as the reference label. Alongside this, we show the corresponding prediction produced by Prot2Text- V2, the score assigned by the LLM-as-a-judge evaluation framework, and the explanation provided for that score.
We selected seven proteins from the target group characterized by low BLAST bitscore, specifically within the 0-50 range. In this challenging weak weak-homology setting, our model generally captures the core concepts conveyed in the original UniProt descriptions, which serve as the reference labels. While the predictions may not replicate every detail, they often reflect the correct functional context, For samples receiving lower LLM-as---jude over-specification or the introduction of overly confident assertions, likely due to the model's attempt to compensate for limited similarity- -aased cues. On the other hand, predictions with higher judge scores tend to miss only minor detail information mentioned in the reference, but omitted in the output, while still maintaining accurate and coherent functional descriptions.
scores (c.g., 5 or 6), we observe occasional issues with
From the qualitative results, we observe that Prot2Text-V2 is capable of generating function de- scriptions that align with the referenced UniProt annotations, even in challenging weak-homology scenarios (BLAST BitScore under 50). This suggests that the model is not simply memorizing known examples or relying on close sequence neighbors, but is instead learning generalize functional patterns from the input modalities.
Table 7: Qualitative examples of protein function predictions generated by Prot2Text- V2 for proteins with low BLAST identity scores (0-30%). We show the reference UniProt description, the model's prediction, the explanation for what is considered wrong and right in the prediction. and the LLM- as-a-judge score for it based on the explanation. These examples illustrate the model's ability to generalize functional information even in low- now-homology settings.
Protein
UniProt Description
ProtTText_ V2 Description
Judge Explanation
GPT Score
Could play role in the
The prediction captures general function related to virulence, lacks specificity oo inflammatory mediation and cmits H. pylori.
LPP20 HELPI
pathogenesis of H. pylori by serving as an inflammatory mediator.
May be involved in virulence.
5
Structural component of flagellum; part the rod stricture of the flagellar basal body.
Describes the rod central, he- lical, flexible structure in the basal body transmitting torque.
Captures general relevance to flagellar ap- paratus, but omits specific motility role and lacks clarity on structural function.
FL.B.BACCE BACSU
Part of the RNAi important for heterochro- matin formation and chm- mosome segregation; mem- ber of RITS complex. Immunity component of I LXG toxin-immant immmmity mod- ule: neutralizes KNase activ- ity 0920.
Involved in RNAi puthway as transcriptional repressor, scrihes dsRNA generation and in- teraction with RNase HI Dicer- like enzymer
Touches on RNAi and dsRNA roles, but inac- curately attributes siRNA source and omits key roles in heterochromaaii and chrome- some dynamics.
SCHIS
Deseribes LXG module involve- ment, kin selecte selection, hiofilm com- petition: neutralizes BC_ DC_0922 toxin in
Captures immunity concept well, but intro- duces unrelated toxin (BC. (BC_0922) and extra context original description.
ATOXL BACCR
6
Anti-sigma factor for BCF sigma factor SigX; inhibits and stabilizes SigX.
Negative regalator of sigma-X; impacts xerCD operan taanerip- tion and protein production.
Deseribes inhibition well but omits ECF text and stubilization; partial view of func- tional implications.
RSIX BACSU
7
Inactivates chloramppenicol transfrrring gamma- phosphate of XTP to C-3' hydroxyl group.
Phosphorylates 3-hydroxyl group of phanolic ring to inactivate chilo- ramphenicol.
Matches key function but less specific about ATP and hydroxyl group position; mostly accurate. Monooxygenase in sys- tem; catalyzes epoxidation: can use altemative substrates; prefers NADPH.
CPT STRVP
Catalytic subunit of enzyme system reducing PA-COA to form sposyphenylacetyy CoA, incorporates one gen atom.
Highly accurate and informative; slight wording variation from ground truth, oth- erwise excellent.
PMA_ECOLI
G.1 Human Expert Evaluation
To further validate our predictions, we compared our model's protein function predictions to those of eight widely used and referenced large language models (LLMs), all based solely on sequence input without internet access. To assess the quality of these predictions, we asked two biology experts to independently and blindly evaluate each model's predictions for nine well-known proteins in the field.
25

---

Score scale: Throughout this qualitative verification we adopted an evaluation scale from 1 to 5 and asked them to evaluate the correctness and completeness of these generated descriptions based on their domain knowledge and familiarity with these protein sequences. The rating of each model's output should be done according to how well it could reflect the known or probable function of the respective protein. We further describe our scale as:
Score = for completely inaccurate generations. The generated description is biologically implausible entirely unrelated to the known function of the protein.
Score = 2 for poorly accurate generations. The description contains significant inaccuracies or misinterpretations, with little relevance to the known function.
Score = 3 for moderately accurate generations. The description is partially correct but includes vague, generie, or somewhat misleading elements.
Score = 4 for mostly accurate generations. The output is largely consistent with the known function, with only minor errors or omissions.
Score = for highly accurate generations. The description is biologically sound, detailed, and aligns closely with the known or expected function of the protein.
LLM models evaluated: In this qualitative assessment, we evaluate a selection of widely used, ceneral-purpost generative LLMs: GPT-4, Grok, Claude 3.7 Sonnet, Gemini 2, LLaMA 4, and Mistral 7B. These models were chosen due to their multi-domain capabilities, and their capacity to generate coherent, context-aware responses from textual input. Although not explicitly trained in protein-specific data, such general-purpose LLMS have demonstrated aptitude in biological tasks because of their exposure to large-scale textual corpora, including protein databases with stored descriptions. Evaluating them in this context allows us to assess their zero-shot generalization ability reasoning alone can approximate biological insight.
and measure how well language-drinn
To contextualize the performance of our proposed model, Prot2Text- V2, we also include comparisons with prior protein function generation approaches: Prot2Text, which introduced protein-to-tex generation using encoder-decoder architectures; and Evolla-10B, 10B, a domain-specific model trained on dataset approximately 200 times larger than the one we currently use, designed for structured biological knowledge decoding. While we do not aim to directly compete with a model trained on a dataset much larger then ours, like Evolla-10B, including them in our evaluation provides valuable insight into the effectiveness and efficiency of Prot2Text-V2 under significantly more constrained training resources, highlighting its potential in low-resource scalable deployment scenarios. This comparative framework enables us to benchmark Prot2Text-V V2 not only against domain-specite baselines but also against the best general-purpont reasoning engines currently available.
Human expert assessment: According to the experts, some of the models provided highly accurate and detailed descriptions, receiving top scores (4 or 5). However, several others were either not relevant to the protein in question or included explanations that were unclear and difficult to follow. Notably, none of the submissions addressed species-speciffe differences, which understandable given the high degree of amino acid conservation across species from humans dog, cow, yeast, ctc. It is worth noting that, according to them, the degree of detail the LLMs returned was strikingly different, and in some cases, the models generated extensive information on residues phosphorylated, binding partners and other molecular/cellular details. It was with such details that the assessors spent most time as there were instances of incorrect information that created concern overall output validity and trustworthiness.
The resulting expert ratings are presented in Table g The blinded biological assessment of LLM outputs showed good correlation between the two assessors. Based on the average scores, our model, Pr0,2Text-V2, clearly outperforms most general-purpose LLMs, reinforcing the value of domain-specific fine-tuning even when using relatively small dataset. Although Evolla- 10B achieves higher scores, as expected from a large-scale model trained on a dataset with 200 times more proteins than what we originally trained our model with, this comparison should interpreted with caution. Evolla-10B and Claude 3.7 benefit from access to an extensive training set, making it likely that evaluated proteins or closely related ones were seen during training. They are as well capable of predicting, from their extensive training database, the names of the protein sequences we use, which facilitates in generating the prediction. In contrast, Prot2Text-V2
Results and discussion:
26

---

Table 8: Human annotations from biology experts (A and B) for nine highly understood proteins across selected models. The score scales are defined as: (1) Completely inaccurate; (2) Poor accuracy; (3) Moderately accurate; (4) Mostly accurate; and (5) Highly accurate. The models used in this comparison were: BLAST (BST); O- (GPT4); Grok-3 (GRK); Claude-3. 3.-oont (CLD3)); Google-Gemini-2 (GM2); Meta-LLaMA-4 (LMM, Mistral-7B (MST); Prot2Text (PTV); our Pro12Text-V2 (PTV2); Pro[T3 (PTT3); and Evolla-10B (EVL)
Protein
BST GPT4 GRK CLD3
GM2 LM MST PTV PTV2
PTT3
EVL
Evaluator
A
B
A
B
A
B
A
B
AB
B
A
B
A
B
A
B
A
B
4
1
2
3
2
2
2
1
11
2
1
5
5
1
1
B2RTY4MYO9A H
2
1
4
4
4
1
3
3
2
4
4
3
3
5
3
Q29537-P53_C C
5
5
3
3
3
5
4
3
Q32KY4-CDK4 B
5
3
3
4
3
1
1
4
3
4
5
5
4
JOCY46-EGFR.
1
5
5
4
3
5
5
4
4
3
5
3
2
2
2
5
3
P04806-HXKA_
5
2
5
2
4
4
5
5
014929.HATI
2
3
1
2
3
3
2
2
1
2
1
2
5
P00846-ATP6 H
4
2
2
2
5
5
4
2
3
2
4
5
4
4
4
5
5
5
P06213-INSR H
1
5
5
4
5
2
4
4
1
1
3
Avg Scores model 3.22
2.55
2.88
3.77
2.55
2.60
1.94
3.38
3.61
3.06
4.21
Table 9: Pearson correlation coefficient between automatic evaluation metrics and human opinions.
Metric
B-2
B-4
R-L
RBT
BBT
GPT
Human 0.63
0.61
0.59
0.55
0.68
0.73
was trained much smaller and more curated dataset, and yet achieves scores within point of Evolla. This demonstrates the efficiency and generalization capability of our model, suggesting that with modest scaling, Prot2Text-VV V2 has the potential to match even surpass the performance of resource-intensive significantly larger and more resource- intensive models. We also computed, in Table 9 the Pearson correlation of the metries from Table with the expert evaluations. The Pearson correlations with expert evaluations show that while BLEU and ROUGE have moderate alignment ( 0.60), BioBERT (0.68) and GPT score (0.73) correlate more strongly, likely due to their deeper semantic understanding in the biological domain.
H LLM as Judge
Although there are studies that use L.LMs as judges in experiments, to our knowledge, no definitive or universally accepted protocol has been established for this approach. In many cases, research that employs LLMs as evaluators does not clearly specify the prompts or design choices used. This lack of standardized protocol poses several challenges, including issues with reproducibilityy transparency, the ability to compare models, and ensuring the validity of evaluations. We describe our protocol below.
The configurations for this experiment involved using GPT-4-mini as the LLM judge. In addition, to minimize variability in the responses, we the temperature O. making the output deterministic. This ensures that the model produces the same result every time for the same input, reducing the possibility of diverse outlier predictions that could introduce noise into the evaluation. Finally, also developed a prompt to compare our predictions for protein function and their respective labels, shown bellow:
27

---

Prompt:
will provide you with three pieces of text:
Protein name: This is the name of the protein.
Ground Truth Description: This is the curated, authoritative description of
protein's function (sourced from UniProt).
Predicted Description: This an automated prediction of the protein's 's function.
Task:
Evaluate how well the predicted description matches the ground truth description. Your evaluation
should focus on:
Check if the prediction includes the essential keywords and concepts present in the ground truth.
Key Functional Terms:
Specificity: Assess whether the prediction captures the specific details (e.g., catalytic activity, binding sites, cellular localization) that are mentioned in the ground truth
Completeness: Determine if the prediction covers all major aspects of the protein function described in the ground truth.
Evaluate how accurately the prediction reflects the true function as described, including any relevant mechanisms or contextual details.
Accuracy:
Scoring Guidelines:
Provide final score scale from 0 to 10, where:
No similarity relevant matching elements between the prediction and the ground truth.
0:
10: A perfect match with all key elements of the ground truth correctly captured.
Output Format:
Your output must include:
A single line with the score, formatted as:
Score:
(where X is an integer between 0 and 10) Another line with brief explanation summarizing the reasoning behind the score.
Explanation: This explanation should highlight the strengths and weaknesses of the prediction relative to the ground truth.
OpenAI GPT-4o-mini user prompt instructions for LLMM-as---jdde evaluation. Name, labeled description and prediction description of every protein are be provided afterwards.
Figure 11:
This experiment aimed to evaluate the predictions of Blast, Prot2Text, and the variations Prot2Text- V2. The results for various Blast similarity scores are shown in Figure From this, that all proposed variations outperform Blast low similarity scenarios.
I Safeguards
Prot2Text- V2 demonstrates strong potential for protein function prediction, though we cannot rule out the possibility of occasional inaccuracies, particularly with novel or poorly characterized proteins, so results should be interpreted with care and verified where possible. While the model is designed for beneficial research applications, we encourage its use in responsible and well-regulated settings ensure alignment with ethical and safety standards.
28


Reviews:
[{"review_id": "BT6fMlSodA", "reviewer": "Reviewer_KRBB", "summary": "The paper tackles the task of protein captioning - given an amino acid sequence, predict a textual description with the hypothesized protein function. The paper proposes a new model which leverages a protein encoder and an LLM decoder linked via a modality projector. The model is trained via a two step training process: a contrastive learning step aimed to align the latent representations of protein sequences and of textual descriptions (H-SCALE), and a fine-tuning step aimed to guide the LLM decoder to produce protein textual descriptions. The approach is evaluated via multiple evaluation metrics on held out protein subsets of various sequence similarity to the training set, and compared to baselines to show improved results across held out sets and metrics. The claimed contributions are 1) the overall Prot2Text-V2 model, 2) the H-SCALE contrastive alignment training procedure, 3) the approach’s strong performance compared to baselines in the low homology regime.", "questions": "H-SCALE\n* From what I understand, the ProtT3 baseline the paper mentions is also using an ESM encoder with an LLM decoder, a two step training process with a contrastive learning step followed with a protein captioning training step. The major difference flagged in the paper is Q-Former vs. H-SCALE. Can the differences in performance reported in Table 2 be attributed primarily to Q-Former vs. H-SCALE or are there other differences between the two approaches as well?\n* H-SCALE is posed as a new contrastive learning scheme that also leverages standard deviation and intermediate representations. Have you considered quantifying how much these choices impact alignment quality (in the style of Figure 4) or downstream performance when using sequence embeddings only?\n* Figure 6 highlights how the model that was trained with H-SCALE converges faster (and to a higher final performance) as a function of the number of training steps performed. Does this account for the steps performed during contrastive learning?\n\nOverall performance\n* Can you please explain the difference between the BLEU-2 (B-2) score in Table 1 and the BLEU-2 F1 Score (%) in Figure 5? What made you choose different metrics in Figure 5? Also, what made you exclude the exact match performance in Figure 6?\n* Are all the baselines included in the main text included in the human expert evaluation? I did not see BLAST and ProtT3. Did you find the human expert assessments to be correlated with the metrics used in the main text?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The paper tackles the task of annotating protein sequences with textual descriptions which is an interesting problem in protein modeling. The proposed approach (pretrained protein encoder, pretrained llm decoder, modality projector, two-step training approach, H-SCALE) is clearly described and the authors add intuitions throughout the text for various architectural choices\n- The overall approach shows improved performance across several evaluation settings including held out protein subsets of various sequence similarity to the training set and multiple evaluation metrics\n\nWeaknesses:\n\nOverall, the originality and significance seems modest and I recommend a borderline accept due to the improved results. In particular, the task of protein captioning and the train-test splits used are established in cited prior work (Prot2Text, ProtT3) and the paper proposes a new approach that improves performance across metrics:\n- Approach: The paper notes the similarity to ProtT3 and highlights the H-SCALE contrastive learning approach as the notable difference. This seems like an incremental improvement to me, but I asked a few questions below that could help me understand the significance of this contribution more and I will reassess this after rebuttal\n- Experimental setting: I think the paper's originality could be improved by describing whether the analysis/metrics are newly introduced or mimic Prot2Text.\n- Results: I think the paper's significance could be improved by providing more insights into the results, e.g. discussing the discrepancy between metrics in Figure 5, whether the metrics highlighted in the main text correlate to the human expert evaluation described in the appendix, etc.", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer: \n\nThank you for your thorough review and positive feedback on our work. We appreciate that you found our intuition well explained, implementation clearly described, the protein captioning task meaningful, and our results showing improved performance to be a key strength. We are grateful for your insightful questions, which greatly helped us identify areas to strengthen in our manuscript. We address your specific concerns below. \n\n**[Weakness-1 (W1) and Question-1 (Q1) on Novelty]** \n\nWhile we build upon established components like ESM and LLMs, we believe that our proposed model Prot2Text-V2 has significant and novel methodological improvements over existing multimodal LLM designs. While there are some similarities with ProtT3 architecture, our model departs in the following essential respects:\n\n* ProtT3 relies on a Q-Former, an additional attention module with learnable queries of a fixed number and a redundant text encoder, to align the protein and text modalities. In contrast, Prot2Text-V2 follows a decoder-only design with a lightweight projector and with a parameter-free alignment method. \n\n* Rather than learning alignment implicitly through a query-attention module, H-SCALE directly aligns the projected protein representations to intermediate text embeddings with a new contrastive learning objective. This eliminates the learned bottleneck, preserves the full expressivity of the pretrained protein encoder, and avoids added parameters, latency, and memory, and also allows us to use a more powerful LLM decoder.\n\n* We also want to emphasize that contrastive learning from intermediate LLM embeddings is not trivial, and it has not been done in any prior work, since all previous methods use an additional text encoder. \n\n**[W2 on Originality]**\n\nThank you for raising this important point. While our core evaluation protocol is compatible with prior work (e.g., BLEU, ROUGE etc), we extend it with key additions that introduce more biologically meaningful semantic evaluation signals, going beyond what was used in Prot2Text or ProtT3.\n\nFirst, we applied additional evaluations and analyses:\n\n* LLM-as-a-Judge: Protein function descriptions can be complex and semantically rich as models can produce valid but lexically novel captions. By leveraging LLMs generalization and domain knowledge, we can evaluate semantic fidelity and biological coherence. We use GPT-4 as a judge, rating the fluency and the functional correctness of the predictions.\n\n* Human Expert Evaluation: We conducted a systematic human evaluation involving domain experts who assessed model outputs along similar dimensions (fluency, function, plausibility). The evaluation is conducted blindly across the multiple models. \n\nSecond, we introduced a novel bitscore-based evaluation benchmark for low-homology proteins, which divides highly dissimilar proteins into finer-grained groups than previous approaches. Unlike identity scores, which only capture raw sequence similarity, bitscore reflects the statistical significance of alignments, incorporating both alignment quality and length. This makes it more sensitive and informative for detecting remote homologs, enabling more precise and meaningful evaluation. In contrast, identity scores often fail to distinguish between weak true homology and random noise, especially in low-similarity regions.\n\nThird, we further investigated the performance of the prediction model when the name and taxonomical information of the protein is provided. The analysis in Appendix F shows that including names boosts performance, probing the upper bound of the model’s capacity with additional information. \n\n**[W3 on Result Insights]**\n\nEven though there are small discrepancies in the metrics, we still observe a consistent trend, which shows that our approach outperforms the baselines on the low-homology settings across all the metrics.  Regarding the small discrepancies observed in Figure 5, it likely stems from the varying sensitivity of the metrics to different aspects of model performance (precision vs. coverage). The thresholds (where the retrieval-based BLAST predictor begins to outperform our model) on semantic metrics are generally higher than on lexical ones, as shown in Figure 5. This suggests that our LLM-based model is capable of generating accurate descriptions, albeit in a different style than the ground truth annotations, capturing the correct semantics while exhibiting variability in lexical choices. We observed a strong alignment between GPT-as-a-judge scores and human expert evaluations, suggesting that both metrics can better handle the diversity and paraphrasing, measuring the true performance of the model beyond surface-level similarity. We appreciate the suggestion and will incorporate this analysis into the final version, along with an expanded discussion of Figure 5 to improve clarity and aid interpretation.\n\n**[Q2 on Alignment Quality]**\n\nYou raised an excellent point about comparing the impact of different alignment techniques on the model. Our motivation is that the average pooling captures the global semantic similarity, whilst the standard deviation pooling (a diagonal approximation to the covariance) captures the local variation and dispersion. Unfortunately, different alignment techniques are designed with different training losses, and a direct quantitative comparison is hardly feasible or meaningful. \n\nHowever, we can observe the impact of each design choice on our downstream protein function prediction task. Based on your insightful suggestion, we performed an ablation study comparing H-SCALE with various baselines. We present the results below:\n\nTable 1: Ablation study of Prot2Text-V2 with Different Contrastive Learning Schemes\n\n| **Ablations** | **Exct** | **B-2** | **B-4** | **R-1** | **R-2** | **R-L** | **RBT** | **BBT** |\n|---|---|---|---|---|---|---|---|---|\n| w/o alignment | 36.52 | 42.70 | 38.98 | 54.71 | 47.62 | 52.90 | 91.33 | 85.58 |\n| w/ mean-pooling alignment | 37.01 | 44.39 | 41.23 | 55.30 | 48.27 | 53.62 | 91.61 | 86.19 |\n| w/ std-pooling alignment | 33.42 | 37.28 | 33.78 | 50.49 | 43.22 | 48.75 | 90.93 | 84.70 |\n| w/ H-SCALE | 39.16 | 46.67 | 43.34 | 57.24 | 50.17 | 55.39 | 91.95 | 86.81 |\n\n**[Q3 on Convergence Speed]**\n\nFigure 6 does not include the preliminary alignment phase but only presents the results from supervised fine-tuning over approximately 48 hours. However, our H-SCALE contrastive learning converges in just 10 hours on the same computing platform. The accelerated convergence in the second training stage offsets the cost of the initial alignment phase. We will make sure to clarify this in the final version of the paper. That said, the primary advantage of H-SCALE lies not just in training efficiency, but in its improved final performance,\n\n**[Q4 on Evaluation Metrics]**\n\n> “the BLEU-2 (B-2) score in Table 1 and the BLEU-2 F1 Score (%) in Figure 5”\n\nThank you for pointing it out. All previously reported BLEU-2 values should be interpreted as the BLEU-2 F1 score, and all the metrics are the same in Table 1 and Figure 5. We will correct it in the final version. \n\nAdditionally, for Figure 6, although exact match ratio, rouge scores etc are valuable to properly evaluate the performance, we consider Bleu-4 F1 score as a direct, simple, and comprehensive way to trace the evolution of the training process. The exact match ratio can be too harsh on minor variations and is sensitive to punctuation or formatting, and thus cannot properly trace the evolution. However, for completeness, we will include all the metrics in the Appendix for the camera-ready version. \n\n**[Q5 on Human Evaluation]**\n\n> “baselines included in the main text included in the human expert evaluation”\n\nWe are very grateful for your suggestion. We asked human expert evaluators to complete the evaluation matrix and we have the results from one of them until now. We expect to have the results from the second reviewer in the discussion phase. We present the results below, where Prot2Text-V2 (3.61) scored higher than ProtT3 (2.78) and BLAST (3.22) in expert evaluation. This reinforces that our model generates more semantically and biologically meaningful captions than both template-based and generative baselines.\n\nTable 2: Human expert evaluation on ProtT3 and BLAST retrieval-based predictor\n\n| Protein\\Model | PTT3 | BLAST |\n|---|---|---|\n| Q9H2D6 | 1 | 4 |\n| B2RTY4 | 3 | 2 |\n| Q29537 | 4 | 5 |\n| Q32KY4 | 4 | 5 |\n| P00533 | 2 | 1 |\n| P04806 | 4 | 5 |\n| O14929 | 2 | 2 |\n| P00846 | 4 | 4 |\n| P06213 | 1 | 1 |\n| **Avg Score** | **2.78** | **3.22** |\n\nAlso, based on your insightful suggestion, we computed the Pearson correlation of the metrics from Table 1 with the expert evaluations. The Pearson correlations with expert evaluations show that while BLEU and ROUGE have moderate alignment (~0.60), BioBERT (0.68) and GPT score (0.73) correlate more strongly, likely due to their deeper semantic understanding in the biological domain.\n\nTable 3: Pearson correlation coefficient among automatic and human evaluation metrics\n\n| Metric  | Bleu2 | Bleu4 | Rouge | Roberta | BioBert | GPT |\n|---|---|---|---|---|---|---|\n| **Human** | 0.63 | 0.61 | 0.59 | 0.55 | 0.68 | 0.73 |"}, {"review_id": "WYVHkvVCQ3", "reviewer": "Reviewer_6AtP", "summary": "The paper introduces Prot2Text-V2, a multimodal framework for generating natural language descriptions of protein functions directly from amino acid sequences. Existing methods rely on structured ontologies or homology, limiting flexibility and generalization to novel proteins. Prot2Text-V2 addresses this by combining the ESM-3B protein language model encoder with an LLaMA-3.1-8B-Instruct decoder, connected via a nonlinear modality projector. A key innovation is Hybrid Sequence-level Contrastive Alignment Learning (H-SCALE), which aligns protein and text embeddings using mean and standard deviation pooling to enhance cross-modal consistency. After contrastive alignment, the model is fine-tuned with instruction-based LoRA to generate functional descriptions. Experiments on 250K SwissProt entries show Prot2Text-V2 outperforms baselines in low-homology scenarios, with improvements in lexical (e.g., BLEU-4: 43.34%) and semantic metrics (e.g., GPT score: 64.58).", "questions": "what kind of wet-lab validation experiments of selected proteins  do you want to do?", "limitations": "Yes. They have.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nH-SCALE Contrastive Learning: Aligns protein and text embeddings using global and local features, improving semantic consistency.\n\nMultimodal Architecture: Bridges protein sequences and natural language via ESM-3B and LLaMA-3.1, enabling free-form functional annotation.\n\nWeakeness:\n\nTraining requires 8x A100 GPUs and 50 hours, restricting accessibility for small labs. LLM decoder (8B parameters) increases inference latency compared to lighter models.", "was_revised": true, "final_justification": "The authors give the extra results with various model sizes.", "rebuttal": "Dear reviewer: \n\nThank you for your positive and encouraging review of our paper. We are delighted that you recognized the strengths of our work, particularly the Prot2Text-V2 multimodal architecture and the H-SCALE contrastive learning method. \nWe address your questions regarding the computational complexity and the wet lab experiments below: \n\n**[Weakness on Computational Complexity]**\n\nWe recognize that the training cost (8× A100 GPUs for ~50 hours) could be a hurdle for smaller labs, but it is still within a reasonable range compared to other standard LLMs that require thousands of GPU hours. While our current focus has been on demonstrating the feasibility and performance of protein captioning using large-scale models, model efficiency is an important future direction. To that end, based on your insightful comment, we trained smaller variations of our model and we aim to release the pretrained checkpoints to support reproducibility and allow researchers to build upon our work without retraining from scratch.\n\nWe present the results of the smaller models in the table below. These models use a reduced decoder size and a lighter backbone encoder, making them more suitable for resource-constrained settings. \nDespite their smaller size, they achieve competitive performance and outperform the other baseline methods. Nonetheless, our largest model yields the best overall results.\n\nTable 1: Ablation results with various model sizes\n\n| Prot2Text-V2 ablations | Exct | B-2 | B-4 | R-1 | R-2 | R-L | RBT | BBT | latency |\n|---|---|---|---|---|---|---|---|---|---|\n| Prot2Text (898M) | 32.05 | 26.26 | 32.45 | 50.49 | 42.60 | 48.33 | 90.56 | 84.26 | - |\n| ESM-150M + LLaMA-3.2-1B | 33.25 | 40.61 | 37.26 | 54.69 | 47.12 | 52.93 | 90.99 | 85.68 | **0.73s** |\n| ESM-650M + LLaMA-3.2-3B | 35.37 | 43.54 | 40.11 | 56.52 | 49.38 | 54.80 | 91.30 | 86.17 | 1.32s |\n| ESM-3B + LLaMA-3.1-8B | **39.16** | **46.67** | **43.34** | **57.24** | **50.17** | **55.39** | **91.95** | **86.81** | 2.03s |\n\nAlso, even our largest model with the 8B LLM decoder can perform inference in less than 2 seconds, which does not pose a significant hurdle.\n\n**[Question on Wet-lab Experiments]**\n\nOur goal is to validate whether the generated functional descriptions from Prot2Text-V2 are biologically meaningful and experimentally actionable. To that end, we envision the following wet-lab validation experiments:\n\n* In vitro expression and solubility screening:\nWe will express selected proteins in heterologous systems (e.g., E. coli, HEK293) to assess expression yield and solubility, which are often indicative of proper folding and biochemical tractability.\n\n* Functional assays:\nPredicted enzymatic or molecular activities (e.g., ATPase, kinase, transport activity) will be tested using established in vitro biochemical assays to confirm functional relevance.\n\n* Subcellular localization:\nProteins will be tagged (e.g., GFP fusion) and imaged using fluorescence microscopy to test localization predictions, such as mitochondrial or membrane targeting.\n\n* Binding and interaction studies:\nTo evaluate predicted protein–protein or protein–ligand interactions, we will use affinity chromatography, pull-down assays, or surface plasmon resonance (SPR) to detect binding specificity and affinity.\n\nThese steps are non-trivial and resource-intensive, but are part of future efforts to use protein captioning for guiding experimental protein function discovery."}, {"review_id": "eaF80EGn18", "reviewer": "Reviewer_4fdJ", "summary": "The paper introduces Prot2Text-V2, a model that aligns protein embeddings with an LLM's feature space using a contrastive strategy, H-SCALE. Following this alignment, the model fine-tunes a LLaMA-3.1-8B-Instruct decoder with LoRA to generate functional protein descriptions. Experiments on a low-homology dataset show the effectiveness of the proposed method.", "questions": "See the Weakness part", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1.\tThe paper solves a meaningful problem, protein-to-text generation.\n2.\tThis paper is easy to understand.\n3.\tThe paper uses a contrastive strategy to better align protein representations with an LLM's feature space, which enhances the model's capability for protein function prediction.\n\nWeaknesses\n1.\tThe novelty is incremental. Most components in this work have existed before, such as ESM, finetuning the LLM with Lora, and contrastive learning. The main contribution of this paper lies in H-SCALE. However, the paper fails to provide a clear theoretical motivation for why aligning both mean and standard deviation is the optimal strategy. Furthermore, it lacks the necessary experimental ablation study (e.g., comparing mean-only, std-only, and the hybrid approaches) to validate this specific design.\n2.\tThe baseline comparisons could be further strengthened, particularly with respect to the state-of-the-art method, ProtT3. Although the paper includes an evaluation against ProtT3, this is conducted on ProtT3's original dataset, which follows a random split. However, one of the core contributions in this work emphasizes generalization under low-homology settings. Therefore, a direct and fair comparison on the more challenging low-homology dataset would be essential to convincingly demonstrate state-of-the-art generalization performance.\n3.\tP2T-GPT [1] also conducted experiments on proteins from a similar database (SwissProt). It trains a protein encoder and GPT from scratch and demonstrates promising results. It would be better to include an analysis and comparison with P2T-GPT.\n[1] Protein Captioning: Bridging the Gap between Protein Sequences and Natural Languages. In TOMM, 2024", "was_revised": true, "final_justification": "Thank you, authors, for providing a detailed rebuttal. The experimental results presented by the authors are convincing, for example, the tables in W1.3, W2, and W3. Regarding novelty, I acknowledge the authors' claim that their method differs from previous approaches. However, it seems more like a combination of several existing techniques. I'm not underestimating the contribution of this work, but I think the novelty is still relatively limited. Taking everything into account, since the authors have provided more experimental results, including comparisons to ProtT3 on low-homology and P2T-GPT, I would prefer to give a result that is somewhere between a weak reject and a weak accept, leaning towards a weak accept.", "rebuttal": "Dear reviewer: \n\nThank you for your valuable feedback. We are very grateful for your recognizing the value of the protein captioning problem, the clarity of the paper and the improved performance. \n\n**[Weakness-1.1 (W1.1) on Novelty]**\n\nWhile we build upon established components like ESM and LLMs, we believe that our proposed model Prot2Text-V2 has significant and novel methodological improvements over existing multimodal LLM designs. While there are some similarities with ProtT3 architecture, our model departs in the following essential respects:\nProtT3 relies on a Q-Former, an additional attention module with learnable queries of a fixed number and a redundant text encoder, to align the protein and text modalities. In contrast, Prot2Text-V2 follows a decoder-only design with a lightweight projector and with a parameter-free alignment method.  Rather than learning alignment implicitly through a query-attention module, H-SCALE directly aligns the projected protein representations to intermediate text embeddings with a new contrastive learning objective. This eliminates the learned bottleneck, preserves the full expressivity of the pretrained protein encoder, and avoids added parameters, latency, and memory, and also allows us to use a more powerful LLM decoder. We also want to emphasize that contrastive learning from intermediate LLM embeddings is not trivial, and it has not been done in any prior work, since all previous methods use an additional text encoder. \n\n**[W1.2 on Theoretical Motivation]**\n\nCompared with the CLIP-based contrastive learning scheme, our H-SCALE approach additionally aligns the standard deviation of sequences of embeddings from two different modalities. Our theoretical motivation is grounded in modeling the sequence of amino-acid token embeddings for a given protein from an underlying multivariate distribution, and the description text token embeddings from another. Under the assumption that these distributions can be reasonably approximated as multivariate Gaussian, they are fully characterized by their first two moments: the mean vector and the covariance matrix. Thus, our approach operationalizes this by aligning the empirical moments of the source (protein) and target (text) distributions. Specifically, we align: \n\n1. The first moment (the mean embedding vector).\n\n2. A diagonal approximation of the second moment (the element-wise standard deviation vector).\n\nThis method serves as a computationally efficient proxy for matching the full covariance matrices, effectively aligning the two distributions via the method of moments. Even when the true distributions deviate from a perfect Gaussian, aligning the second moment provides a richer, more robust alignment than matching the mean alone, as it also captures the feature dispersion. While aligning higher-order moments could offer further refinement, our approach represents a principled trade-off between fine-tuning flexibility, computational tractability and alignment fidelity. It provides a lightweight yet effective mechanism for distribution-level, cross-modal bridging.\n\n**[W1.3 on Ablation Study for CL]**\n\nTo empirically validate this design, we conduct ablation studies that isolate the contribution of mean and std pooling: \n\nTable 1: Ablation study of Prot2Text-V2 with Different Contrastive Learning Schemes\n\n| **Ablations** | **Exct** | **B-2** | **B-4** | **R-1** | **R-2** | **R-L** | **RBT** | **BBT** |\n|---|---|---|---|---|---|---|---|---|\n| w/o alignment | 36.52 | 42.70 | 38.98 | 54.71 | 47.62 | 52.90 | 91.33 | 85.58 |\n| w/ mean-pooling alignment | 37.01 | 44.39 | 41.23 | 55.30 | 48.27 | 53.62 | 91.61 | 86.19 |\n| w/ std-pooling alignment | 33.42 | 37.28 | 33.78 | 50.49 | 43.22 | 48.75 | 90.93 | 84.70 |\n| w/ H-SCALE | 39.16 | 46.67 | 43.34 | 57.24 | 50.17 | 55.39 | 91.95 | 86.81 |\n\nThese results confirm that aligning both mean and variance yields the strongest performance, empirically supporting our design. The results also suggest that aligning only the second-order moment is not sufficient, which supports our decision to align both first- and second-order moments simultaneously for better performance. We sincerely thank the reviewer for the insightful suggestion, and we will include these findings and the corresponding analysis in the camera-ready version of the paper.\n\n**[W2 on Remote Homolog Analysis for ProtT3]**\n\nWe fully agree that evaluating ProtT3 under the same low-homology settings is important for a fair and thorough comparison. In the tables below, we filtered test proteins with the same bitscore ranges as shown in Figure 5, and made a direct comparison between different approaches on the same ranges. Additional evaluation results on BLEU-4 F1 score and BioBert F1 score are shown below.\n\nTable 2: BLEU-4 F1 Score of ProtT3 across BLAST sequence bitscore bins (%) \n\n| Model\\BitScore | [0,30] | (30,35] | (35,40] | (40,45] | (45,50] | (50,55] | (55,60] | (60,65] | (65,70] |\n|---|---|---|---|---|---|---|---|---|---|\n| Prot2Text | 6.58 | 6.41 | 7.51 | 8.63 | 12.58 | 16.03 | 20.76 | 21.68 | 27.04 |\n| ProtT3 | 6.19 | 5.26 | 7.12 | 9.76 | 13.10 | 10.87 | 18.97 | **30.95** | **40.82** |\n| Prot2Text-V2 | **11.89** | **15.21** | **14.78** | **14.00** | **16.05** | **22.20** | **28.65** | 29.26 | 31.08 |\n\nTable 3: BioBert F1 Score of ProtT3 across BLAST sequence bitscore bins (%) \n\n| Model\\BitScore | [0,30] | (30,35] | (35,40] | (40,45] | (45,50] | (50,55] | (55,60] | (60,65] | (65,70] |\n|---|---|---|---|---|---|---|---|---|---|\n| Prot2Text | 73.38 | 73.92 | 74.30 | 74.11 | 75.04 | 76.64 | 78.24 | 78.31 | 78.98 |\n| ProtT3 | 67.57 | 66.44 | 69.19 | 74.37 | **79.73** | 78.11 | 76.44 | 81.06 | **83.99** |\n| Prot2Text-V2 | **76.74** | **77.23** | **77.52** | **77.68** | 78.99 | **80.61** | **81.07** | **81.95** | 83.49 |\n\nThese two additional tables further prove that Prot2Text-V2 outperforms both Prot2Text and ProtT3 across nearly all low-homology bins, especially in the [0,40%] range in both BLEU-4 F1 score and BioBERT F1 score. \n\nHowever, we restate that the original train-test split of the ProtT3 paper is strongly imbalanced and lacks attention to the low-homology proteins. Although our test set contains 60% fewer proteins overall, it includes more than twice as many low-homology proteins compared to theirs. \n\n**[W3 on P2T-GPT]**\n\nWe thank the reviewer for pointing out the relevant paper, P2T-GPT. While P2T-GPT trains its encoder and GPT decoder from scratch, our work explores the opposite direction, leveraging large pretrained models with novel contrastive learning scheme to maximize transfer learning and scalability.\n\nTable 4: Comparison between Prot2Text-V2 and P2T-GPT \n| model | Exct | B-2 | B-4 | R-1 | R-2 | R-L | RBT | BBT |\n|---|---|---|---|---|---|---|---|---|\n| P2T-GPT | 17.24 | 33.30 | 26.06 | 38.14 | 23.72 | 35.47 | 88.23 | 78.66 |\n| Prot2Text-V2 | **39.16** | **46.67** | **43.34** | **57.24** | **50.17** | **55.39** | **91.95** | **86.81** |\n\nP2T-GPT is a relatively lightweight model (~160M parameters) trained entirely from scratch without leveraging large-scale pretrained components or proper multimodal alignment. We conclude from these results that Prot2Text-V2 benefits from integrating powerful pretrained models (ESM and LLaMA) and an H-SCALE adapter, enabling it to capture richer biological and linguistic priors. This architectural difference is reflected in the substantial performance gains across all evaluation metrics, especially in semantic fidelity and functional relevance. We still find this paper of great value as it pioneered in addressing the problem in a novel approach at that time. We will make sure to add these comparison results and cite the paper in the camera-ready version."}, {"review_id": "U3qRRJlg9S", "reviewer": "Reviewer_qp8d", "summary": "This paper presents Prot2Text-V2, a multimodal sequence-to-text model for generating free-form protein function descriptions from amino acid sequences. It integrates a protein encoder (ESM2) with a decoder (LLaMA) via a lightweight projector and introduces a contrastive alignment strategy to improve semantic consistency. The method is evaluated under low-homology settings and shows strong performance compared to prior baselines.", "questions": "The appendix shows that including protein names significantly boosts performance. Could the authors clarify to what extent the model relies on such semantic cues, and whether this impacts its generalization to unnamed or novel proteins?", "limitations": "The authors clearly acknowledge the lack of wet-lab validation and discuss potential extensions to structural modalities and reasoning capabilities.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\nThe paper addresses the underexplored task of generating free-text protein function descriptions directly from sequences, which has practical value in annotating novel proteins beyond fixed ontology terms.\n\nWeaknesses\n1. The core modeling choices, including the use of a nonlinear projector and decoder-only architecture, are more of engineering simplifications than methodological innovations. While effective, these components largely build on existing paradigms rather than introducing fundamentally new ideas.\n\n2. Although the paper emphasizes generalization under weak-homology settings, it lacks direct comparison to other function prediction methods that also claim to improve generalization. In addition, ProtT3 is only evaluated under random splits and not under the same low-homology CD-HIT split, limiting the fairness of the comparison.", "was_revised": true, "final_justification": "Thank you for the new results—they address a major concern and clearly show improved low-homology performance over ProtT3. I was originally referring to comparisons with other methods specifically designed for low-homology settings, which are not discussed. Also, the performance gap between low and high homology bins raises questions about the impact of training data distribution. Finally, I appreciate the clarifications and additional effort.", "rebuttal": "Dear reviewer: \n\nThank you for your thoughtful review and for recognizing the value of our work in annotating novel proteins. We appreciate your feedback, which points to important areas where we can further clarify our contributions and strengthen our experimental comparisons. We address the concerns below:\n\n**[Weakness 1 (W1) on Novelty]**\n\nWhile we build upon established components like ESM and LLMs, we believe that our proposed model Prot2Text-V2 has significant and novel methodological improvements over existing multimodal LLM designs. While there are some similarities with ProtT3 architecture, our model departs in the following essential respects:\n\nProtT3 relies on a Q-Former, an additional attention module with learnable queries of a fixed number and a redundant text encoder, to align the protein and text modalities. In contrast, Prot2Text-V2 follows a decoder-only design with a lightweight projector and with a parameter-free alignment method.  Rather than learning alignment implicitly through a query-attention module, H-SCALE directly aligns the projected protein representations to intermediate text embeddings with a new contrastive learning objective. This eliminates the learned bottleneck, preserves the full expressivity of the pretrained protein encoder, and avoids added parameters, latency, and memory, and also allows us to use a more powerful LLM decoder.  We also want to emphasize that contrastive learning from intermediate LLM embeddings is not trivial, and it has not been done in any prior work, since all previous methods use an additional text encoder. \n\nCompared with the CLIP-based contrastive learning scheme, our H-SCALE approach additionally aligns the standard deviation of sequences of embeddings from two different modalities. Our theoretical motivation is grounded in modeling the sequence of amino-acid token embeddings for a given protein from an underlying multivariate distribution, and the description text token embeddings from another. Under the assumption that these distributions can be reasonably approximated as multivariate Gaussian, they are fully characterized by their first two moments: the mean vector and the covariance matrix. Thus, our approach aligns the empirical moments of the source (protein) and target (text) distributions.\n\n**[W2 on Comparison with ProtT3]**\n\nWe fully agree that evaluating ProtT3 under the same low-homology settings is important for a fair and thorough comparison. In the tables below, we filtered test proteins with the same bitscore ranges as shown in Figure 5, and made a direct comparison between different approaches on the same ranges. Additional evaluation results on BLEU-4 F1 score and BioBert F1 score are shown below.\n\nTable 2: BLEU-4 F1 Score of ProtT3 across BLAST sequence bitscore bins (%) \n\n| Model\\BitScore | [0,30] | (30,35] | (35,40] | (40,45] | (45,50] | (50,55] | (55,60] | (60,65] | (65,70] |\n|---|---|---|---|---|---|---|---|---|---|\n| Prot2Text | 6.58 | 6.41 | 7.51 | 8.63 | 12.58 | 16.03 | 20.76 | 21.68 | 27.04 |\n| ProtT3 | 6.19 | 5.26 | 7.12 | 9.76 | 13.10 | 10.87 | 18.97 | **30.95** | **40.82** |\n| Prot2Text-V2 | **11.89** | **15.21** | **14.78** | **14.00** | **16.05** | **22.20** | **28.65** | 29.26 | 31.08 |\n\nTable 3: BioBert F1 Score of ProtT3 across BLAST sequence bitscore bins (%) \n\n| Model\\BitScore | [0,30] | (30,35] | (35,40] | (40,45] | (45,50] | (50,55] | (55,60] | (60,65] | (65,70] |\n|---|---|---|---|---|---|---|---|---|---|\n| Prot2Text | 73.38 | 73.92 | 74.30 | 74.11 | 75.04 | 76.64 | 78.24 | 78.31 | 78.98 |\n| ProtT3 | 67.57 | 66.44 | 69.19 | 74.37 | **79.73** | 78.11 | 76.44 | 81.06 | **83.99** |\n| Prot2Text-V2 | **76.74** | **77.23** | **77.52** | **77.68** | 78.99 | **80.61** | **81.07** | **81.95** | 83.49 |\n\nThese two additional tables further prove that Prot2Text-V2 outperforms both Prot2Text and ProtT3 across nearly all low-homology bins, especially in the [0,40%] range in both BLEU-4 F1 score and BioBERT F1 score. \n\nHowever, we restate that the original train-test split of the ProtT3 paper is strongly imbalanced and lacks attention to the low-homology proteins. Although our test set contains 60% fewer proteins overall, it includes more than twice as many low-homology proteins compared to theirs. \n\n**[Question on Protein Name]**\n\nIn our main experiments, protein names are not included in the training or in the inference, ensuring that the model does not trivially memorize or rely on name identifiers. The analysis in the appendix shows that including names boosts performance, as expected, but this setting was only used to probe the upper bound of the model’s capacity, not in our default evaluation. Furthermore, our low-homology setup ensures that the model generalizes to novel and unseen proteins, without the name information."}]

Author rebuttal:

Meta-review: This paper introduces Prot2Text-V2, a multimodal framework for generating natural language descriptions of protein functions directly from amino acid sequences. A key innovation is the hybrid sequence-level contrastive alignment learning. The model uses a novel contrastive method by combining ESM and LLAMA. While the originality is modest as it primarily combines existing components, this method shows consistent improvements. All reviewers vote for acceptance, recognizing the practical significance, despite incremental novelty concerns.
Decision: Accept (poster)