ProtT3: Protein-to-Text Generation for Text-based Protein Understanding (2024)

Abstract: Language Models (LMs) excel in understanding textual descriptions of proteins, as evident in biomedical question-answering tasks. However, their capability falters with raw protein data, such as amino acid sequences, due to a deficit in pretraining on such data. Conversely, Protein Language Models (PLMs) can understand and convert protein data into highquality representations, but struggle to process texts. To address their limitations, we introduce ProtT3, a framework for Protein-to-Text Generation for Text-based Protein Understanding. ProtT3 empowers an LM to understand protein sequences of amino acids by incorporating a PLM as its protein understanding module, enabling effective protein-to-text generation. This collaboration between PLM and LM is facilitated by a cross-modal projector (i.e., Q-Former) that bridges the modality gap between the PLM’s representation space and the LM’s input space. Unlike previous studies focusing on protein property prediction and protein-text retrieval, we delve into the largely unexplored field of protein-to-text generation. To facilitate comprehensive benchmarks and promote future research, we establish quantitative evaluations for protein-text modeling tasks, including protein captioning, protein question-answering, and protein-text retrieval. Our experiments show that ProtT3 substantially surpasses current baselines, with ablation studies further highlighting the efficacy of its core components. Our code is available at https://github.com/acharkq/ProtT3.

heading: 1 Introduction
text: Language Models (LMs) have achieved impressive successes across diverse domains (Brown et al., 2020; Touvron et al., 2023; Liu et al., 2023c). Remarkably, the extensive biological literature in their
* Corresponding author. Xiang Wang is also affiliated with Institute of Dataspace, Hefei Comprehensive National Science Center.
training data has enabled LMs to excel in text-based protein understanding tasks, such as biological and medical question-answering (QA) (Taylor et al., 2022; OpenAI, 2023). These results show the potential of using LMs as the natural language interface for biomedical tasks. It further accentuates the importance of harnessing LMs to drive advancements in areas like drug discovery and protein property prediction (Kim et al., 2021; Fan et al., 2023).
Here we focus on LM’s capability in text-based protein understanding, which enables textual interpretations of proteins. It can be assessed through diverse downstream tasks, particularly protein-to-text generation (Guo et al., 2023) and retrieval (Xu et al., 2023). Specifically, the retrieval task is finding an existing text description best matching a particular protein, while the generation task is delineated into
5949
two problems, as illustrated in Figure 1: 1) protein captioning, where the LM generates a description of a specific protein’s functions, and 2) protein QA, where the LM answers questions about a protein. Table 1 overviews prior efforts (Liu et al., 2023b; Xu et al., 2023; Taylor et al., 2022; Guo et al., 2023) in this field. Delving into these previous studies, we identify two research gaps:
• Lack of Exploration for Protein-to-Text Generation. Protein-to-text generation is a conditional generation task (Keskar et al., 2019), requiring the LM to perceive proteins as the generation condition. Previous studies (Xu et al., 2023; Liu et al., 2023b) based on cross-modal contrastive learning (Radford et al., 2021) hardly interpret proteins as the direct inputs to the LM. While notably two studies, Galactica and ProteinChat, have paid certain explorations, they unfortunately appear to be quite constrained by key limitations. Specifically, Galactica (Taylor et al., 2022) incorporates only a limited set of protein sequences in its pretraining data, thereby potentially restricting its capacity for comprehensive protein understanding; ProteinChat (Guo et al., 2023) seeks to project protein representations to text space by training a linear projector, which might prove inadequate in capturing the intricate relations between proteins and texts.
• Missing Quantitative Evaluation. The progress in protein-text modeling is difficult to track without proper benchmarks. As Table 1 illustrates, the quantitative evaluations for these models are mostly missing, posing a challenge to further advancement in this field.
To bridge these research gaps, we propose ProtT3: Protein-to-Text Generation for Textbased Protein Understanding. As Figure 2a illustrates, ProtT3 empowers an LM to understand protein sequences of amino acids by incorporating a Protein Language Model (PLM) as its protein understanding module, thereby effectively conditioning the protein-to-text generation process.
PLMs (Madani et al., 2023) are specialized LMs pretrained solely on protein sequences. They can generate powerful protein representations that are instrumental in presenting the proteins’ 3D structures and indicating their potential properties (Chen et al., 2023). To enable the LM to understand the PLM’s protein representations, ProtT3 integrates an expressive cross-modal projector – QFormer (Li et al., 2023a) – to map protein representations into the text space of the LM. This design enables the LM to consume proteins as inputs. However, working with a large LM with billions of parameters raises a new challenge of maintaining the efficiency of downstream adaptation. Therefore, we incorporate a LoRA (Hu et al., 2022) adapter into the LM for efficient fine-tuning purposes.
To facilitate effective protein-text modeling, ProtT3 employs a two-stage training process to enhance protein-text modeling, as outlined in Figure 2b. The first stage involves protein-text retrieval training with three cross-modal tasks (Li et al., 2022a): protein-text contrasting, protein-text matching, and protein captioning. This stage not only empowers the cross-modal projector with the capability of protein-text retrieval, but also serves as a “warmup” before the second stage by encouraging the extraction of text-relevant protein features. In the second stage, we connect the crossmodal projector to the LM and conduct protein-totext generation training.
Our key contributions are summarized as:
• We introduce ProtT3, a new framework aiming to bridge the modality gap between texts and proteins. Through a cross-modal projector, ProtT3 jointly uses a PLM for protein understanding and an LM for text processing, enabling effective protein-to-text generation.
• To set benchmarks and promote future research, we establish quantitative evaluations for proteintext modeling tasks, including protein captioning, protein QA, and protein-text retrieval. The datasets, evaluation scripts, and our pretrained
checkpoints will be made available online.
• ProtT3 achieves state-of-the-art performances across various tasks. For protein captioning, ProtT3 surpasses the baseline by over 10 BLEU2 scores in the Swiss-Prot (Bairoch and Apweiler, 2000) and the ProteinKG25 (Zhang et al., 2022) datasets. For protein-text retrieval, ProtT3 outperforms baselines by over 14% in retrieval accuracy on the Swiss-Prot and ProteinKG25 datasets. Lastly, ProtT3 achieves 2.5% improvement of exact match performance for protein QA on the PDB-QA (Guo et al., 2023) dataset.

heading: 2 Related Works
text: Here we briefly review relevant fields of PLMs, protein-text modeling, and multi-modal LMs.
Protein Language Models (PLMs). PLMs are transformer-based LMs pretrained on large corpora of protein sequences for protein understanding and protein generation (Lin et al., 2022; Madani et al., 2023; Nijkamp et al., 2022; Chen et al., 2023; Elnaggar et al., 2021; Rives et al., 2021a; Meier et al., 2021). Similar to LMs for texts, PLMs are pretrained by the objective of masked language modeling (Lin et al., 2022) or auto-regressive modeling (Madani et al., 2023). PLMs have demonstrated promising performances on downstream tasks of 3D structure prediction and protein property prediction (Lin et al., 2022; Chen et al., 2023). However, they cannot process texts due to the absence of text in their pretraining data.
Protein-Text Modeling. The pioneering study, Galactica (Taylor et al., 2022), is an LM pretrained on a text corpus that includes a small amount of protein sequences. Its protein understanding ability can be suboptimal compared to PLMs due to insufficient protein pretraining data. To mitigate this gap, subsequent works (Xu et al., 2023; Liu et al., 2023b) jointly leverage a PLM and an LM by cross-modal contrastive learning (Radford et al., 2021). However, cross-modal contrastive learning is insufficient for protein-to-text generation. This is because protein-to-text generation is a conditional generation task. It demands the LM to understand proteins as the generation condition, which contrastive learning cannot achieve. Further, ProteinChat (Guo et al., 2023) attempts to enable an LM to understand proteins by training a linear projector between a PLM and an LM. Considering both the PLM and LM are kept frozen during training, this approach struggles to capture the nonlinear re-
lations between proteins and texts. In a different approach, OntoProtein (Zhang et al., 2022) finetunes a PLM using a biology knowledge graph, focusing more on protein property prediction and less on text-related tasks.
Multi-modal LMs. Enabling LMs to understand another modality, such as image (Li et al., 2022a; Tsimpoukelli et al., 2021), video (Zhang et al., 2023), audio (Lyu et al., 2023), user-items (Liao et al., 2024), and molecule (Liu et al., 2023c; Shi et al., 2023; Li et al., 2024; Fang et al., 2024b; Edwards et al., 2022), has been an activate research direction. Notably, the research of multi-modal LMs is pioneered by vision-language modeling (VLM). VLM has been successfully applied for few-shot image classification, image captioning, and image QA (Li et al., 2022a; Alayrac et al., 2022). Specifically, to enable LMs to understand images, the leading VLM methods employ either 1) nonlinear and expressive cross-modal projectors (Li et al., 2023a; Alayrac et al., 2022), or 2) fine-tuned visual encoders and LMs on multi-modal datasets (Driess et al., 2023), while such studies are missing for protein-text modeling. Our work takes the initiative to explore this direction.

heading: 3 Model Architecture
text: Here we introduce ProtT3’s three key components: a PLM for protein understanding, an LM for text processing, and a cross-modal projector to bridge the modality gap between the first two components.
Protein Language Model (PLM). We utilize ESM-2 (Lin et al., 2022) to encode protein sequences of amino acids. ESM-2 is an encoder-only transformer LM (Vaswani et al., 2017) pretrained on 60M protein sequences by masked language modeling (Devlin et al., 2019). PLMs have shown promising performances for protein folding (Lin et al., 2022), multiple sequence alignment (Rives et al., 2021b), and protein property prediction (Xu et al., 2023), demonstrating their effectiveness for capturing protein characteristics. For efficiency, we freeze ESM-2’s weights in our training process.
Language Model (LM). We choose Galactica (Taylor et al., 2022) as the base LM. Galactica is a decoder-only transformer LM pretrained on a large collection of scientific papers, spanning disciplines like biology and medicine. Notably, Galactica has demonstrated a high-level understanding of protein concepts through its promising performances in biomedical QA benchmarks (Jin et al., 2019; Hendrycks et al., 2021). Furthermore, Galac-
tica includes a small set of protein sequences in its pretraining data, and shows a capability for understanding protein sequences through the task of protein keyword prediction (Taylor et al., 2022). Therefore, we also leverage Galactica as a baseline for protein-to-text generation to ablate the effectiveness of incorporating an additional PLM for protein understanding.
Cross-Modal Projector. We employ a crossmodal projector based on Q-Former (Li et al., 2023a) to bridge the modality gap between the PLM and the LM. Q-Former has demonstrated promising performances in vision-language tasks. As Figure 3a illustrates, Q-Former consists of two transformers: one for protein encoding and another for text processing. Specifically, the protein transformer maintains Nq learnable query tokens {qi ∈ Rd} Nq i=1 as inputs. These query tokens can interact with the PLM through the crossattention modules, in order to extract protein features. We denote the protein transformer’s output as Z ∈ RNq×d, containing protein features. For text input, the text transformer adds a [CLS] token at the beginning, and uses the [CLS] token’s output as the text representation. Both transformers share the self-attention to enable interactions between proteins and texts. Details are in Section 4.1.
Q-Former’s weights are initialized from PubMedBERT-Abstract (Gu et al., 2022), a BERT LM pretrained on paper abstracts from the PubMed database1. It has shown promising performances in understanding biomedical concepts under the BLURB benchmark (Gu et al., 2022). The cross-attention module is added into the Q-Former every two layers and is randomly initialized.

heading: 4 Training Method
text: In this section, we introduce ProtT3’s two training stages: protein-text retrieval and protein-to-text
1https://pubmed.ncbi.nlm.nih.gov/
generation. The training process leverages a dataset of protein-text pairs D = {(p1, t1), (p2, t2), ...}, where pi is a protein sequence and ti is the corresponding text sequence.

heading: 4.1 Stage 1: Protein-Text Retrieval Training
text: Inspired by BLIP (Li et al., 2022a, 2023a), we jointly employ three objectives for protein-text retrieval training: protein-text contrasting, proteintext matching, and protein captioning. These objectives are tailored for Q-Former’s architecture, training it to extract protein features that are relevant to the text descriptions. This stage empowers the cross-modal projector with retrieval ability, and also serves as a “warmup” before the next stage.
Protein-Text Contrasting (PTC). We employ cross-modal contrastive learning (Radford et al., 2021) to align the protein representation and text representation from the Q-Former. As illustrated in Figure 3b, Q-Former’s self-attention module separately processes the query tokens and text tokens without any interaction. This enforces the query tokens to extract protein features from the PLM, in order to generate protein representations that align with the corresponding text representations in contrastive learning.
Formally, let {(p1, t1), ...(pB, tB)} be a batch of protein-text pairs. We denote the protein representations as {Zi ∈ RNq×d}Bi=1, where Zij ∈ Rd is the reprentation of the protein pi’s j-th query token; and denote text ti’s representation as mi ∈ Rd, which is the [CLS] token’s output. Protein-text similarity is measured by the maximum similarity between m and each row of Z. The contrastive learning loss LPTC can then be written as:
Lp2t = 1 B
B∑
i=1 log exp(maxk cos(Zik,mi)/τ)∑B j=1 exp(maxk cos(Zik,mj)/τ) ,
Lt2p = 1 B
B∑
i=1 log exp(maxk cos(Zik,mi)/τ)∑B j=1 exp(maxk cos(Zjk,mi)/τ) ,
LPTC = −Lp2t − Lt2p, (1)
where cos(·, ·) is the cosine similarity; Temperature τ is empirically set to 0.1.
Protein-Text Matching (PTM). PTM is a binary classification objective aiming to discriminate whether a protein-text pair matches or not. Unlike PTC which computes protein-text similarity by applying cosine similarity on their output representations, PTM can obtain more fine-grained protein-text similarity. As illustrated in Figure 3b, PTM feeds the query tokens and text tokens into the same self-attention module, allowing them to interact at the Q-Former’s early layers. In this way, the query tokens can capture information on both proteins and texts. The mean pooling of query tokens’ representations is then fed into a linear classifier for PTM prediction. During training, for each positive protein-text pair (pi, ti), we randomly sample two negative pairs: (pi, ti′) and (pi′′ , ti). Let f(p, t) ∈ [0, 1] be the Q-Former’s predicted probability that (p, t) is a matched pair, the PTM loss function LPTM can be written as:
LPTC = 1
B
B∑
i=1
[− log f(pi, ti) + log f(pi′ , ti)
+ log f(pi, ti′′)]. (2)
In experiments, we find that PTM surpasses PTC for protein-text retrieval. However, PTM incurs a higher computational cost for encoding every protein-text pair. To balance performance and speed, we use PTC to obtain the top k ranked candidates, and then use PTM for re-ranking.
Protein Captioning (PCap). Protein captioning trains the Q-Former to generate text descriptions of given proteins. As shown in Figure 3b, we apply a special masking strategy for this objective: 1) Bi-directional attention mask is applied to query tokens, enabling them to interact with each other but not text tokens; 2) Causal attention mask is used for text tokens, allowing them to attend query tokens and the preceding text tokens, but not the following text tokens. This design ensures the text tokens extract protein features exclusively from the query tokens, because they cannot directly interact with the proteins. Meanwhile, the query tokens are enforced to extract protein features through crossattentions, satisfying the informational needs of protein captioning. Let P (t|p) be the probability that Q-Former generate text t given protein p. The
Size Avg Protein Len Avg Text Len
Train 430595 336 48 Valid 10000 358 59 Test 10000 357 60
(a) Swiss-Prot dataset of proteins and their text descriptions.
Size Avg Protein Len Avg Text Len
Train 422315 338 101 Valid 10000 360 104 Test 10000 360 107
(b) ProteinKG25 dataset of proteins and their text descriptions.
#Protein #QA Pair Avg Protein Len Avg Text Len
Train 114690 3359693 291 10 Valid 10000 291795 257 10 Test 10000 291779 259 10
(c) PDB-QA dataset. Each protein is acompanied by multiple QA pairs, therefore proteins and QA pairs have different numbers.
Table 2: Statistics of our used protein-text datasets. Text lengths are counted by splitting at spaces.
protein captioning loss LPCap can be written as:
LPCap = − 1
B
B∑
i=1
logP (ti|pi). (3)

heading: 4.2 Stage 2: Protein-to-Text Generation Training
text: In this stage, we train ProtT3 for protein-to-text generation. As Figure 2a illustrates, we connect the cross-modal projector to the LM, feeding the protein representations Z into the LM, so as to condition the text generation process by protein information. Note that, we use a linear layer to project Z to the same dimension of the LM’s input.
We train ProtT3 for each generation dataset separately, and append different text prompts after the protein representations to further control the generation process. For example, we use the text prompt of “Describe this protein’s function” for protein captioning, and “Question: Does this protein contain polymer entities? Answer:” for protein QA. For training, we use the same loss as the protein captioning task in the previous section.
For training efficiency, we incorporate LowRank Adaptation (LoRA) (Hu et al., 2022) adapters into the LM. LoRA adds pairs of trainable rank decomposition matrices into the selected weights of the LM. For example, LoRA modifies a pretrained weight W0 ∈ Rd1×d2 by adding a pair of matrices B ∈ Rd1×r and A ∈ Rr×d2 :
W = W0 + BA, (4)
where W0 is kept frozen and only the newly added BA is tuned. By using a small rank r ≪ min(d1, d2), LoRA can adapt an LM to a new task while requiring little memory for storing gradients. This method has shown comparable performances to full-parameter fine-tuning (Hu et al., 2022).

heading: 5 Experiments
text: We begin by introducing the datasets of protein-text pairs, followed by the experimental results. Details on experimental settings, such as hyperparameters and baseline implementations, are provided in Appendix C. In the experiments, ProtT3 utilizes Galactica1.3B as the base LM and ESM-2150M as the PLM.

heading: 5.1 Protein-Text Dataset Collection
text: In this section, we detail the protein-text pair datasets and the data processing procedure employed in our study. For all the used datasets, we discard protein sequences longer than 1022 tokens, and carefully split the datasets to ensure no overlap between train/valid/test sets. Dataset statistics are available in Table 2, and details are in Appendix A.
Swiss-Prot (Bairoch and Apweiler, 2000) is a protein sequence database with text annotations. We process the dataset following (Xu et al., 2023), but excluding protein names from the text annotations to prevent information leakage. The resulting
text descriptions concatenate the annotations of protein functions, locations, and families.
ProteinKG25 (Zhang et al., 2022) is a knowledge graph derived from the Gene Ontology (Aleksander et al., 2023) database. We transform its triples into free texts by first aggregating the triples of the same protein, and then filling the protein information into a pre-defined text template.
PDB-QA (Guo et al., 2023) is a protein singleturn QA dataset derived from RCSB PDB2. It includes 30 question templates about proteins’ structures, properties, and supplementary information. As shown in Table 3, to enable a fine-grained evaluation, we categorize the questions into four types, based on the answer’s format (string or number) and content focus (structure/property or supplementary information). It is worth noting that supplementary information is hard to predict given the protein sequence alone.
Training Pipeline. In training stage 1, we train ProtT3 on the combination of the Swiss-Prot and ProteinKG25 datasets for protein-text retrieval. In training stage 2, we load the checkpoint from stage 1 and conduct separate fine-tuning on the three datasets for protein-to-text generation tasks.
2https://www.rcsb.org

heading: 5.2 Protein Captioning
text: We evaluate protein captioning performances on the Swiss-Prot and ProteinKG25 datasets. Following (Edwards et al., 2022), we use the evaluation metrics of BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and METEOR (Banerjee and Lavie, 2005). Additionally, we report the percentage that the prediction exactly matches the ground truth annotation. For comparison and ablation study purposes, we report the performances of LoRA fine-tuned Galactica1.3B, and ProtT3’s two variants: 1) ProtT3 w/ MLP Proj., which replaces ProtT3’s cross-modal projector by an MLP, following (Liu et al., 2023a); and 2) ProtT3 w/o stage 1, which skips ProtT3’s training stage 1. We do not compare with ProteinChat (Guo et al., 2023) because it requires 3D protein structures, which are unavailable for these two datasets.
Table 4 presents the results. We observe the following: 1) ProtT3 and its variants substantially outperform the LoRA fine-tuned Galactica1.3B, with ProtT3 showing a 10-point improvement in BLEU2 scores. This underscores the significance of incorporating a PLM for protein understanding and ProtT3’s effectiveness in undertanding protein inputs. 2) ProtT3 consistently surpasses its two variants on both datasets, highlighting the advantages of using a Q-Former projector and training stage 1.

heading: 5.3 Protein-Text Retrieval
text: We evaluate protein-text retrieval performances on the Swiss-Prot and ProteinKG25 datasets. Fol-
lowing (Su et al., 2022), our evaluation includes protein-text retrieval in a batch of 64 random samples and retrieval in the entire test set. We use Accuracy and Recall@20 as the evaluation metrics. For retrieval, ProtT3 first retrieves the top 128 candidates by PTC, and then uses PTM for reranking. For comparison, we employ ProtST (Xu et al., 2023) and ProteinCLAP (Liu et al., 2023b) as baselines, and present ProtT3 w/o PTM and ProtT3 w/o PCap for ablation studies.
Table 5 shows the results. We observe that: 1) ProtT3 outperforms baselines by over 14% accuracy for retrieval in the test set, highlighting its capability in aligning proteins with corresponding text descriptions. 2) PTM improves ProtT3’s retrieval accuracy in test set by 1 ∼ 2% on both datasets. This is because PTM allows protein and text information to interact at the Q-Former’s early layers, achieving more fine-grained protein-
text similarity measurements. 3) PCap improves ProtT3’s retrieval accuracy by ∼ 2%. This is because PCap encourages the query tokens to extract protein information most relevant to the text input, therefore aiding in protein-text alignment.

heading: 5.4 Protein Question-Answering
text: We evaluate protein QA performances on the PDBQA dataset. Considering that the answers in PDBQA typically consist of 1 ∼ 2 words, we select exact match as the evaluation metric. For comparison, we employ ProteinChat (Guo et al., 2023) and LoRA fine-tuned Galactica1.3B as baselines. We also assess a Galactica1.3B that consumes only questions without proteins during training and prediction (i.e., LoRA ft, Q-only). This baseline measures the dataset’s uni-modal bias (Cadène et al., 2019): the proportion of questions that can be correctly answered without looking at proteins.
The results are shown in Table 6. We observe that: 1) ProtT3 surpasses baselines by 2.5% overall, and consistently outperforms them in predicting protein structures and properties. This demonstrates ProtT3’s multi-modal understanding ability for the proteins and the textual questions. 2) The baseline consuming question only (i.e., LoRA ft, Q-only) presents comparable performance, indicating that the dataset has a substantial uni-modal bias. This finding suggests an opportunity to ex-
plore debias techniques (Mahabadi et al., 2020) in future studies. 3) ProteinChat underperforms other methods, possibly because it only trains a linear layer between the frozen PLM and LM. The linear layer is insufficient to map the protein representations into the text space of the LM, and adapt the LM to QA. 4) Models perform worse on questions about numbers or supplementary information. This observation suggests the potential benefit of augmenting these models with external tools (Schick et al., 2023) like calculators and search engines.

heading: 5.5 Ablation Studies on Pretrained Models
text: Here we conduct ablation studies on the pretrained models used in our method. Specifically, we ablate the impact of different PLMs in training stage 1, and ablate different LMs in training stage 2.
Ablating PLMs. In training stage 1, we replace the ESM-2150M protein encoder with its smaller variants, namely ESM-28M and ESM-235M, to evaluate their performances for protein-text retrieval. As shown in Table 7, we can observe that retrieval performance increases monotonically with model size, a trend consistent with previous observations in the LM domain.
Ablating LMs. In training stage 2, we replace Galactica1.3B to Phi-1.51.3B (Li et al., 2023c). Unlike Galactica, Phi-1.51.3B is pretrained on general domain data but not focusing on scientific liter-
ature. Table 8 shows protein captioning performance on the ProteinKG25 (Zhang et al., 2022) dataset. We can observe that Galactica1.3B significantly outperforms Phi-1.51.3B for protein captioning, although they have similar sizes. We attribute this performance gap to their pretraining corpus, with Galactica1.3B performing better for pretraining on more scientific literature.

heading: 5.6 Examples of Protein-to-Text Generation
text: Figure 4 shows three examples of protein captioning. In the first example, ProtT3’s caption is more accurate by correctly identifying the DMRT family, while Galactica does not. In the second example, both models fail to identify the protein’s function. Nevertheless, ProtT3’s prediction regarding the protein family is more accurate. The third example shows that both models successfully predict the subcellular location and protein family. ProtT3 goes a step further by predicting the protein’s function, which is closer to the ground truth description.
Figure 5 shows three examples of protein QA. We can observe that both ProtT3 and Galactica answer the first two questions about protein property/structure correctly, and fail on the third question, which requires a numerical answer. On the other hand, ProteinChat struggles with all three questions, failing to answer each of them.

heading: 6 Conclusion and Future Works
text: In this work, we propose ProtT3, a new proteintext modeling framework. ProtT3 aims to facilitate text-based protein understanding via protein-to-text generation and protein-text retrieval. To achieve this, ProtT3 integrates a PLM into an LM to enhance the LM’s protein understanding ability. A cross-modal projector enables this integration by bridging the modality gap between the two modules. To promote future research, we set benchmarks for protein-text modeling tasks, including protein captioning, protein QA, and protein-text retrieval, where ProtT3 significantly outperforms existing baselines.
Looking ahead, we plan to explore enabling LMs to understand 3D protein structures (Li et al., 2024) and apply this understanding to more tasks of drug discovery, property prediction (Liu et al., 2023d; Li et al., 2022b, 2023b), molecule generation (Luo et al., 2024), and OOD generalization (Fang et al., 2023, 2024a).
Limitations
On Representing Proteins by 1D Sequences. Proteins can be represented by either 1D sequences of amino acids or 3D coordinates of their atoms. In this study, we represent proteins by 1D sequences considering their abundance compared to 3D structures. As of 2023, the Protein Data Bank contains 220K 3D structures 3, whereas the UniProt database houses 227M 1D sequences (Consortium, 2023). This abundance of 1D sequences enables us to collect a larger dataset. In Section 5.1, we collect a total of 1M protein-text pairs, a significant increase from the 143K pairs in the earlier work (Guo et al., 2023) using 3D structures.
3https://www.wwpdb.org/stats/deposition
While it is feasible to predict the 3D structures of protein sequences by running protein folding algorithms (Jumper et al., 2021; Lin et al., 2022), this prediction process is notably time-consuming. For example, predicting the 3D structures of the 440k protein sequences in the ProteinKG25 (Zhang et al., 2022) dataset would take approximately four months when using an A100 80GB GPU. Due to this computational demand, we choose to focus on 1D sequences, leaving the exploration of 3D structures for future research.
On Dynamic Protein Structures. Proteins have dynamic structures, and natural language is limited in fully depicting protein dynamics. Nonetheless, similar to video-to-text generation (Aafaq et al., 2020), where language partially describes dynamic videos, our work on protein-to-text generation is a crucial step towards encapsulating complex biological phenomena in a human-readable form.
Potential Ethics Impact
In this study, the proposed method and dataset focus on proteins and their properties, and include no human subjects. Consequently, we believe this study presents no direct ethical concerns. However, the proposed LMs can be abused to generate biased or toxic information, and can generate inaccurate description of protein properties and structures. Therefore, the ethical implications of our work align with those common to LM research.

heading: Acknowledgement
text: This research is supported by the National Science and Technology Major Project (2023ZD0121102), National Natural Science Foundation of China (92270114). This research is partially supported by the National Research Foundation Singapore under the AI Singapore Programme (AISG Award No: AISG2-TC-2023-010-SGIL), the Singapore Ministry of Education Academic Research Fund Tier 1 (Award No: T1 251RES2207) and the Google Cloud Research Credits program with the award (Q4MJ-YH1K-3MVX-FP6Q). This research is supported by NExT Research Center.

heading: A Details of Protein-Text Dataset Collection
text: Swiss-Prot (Bairoch and Apweiler, 2000). Dataset samples are provided in Table 11. We largely follow the instructions in (Xu et al., 2023) for data processing. We first download the lastest dump of Swiss-Prot from the UniProt website4, and then select three annotation fields: Function, which describes protein’s roles such as catalysis and transport; Subcellular Location, indicating where a protein is typically found within a cell; Similarity, which details the protein’s family. These three annotation fields are then concatenated to form the text description of proteins. Different from (Xu et al., 2023), we do not include the annotation of Protein Name in the text description. This is because Protein Name can be potential information leakage for the protein-text retrieval task.
ProteinKG25 (Zhang et al., 2022). Dataset samples are shown in Table 13. ProteinKG25 is originally a knowledge graph derived from the Gene Ontology database (Aleksander et al., 2023). ProteinKG25 is stored as triples in the format of (protein sequence, relation, property), such as (protein 1, enables, structural constituent of ribosome), (protein 2, is involved in metabolic process, rRNA processing), and (protein 3, is located in, cytoplasm). To synthetic text descriptions using such triples, we perform the following steps:
• Group triples by protein sequences. Each resulting group has triples of the same protein.
• Further group triples by relation types. Each resulting group has triples of the same protein and the same relation type. For example, a group can include triples of (protein 1, is involved in lipid metabolic process, fatty acid biosynthetic process), (protein 1, is involved in lipid metabolic process, lipid catabolic process), and (protein 1, is involved in lipid metabolic process, fatty acid metabolic process).
• Transform the triples in each group into a sentence by filling the information into a text template. For example, the three triples in the previous step will be transformed into: This protein is involved in the following lipid metabolic process: fatty acid biosynthetic process, lipid catabolic process, and fatty acid metabolic process.
4https://www.uniprot.org/
• Concatenate the sentences of the same protein together to form the final text description.
Note that, we do not include the protein sequence information in the text description to avoid information leakage. The Python script for data processing and the text templates will be made available online to facilitate future research.
PDB-QA (Guo et al., 2023). The categorization of all 30 question templates is detailed in Table 14. The original PDB-QA dataset provides only 3D structures of proteins. To retrieve their 1D sequences, we call the web API of the RCSB PDB website5 using the proteins’ PDB IDs.

heading: B Error Analysis
text: Questions of Structures/Properties. Table 9 presents the statistics of sampled questions in this category. We can observe a strong correlation between accuracy and the number of candidate answers. This observation is intuitive, as questions become more challenging with an expanded answer pool. Further, questions that have a large number of candidate answers are those that require numerical outputs. This highlights the potential of improving the ability of counting substructures and processing numbers in future studies.
Questions of Supplementary Information. Table 10 shows the sampled questions of supplementary information and the candidate answers. We can observe that the questions and answers are not directly relevant to protein structures or functions. Therefore, we infer that the success of these questions often relies on exploiting the dataset’s inherent biases rather than understanding protein structures and properties. This realization prompts a concern that the model’s performance could significantly diminish when applied to datasets without similar biases, indicating a potential area for improvement using retrieval augmentation.

heading: C Experimental Details
text: Following MolT5 (Edwards et al., 2022) and ProteinCLAP (Liu et al., 2023b), we use a single same random seed for experiments. Using multiple random seeds is cost-prohibitive for tuning large LMs.
The number of parameters of ProtT3 and keybaseline are summarized in Table 12. All the experiments are conducted on either two A100-40GB GPUs or four V100-32GB GPUs. We have used
5https://www.rcsb.org/
FlashAttention-2 (Dao, 2023) to speedup the ESM2 and Galactica model.
C.1 Experimental Details of ProtT3
Hyperparameters. The training stage 1 has 50 epochs and stage 2 has 10 epochs. The batch size is 128 for both stage 1 and stage 2. Q-Former has 8 query tokens (Nq = 8). The optimizer is configured following (Li et al., 2023a). We use an AdamW optimizer and a learning rate scheduler that is a combination of linear warmup and cosine decay. The peak learning rate is 1e − 4 and the warmup has 1000 steps. Weight-decay is set to 0.05. For all experiments that involve fine-
tuning the Galactica1.3B LM, we set LoRA’s rank r to 8 and apply LoRA to Galactica’s weights of [q_proj, v_proj, k_proj, out_proj, fc1, fc2]. This configuration yields LoRA adapters with 7M parameters, constituting merely 0.54% of the parameters in the Galactica1.3B. LoRA is implemented using the OpenDelta (Ding et al., 2022) library. For the text generation training, we truncate all the texts to at most 128 tokens.
Case Studies. The examples in Figure 4 are sampled from the predictions of models in Table 4a. The examples in Figure 5 are sampled from the predictions of models in Table 4b. Specifically, the Galactica model in Figure 4 and Figure 5 refers to
the LoRA fine-tuned Galactica on the corresponding dataset.
C.2 Experimental Details of Baselines and ProtT3 Variants.
LoRA fine-tune Galactica1.3B. We use LoRA finetune Galactica1.3B as a baseline to verify the effectiveness of using a PLM for protein understanding. The hyperparameters are the same as ProtT3: they use the same optimizer, learning rate, batch size, epochs, and LoRA configuration. This baseline is applied directly for training stage 2 without training stage 1 because of its incapability of protein-text retrieval.
ProtT3 w/ MLP Proj. This ProtT3 variant replaces ProtT3’s Q-Former projector with a twolayer MLP following (Liu et al., 2023a). Hyperparameters are the same as ProtT3. This variant includes training stage 2 without stage 1, because the MLP projector cannot process multi-modal inputs, therefore cannot be applied in stage 1.
ProtT3 w/o Stage 1. This ProtT3 variant is to verify the effectiveness of training stage 1 for downstream generation tasks. Hyperparameters are the same as ProtT3.
ProtT3 w/o PTM. This is a variant of ProtT3’s stage 1 model that removes the PTM objective function. Without the PTM objective, ProtT3 relies on retrieval ability of the PTC objective for proteintext retrieval.
ProtT3 w/o PCap. This is a variant of ProtT3’s stage 1 model that removes the PCap objective function.
ProtST (Xu et al., 2023). We re-implement their method on our datasets, using their released codes and the original hyperparameters in their paper. Specifically, we have implemented their ProtST-ESM-2 model and their multi-modal pretraining stage. Following their original paper, we adopt ESM-2650M as the protein encoder and adopt PubMedBERT-abs109M (Gu et al., 2022) as the text encoder. The model is trained on the combination of the Swiss-Prot and ProteinKG25 datasets, which is the same as ProtT3’s training stage 1. The
pretrained model is then applied for protein-text retrieval directly.
ProteinCLAP (Liu et al., 2023b). We reimplement their method from scratch for their source codes are not publicly available. Our implementation largely follows the description in their paper: using SciBERT (Beltagy et al., 2019) for text encoding and ProtBERT (Elnaggar et al., 2021) for protein encoding, learning rate 1e − 5. We have trained the model on the combination of the Swiss-Prot and ProteinKG25 datasets for 50 epochs, which is the same as ProtT3’s training stage 1. The pretrained model is then applied for protein-text retrieval.
ProteinChat. ProteinChat (Guo et al., 2023) is developed using the base LM of Vicuna13B (Zheng et al., 2023). We did not evaluate ProteinChat for the Swiss-Prot and ProteinKG25 datasets because these two datasets do not include proteins’ 3D structures, which is required by ProteinChat’s protein encoder. For the PDB-QA dataset, we evaluate a checkpoint shared by the authors, which is the checkpoint used in their original paper.

heading: D Further Discussions
text: D.1 Intuitions on Q-Former’s Capabilities For clarity, we briefly summarize Q-Former’s capability in the two stages: Stage 1: Protein-Text Retrieval Training.
• Protein-Text Retrieval with Protein-Text Contrasting (PTC) and Protein-Text Matching (PTM). Both of these two pretraining tasks are designed for retrieval: PTC is faster for using cosine similarity to model sequence-level similarity between proteins and texts; PTM is slower but more accurate for using self-attention to model the token-level similarity. The other pretraining task, Protein Captioning, although cannot be applied for retrieval directly, improves Q-Former’s representation learning ability, so as to improve the retrieval performance.
• Warmup before stage 2. The three pretraining
tasks pretrain Q-Former to have multi-modal representation ability of texts and proteins, acting as a warmup training before stage 2.
Stage 2: Protein-to-Text Generation Training.
• Nonlinear Mapping from Protein Space to Text Space. Q-Former maps the protein encoder’s output to the input space of an LM. This innovative approach allows the LM to leverage pre-existing knowledge from the protein encoder, facilitating a deeper understanding of protein structures and functions.
D.2 Discussion on Linear Mapping
Proteins and texts inherently exist within highdimensional spaces, exhibiting intricate and nonlinear semantic relationships. The utilization of a mere linear layer to map between protein and text domains often proves insufficient, potentially resulting in a loss of critical structural details of proteins or inaccuracies in text representatio. This claim is supported by the observation in Table 6, where a baseline using a linear projector significantly underpforms the others.
It is important to note, however, that applying a linear layer between the protein encoder and the LM does not necessarily produce a strictly linear mapping from proteins to texts. Should either the protein encoder or the LM undergo fine-tuning — as opposed to remaining frozen — their weights can adapt to facilitate a nonlinear mapping between protein and text. Thus, our critique is not directed at the use of a linear layer per se, but rather at its employment in conjunction with frozen protein encoders and LMs.

heading: E Discussion on Licensing
text: Pretrained Models and Codes. The Galactica (Taylor et al., 2022) model is under the CC BYNC 4.0 license, ESM-2 (Lin et al., 2022) under the MIT License, and Sci-BERT (Beltagy et al., 2019) under the Apache License. Collectively, these licenses permit non-commercial use and redistribution of the models.
Datasets. The Swiss-Prot (Bairoch and Apweiler, 2000) dataset is distributed under the CC BY 4.0 License. ProteinKG25 (Zhang et al., 2022) dataset is released under the MIT License, its original source GeneOntology (Aleksander et al., 2023) is under the CC BY 4.0 License. The PDBQA (Guo et al., 2023) dataset is released under
the BSD 3-Clause License, its original source RCSB-PDB is under the CC0 1.0 Universal License. These licenses collectively allow the use and redistribution of the datasets.
Our Licensing Approach. In line with the licensing terms of the codes, models, and datasets we utilized, we will release our pretrained models and codes under the CC BY-NC 4.0 License. For our datasets, we will apply the CC BY 4.0 License.

heading: Future Work
text: This highlights the potential of improving the ability of counting substructures and processing numbers in future studies. Questions of Supplementary Information. Table 10 shows the sampled questions of supplementary information and the candidate answers. We can observe that the questions and answers are not directly relevant to protein structures or functions. Therefore, we infer that the success of these questions often relies on exploiting the dataset’s inherent biases rather than understanding protein structures and properties. This realization prompts a concern that the model’s performance could significantly diminish when applied to datasets without similar biases, indicating a potential area for improvement using retrieval augmentation