PROTLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training (2024)

Abstract: We propose PROTLLM, a versatile crossmodal large language model (LLM) for both protein-centric and protein-language tasks. PROTLLM features a unique dynamic protein mounting mechanism, enabling it to handle complex inputs where the natural language text is interspersed with an arbitrary number of proteins. Besides, we propose the proteinas-word language modeling approach to train PROTLLM. By developing a specialized protein vocabulary, we equip the model with the capability to predict not just natural language but also proteins from a vast pool of candidates. Additionally, we construct a largescale interleaved protein-text dataset, named InterPT, for pre-training. This dataset comprehensively encompasses both (1) structured data sources like protein annotations and (2) unstructured data sources like biological research papers, thereby endowing PROTLLM with crucial knowledge for understanding proteins. We evaluate PROTLLM on classic supervised protein-centric tasks and explore its novel protein-language applications. Experimental results demonstrate that PROTLLM not only achieves superior performance against proteinspecialized baselines on protein-centric tasks but also induces zero-shot and in-context learning capabilities on protein-language tasks.

heading: 1 Introduction
text: Understanding proteins is essential for unraveling the mysteries of life and enabling artificial intelligence systems to advance bioscience research (Wang et al., 2023a). Thanks to the development of deep learning techniques, neural network models encompass extensive protein-centric applications, such as protein-folding prediction (Jumper et al., 2021), protein-protein interaction analysis (Li et al., 2018; Su et al., 2023), function prediction (Zhang et al., 2023a), etc.
*Equal contribution. †Corresponding author.
MV_ST_KTS_KE….
Protein Encoder
MVKSTSK….
__K__S___T__….
Protein Encoder
[Text description]
Language Encoder
Protein sequence modeling (e.g., ESM)
Paired protein-text training (e.g., ProtST)
Protein-as-word pre-training on interleaved protein-text articles (this work)
Protein representation learning methods typically employ large-scale pre-training, which learns unsupervised protein representations on massive protein sequences with masked language modeling (Rives et al., 2021), or autoregressive language modeling (Elnaggar et al., 2020). In addition to protein-centric tasks, recent studies have attempted to extend protein models to proteinlanguage scenarios. ProtST (Xu et al., 2023b) integrates textual information into the protein encoder through multimodal pre-training on protein-text pairs, achieving zero-shot text-to-protein retrieval. Fang et al. (2023) introduces an instruction dataset tailored for the biomolecular domain and investigates how fine-tuned LLM performs on proteindomain instruction-following tasks, such as function description generation.
Despite the success of protein representation methods on specific tasks, developing a model that excels in both protein-centric and protein-language
8950
tasks is still under-explored, facing three main challenges. Firstly, architectures are designed for particular downstream tasks, making it difficult to accommodate a wide range of tasks simultaneously. Secondly, current methods primarily derive cross-modal supervision from explicitly annotated protein-text pairs, which is not scalable to largescale pre-training. Lastly, supporting a variable number of proteins in the input sequence introduces computational uncertainty in each training step, leading to inefficiencies during pre-training.
In this work, we propose PROTLLM, which is a versatile LLM for both protein-centric and protein-language tasks. Instead of designing for specific tasks, PROTLLM supports complex interleaved protein-text inputs and outputs, which enables our model to simultaneously handle diverse downstream tasks without re-designing taskspecific architecture (see Figure 1 for illustrations). Specifically, our dynamic protein mounting mechanism enables the model to seamlessly process text interspersed with an arbitrary number of proteins. Besides, we propose protein-as-word language modeling to ensure interleaved protein-text outputs. By building a protein vocabulary, PROTLLM is trained to autoregressively predict words and proteins from their respective vocabularies.
Additionally, we present a large-scale interleaved protein-text dataset, named InterPT, for PROTLLM pre-training. InterPT is constructed from diverse data sources, consisting of both structured data such as paired protein annotation data, and unstructured data from biological research papers, which encourages PROTLLM to harness crucial knowledge from the scientific articles.
We conduct extensive experiments on a wide range of downstream tasks, ranging from classic supervised protein-centric tasks to novel proteinlanguage applications. Experimental results demonstrate that PROTLLM outperforms specialized baselines on protein-centric tasks. PROTLLM also unlocks the in-context learning capability for protein-protein interaction prediction, and achieves zero-shot text-guided functional protein retrieval.
Our contributions are as follows:
• We propose PROTLLM, a versatile crossmodal LLM for both protein-centric and protein-language tasks. PROTLLM could process complex interleaved protein-text inputs and outputs, thereby supporting diverse tasks.
• We introduce a large-scale pre-training
dataset, InterPT, interleaving proteins and text from both structured data sources and unstructured multi-protein scientific articles.
• We show that PROTLLM achieves superior results on protein-centric tasks against proteinspecialized baselines, and induces zero-shot and in-context learning capabilities.

heading: 2 Related Work

heading: 2.1 Large Language Models
text: The evolution of LLMs has been a cornerstone in the field of natural language processing , showcasing extraordinary capabilities across a broad spectrum of tasks (Devlin et al., 2018; Raffel et al., 2020; Brown et al., 2020; OpenAI, 2023; Touvron et al., 2023; Longpre et al., 2023; Chowdhery et al., 2022). These models, once thought to be limited to text-based tasks, have now crossed boundaries into areas traditionally dominated by human expertise, including mathematical problemsolving (Wei et al., 2022; Imani et al., 2023), drug discovery (Liang et al., 2023; Liu et al., 2023b), and complex decision making (Yu et al., 2023; Ma et al., 2023). Recent explorations further extend LLMs’ expertise into the multimodal domain where they demonstrate significant promise in processing and generating content from diverse modalities (Huang et al., 2023; Zhu et al., 2023; Liu et al., 2023a; Zhao et al., 2023; Wu et al., 2023). Most of these works focus on aligning pre-trained encoders from various modalities with LLMs through instruction tuning, thus equipping LLMs to interpret multimodal inputs. In the realm of scientific research, specialized molecular LLMs have been devised for tasks like molecular property prediction (Liu et al., 2023c; Pei et al., 2023), captioning (Fang et al., 2023), and retrieval (Liu et al., 2023d). Despite these advances, the progress in protein understanding with LLMs lags, hindered by the scarcity of comprehensive datasets for alignment and the absence of efficient architectures to model protein-language sequences.

heading: 2.2 Protein Representation Learning
text: Current mainstream methods for protein understanding tasks have focused on protein representation learning. Protein language models (PLMs) (Elnaggar et al., 2020; Rives et al., 2021; Meier et al., 2021; Lin et al., 2022) have marked significant progress in the area by training the protein sequence encoders on massive protein sequence data. Protein structure encoding methods aim to
learn coarse-grained amino-acid-level representations (Gligorijević et al., 2021; Fan et al., 2022; Zhang et al., 2023a; Xu et al., 2023a) or finegrained atom-level representations (Hermosilla et al., 2021; Jing et al., 2021; Zhang et al., 2023b). Despite the success in protein modeling, proteinrelated text data are left unexplored, which contains valuable supervision signals crucial for understanding proteins. To enhance protein understanding with text supervision, OntoProtein (Zhang et al., 2022a) leverages knowledge graphs, utilizing gene ontology annotations to implicitly enrich protein representation with textual information. ProtST (Xu et al., 2023b) integrates textual information into the protein encoder through multimodal pre-training on protein-text pairs, achieving zero-shot text-to-protein retrieval.
Mol-Instruction (Fang et al., 2023) introduces a comprehensive instruction dataset specialized for biomolecules and further fine-tunes LLMs on this dataset. Similarly, InstructProtein (Wang et al., 2023b) improves the quality of instruction datasets by sampling protein-text pairs from a structured knowledge graph. This line of work focuses on aligning protein with human language using LLMs. However, a limitation of these approaches lies in their direct incorporation of protein sequences into LLMs as text, leading to suboptimal protein modeling due to the LLMs not being pre-trained on extensive protein sequence datasets. In contrast, PROTLLM provides a versatile framework that excels in both classic protein-centric tasks and novel protein-text applications.

heading: 3 Methods
text: In this section, we elaborate on our proposed method, PROTLLM, which is illustrated in Figure 2. Initially, we detail the model architecture in Section 3.1. Subsequently, the pre-training strategy is explained, introducing the concept of proteinas-word modeling, as outlined in Section 3.2. We then present the uniquely constructed interleaved protein-text dataset, InterPT, in Section 3.3. Lastly, we explore the application of PROTLLM on a variety of tasks in Section 3.4.

heading: 3.1 PROTLLM Framework
text: Model architecture PROTLLM consists of an LLM for natural language modeling, a protein encoder, and cross-modal connectors that connect the protein encoder and the LLM. We use
LLaMA-7b (Touvron et al., 2023) as the backbone of PROTLLM, which is an autoregressive Transformer language model pre-trained on large-scale natural language data. To make PROTLLM understand protein sequences (i.e., sequences of amino acid tokens, which are the primary structure of proteins), we employ ProtST (Xu et al., 2023b) as the protein encoder. ProtST follows the backbone architecture of ESM-2 (Lin et al., 2022) and introduces an additional two-layer MLP projection head. Pre-trained on large-scale protein-text pairs with contrastive learning, ProtST learns protein representations that are well-aligned with text. Besides, we introduce cross-modal connectors that bidirectionally connect the LLM with the protein encoder, thereby enabling PROTLLM to accept multimodal inputs and outputs. Specifically, PROTLLM has two cross-modal connector layers, which are placed at the input layer and the output layer of the LLM, respectively. The input-layer connector is a trainable projection matrix and transforms the output vectors from the protein representation space to the LLM representation space. Similarly, the outputlayer connector transforms the LLM output vectors back to the protein representation space. Significantly, the output-layer connector also serves as a prediction head, allowing our model to perform protein retrieval and multi-choice protein answering tasks without requiring the LLM to generate complicated protein names.
Dynamic protein mounting PROTLLM considers not only structured protein-text paired data but also free-form interleaved protein-text sequences. Although the widely used encoder-decoder architecture can handle paired data, it encounters difficulties when dealing with interleaved proteintext inputs with multiple proteins. Therefore, we propose dynamic protein mounting, which allows PROTLLM to accept an arbitrary number of proteins as either input. Specifically, given an input sequence interleaved with proteins and text,
...[text1] [protein1] [text2] [protein2] [text3]...
we do not directly feed the protein sequence to the LLM, but replace sequences with mount points.
...[text1] <PROT> [mount1] </PROT> [text2] ...
At each mount point, we mount the protein encoder to the LLM with the cross-modal connector. Additionally, these mount points are delineated by protein tags, signaling to the LLM that it is receiving protein vector inputs at these positions, rather
Predict the index in protein vocabulary
Predict the index in protein vocabulary
than text data.

heading: 3.2 PROTLLM Pre-Training
text: Protein-as-word language modeling We introduce the protein-as-word language modeling training objective, which unifies protein prediction and word prediction as an autoregressive language modeling task. Consider an input sequence interleaved with n tokens [x1, x2, ..., xn], where the i-th token xi represents either a natural language token or a protein. The protein-as-word language modeling object is to maximize the likelihood:
argmax θ
n∑
i=1
log p(xi|x<i;θ), (1)
where p(xi|x<i;θ) is a categorical probability distribution over a natural language vocabulary when predicting natural words, or a protein vocabulary when predicting proteins. The probability is computed by
p(xi|x<i;θ) ={ softmax(h⊤i ej)j , j ∈ V if xi is word softmax(h⊤i Wvk)k, k ∈ Vp if xi is protein
(2)
where hi is the last-layer LLM hidden states of xi; ej is the word embedding of the word j from the natural language vocabulary V; W stands for the output connector matrix, and vk is the protein embeddings of the protein k from the protein vocabulary Vp. To construct the protein vocabulary, we collect all protein sequences in the training data. We then filter out proteins present in the downstream test sets to prevent data leakage. Finally, we utilize the pre-trained protein encoder to extract protein presentations, resulting in a vocabulary consisting of the 1, 076, 781 proteins. Notably, this protein vocabulary is fixed during pre-training to facilitate the learning of cross-modal connectors. During inference, our ProtLLM can process arbitrary protein as both inputs and outputs, even if it is not part of our pre-established vocabulary.
Pre-training acceleration with protein cache Although our dynamic protein mounting design introduces flexibility for the input format, it also introduces computational uncertainty into the pretraining process, i.e., the computational cost of each step can vary significantly with the number of input proteins. Consequently, the throughput is limited by the worst case, leading to markedly reduced training efficiency. To accelerate the pre-
training, we build a protein cache where we store all the pre-computed protein vectors encoded by the protein encoder. With the protein cache, we eliminate the heavy computational cost of the protein encoder, thereby accelerating the pre-training procedure with stable throughput. Besides, we utilize LoRA (Hu et al., 2022) for efficient training.

heading: 3.3 InterPT: Interleaving Protein-Text Data
text: We propose a large-scale interleaved protein-text multimodal dataset, named InterPT, to pre-train PROTLLM with comprehensive protein-related knowledge. This dataset encompasses three types of data sources, i.e., multi-protein scientific articles, protein-annotation pairs, and protein instructionfollowing data. The statistics of each component are listed in Table 1.
Multi-protein scientific articles Multi-protein scientific articles describe complex relationships among different proteins found in biological research, where each sample could contain multiple proteins. Unlike data presented in structured formats such as pairs or knowledge graphs, these articles offer detailed insights in unstructured natural language. Guided by the recording in the STRING database (Mering et al., 2003) of multiprotein interactions and the scientific articles supporting them, we retrieve all involved articles from the PubMed database (Canese and Weis, 2013), specifically selecting instances where multiple proteins co-occur within the same paragraph. All proteins in these paragraphs are linked to the UniProt database (Consortium, 2015) for their amino acid sequences. Finally, we collect 165K interleaved protein-text sequences from PubMed articles.
Protein-annotation pairs This data maps individual proteins to their textual annotations such as function descriptions. We integrate two data sources, i.e., the UniProt database (Consortium, 2015) and the STRING database (Mering et al., 2003), adding up to 90K protein-annotation pairs. Given such a pair, we utilize it for two tasks, i.e., protein-to-text prediction and text-to-protein pre-
diction, with the probability of 0.8 and 0.2, respectively. Besides, during pre-training, we interleave the data into longer sequences by concatenating multiple pairs into a single sequence, which has two advantages: (1) this operation can bridge the data length gap across different data sources and reduce the number of padding tokens, leading to higher training efficiency; (2) training multiple pairs in a single sequence encourages the model to obtain in-context learning capabilities (Gu et al., 2023).
Protein instruction-following data This data is in the instruction-following style (Ouyang et al., 2022), typically requiring the model to generate open-ended text given a protein and an instruction (Fang et al., 2023). We select the data items of proteins from the Mol-Instructions dataset (Fang et al., 2023) and include them into InterPT. Similar to the processing of protein-annotation pairs, we also concatenate multiple instruction-following data into a single pre-training example, so as to improve training efficiency and acquire in-context learning capabilities.

heading: 3.4 Applying PROTLLM to Diverse Tasks
text: Supervised fine-tuning The best practice for adapting PROTLLM to downstream tasks is supervised fine-tuning when training data are available. Since PROTLLM supports flexible input and output formats, we can simply transform the downstream task data into an interleaved format and directly perform protein-as-word language modeling for supervised fine-tuning. The input and output prompt format for each downstream task can be found in the Appendix A. During fine-tuning, we also apply the LoRA adapter to the LLM for efficient fine-tuning while preventing the model from overfitting several proteins in the training set.
In-context learning In-context learning is a promising capability of LLM, which can adapt the LLM to specific tasks with a few examples without training the model. PROTLLM can achieve in-context learning by pretending a few demonstration examples. To the best of our knowledge, PROTLLM is the first protein-language LLM that is capable of in-context learning.
Instruction-following protein retrieval For another interesting application, PROTLLM can be programmed to execute protein retrieval with customized requirements by following instructions. In Section 4.3, we show that PROTLLM can well re-
trieve functional proteins based only on function descriptions, and it can be further improved by prepending a one-shot demonstration.

heading: 4 Experiments
text: We evaluate PROTLLM on three types of downstream tasks: (1) protein-centric tasks, which include supervised fine-tuning on conventional benchmarks for protein understanding; (2) proteintext in-context learning, where we show the unique ability of PROTLLM by in-context learning on protein-protein interaction prediction; (3) textguided functional protein retrieval, where we conduct a real-world enzyme mining task as a proofof-concept study to validate the retrieval capability of PROTLLM. We present detailed hyperparameters, and prompt templates for pre-training and fine-tuning in Appendix A.

heading: 4.1 Protein-Centric Tasks
text: Setup Following the settings in PEER benchmark (Xu et al., 2022), we adopt three standard tasks in protein understanding to validate our method. Enzyme Commission (EC) number prediction (Gligorijević et al., 2021) aims to predict all possible EC numbers of a protein simultaneously, reflecting the chemical reactions it catalyzes. Gene Ontology (GO) term prediction (Gligorijević et al., 2021) extends as a multi-label classification task, seeking to predict whether a protein belongs to specific GO terms. The GO benchmark is categorized into three branches, namely biological process (BP), molecular function (MF), and cellular component (CC). Protein-Protein Interaction (PPI) prediction aims to determine whether
two given proteins interact or not. We adopt the human PPI dataset (Pan et al., 2010) for experiments.
To evaluate performances on multi-label classification tasks including EC and GO prediction, we report pair-centric area under precision-recall curve (AUPR) values and Fmax, a widely used metric in the CAFA challenges (Radivojac et al., 2013). PPI prediction results are evaluated by mean accuracy. These metrics require the soft probability of each target label. To achieve this, we initially extract the probabilities of “Yes” for the positive label and “No” for the negative label, respectively. Then, these probabilities are normalized via the softmax function to get the final predicted probabilities.
Baselines We compare PROTLLM with seven existing protein representation learning methods. As shown in Table 2, these methods can be categorized into two distinct categories: protein-only approaches and protein-text learning approaches. The former encompasses sequence-based models including ProtBert (Elnaggar et al., 2020), ESM1b (Rives et al., 2021), and ESM-2 (Lin et al., 2022), which are pre-trained using extensive collections of protein sequences, alongside structurebased models, such as DeepFRI (Gligorijević et al., 2021), and GearNet (Zhang et al., 2022b). The latter, protein-text learning approaches, includes OntoProtein (Zhang et al., 2022a), and ProtST (Xu et al., 2023b).
Note that Mol-Instructions (Fang et al., 2023) and InstructProtein (Wang et al., 2023b) also belong to the protein-text learning approaches. However, their methods directly take protein sequences as human language and tokenize the data using byte-pair encoding. Contrasting with the protein-
as-word strategy in PROTLLM, this exponentially increases the context length, rendering the evaluation on tasks with extensive label sets, like EC and GO prediction, or those requiring multiple protein inputs, such as PPI prediction, impractical for their approaches.
Results The results are shown in Table 2. PROTLLM consistently shows competitive or even superior performance compared to both proteinonly and protein-text approaches across all benchmarks, indicating the effectiveness of our proposed framework on conventional close-ended protein understanding tasks. Remarkably, PROTLLM obtain 0.596 Fmax and 0.469 AUPR on GO-CC, which outperforms ProtST by a large margin. As depicted in Section 3.1, PROTLLM directly uses pre-trained ProtST as the protein encoder, with the key difference lying in our LLM decoder and pre-training stage for alignment. By comparing PROTLLM with ProtST, the overall improvements strongly highlight the potential benefits of incorporating richer protein-text information and scaling the size of the language model.
Moreover, PROTLLM also outperforms two structure-based models on GO and PPI prediction despite we only leverage sequence information during training. Protein structures encode rich information and have direct relations to their functions. This opens up a promising direction to further incorporate protein structure into our framework, which we leave for future work.

heading: 4.2 Unlocking In-Context Learning
text: In-context learning is the capability that rapidly adapts the model to specific tasks using only a few annotated demonstration examples, which is originally found in autoregressive language models (Brown et al., 2020) and then is extended to
visual language models (Alayrac et al., 2022). In this section, we investigate whether PROTLLM can achieve in-context learning on the human proteinprotein interaction (PPI) prediction task.
Setup We directly evaluate the pre-trained PROTLLM model on the human PPI task without updating any parameters. For the k-shot in-context learning, we randomly sample k examples from the validation set as the demonstrations and prepend them to each test sequence. Both the demonstration example and test example are prompted with the same template. For example, a one-shot prompted input is as follows:
Do <PROT> [mount1] </PROT> and <PROT> [mount2] </PROT> interact with each other? Yes\n Do <PROT> [mount3] </PROT> and <PROT> [mount4] </PROT> interact with each other?
The protein sequences of demonstration and test examples are first encoded by the protein encoder and then fed to the language model at each mount point. The final answer is predicted by selecting the verbalizer, i.e., “Yes” or “No”, with the higher probability. Besides, to understand how multi-protein pre-training data from scientific articles improves PROTLLM, we also evaluate a variant of our model by removing the multi-protein scientific articles from the pre-training corpora.
Results Figure 3 presents the in-context learning performance on human PPI with varying numbers of demonstration examples. Our model consistently achieves higher PPI accuracy with an increasing number of demonstration examples, demonstrating its effective in-context learning capability for protein-centric tasks. In comparison, the model performs drastically worse upon removing the multi-protein scientific articles, and fails to learn in context with the 2, 6, and 12 demonstrations. We believe that the in-context learning capability of our model could empower biologists to apply it to specialized tasks that lack annotated data, using minimal examples. Our experiments on enzyme mining illustrate a tangible application of in-context learning, as detailed in Section 4.3.

heading: 4.3 Text-Guided Functional Protein Retrieval
text: Setup This experiment aims to study the capability of PROTLLM to retrieve functional proteins based on text prompts and demonstrations. For this purpose, we apply PROTLLM to enzyme mining, which is a critical stage in enzyme and metabolic
engineering pipelines. In this experiment, we evaluate our model on mining carboxylate reductases that transform various ketoacids into their corresponding aldehydes. Four ketoacid reactants, i.e., 2-ketoisovaleric acid (IsoC5), pyruvic acid (C3), 2-ketovaleric acid (C5), and 2-ketooctanoic acid (C8), studied in Mak et al. (2015) are employed for evaluation.
Using a reported enzyme for IsoC5, ketoisovalerate decarboxylase (KIVD) (De La Plaza et al., 2004), as the query, we first search for a pool of enzyme candidates by BLASTp (McGinnis and Madden, 2004), where the pools with the size of 500 and 1000 are respectively tested. We then leverage PROTLLM to retrieve active enzymes from the pool for each reactant in two modes. In the zeroshot retrieval setting, given the prompt:
Identify the enzymes: {Reactant} → Isobutanal. <PROT>
describing the reaction from reactant (IsoC5, C3, C5 or C8) to product, PROTLLM generates a protein embedding at the token <PROT>. Then, we encode all the candidate enzymes as embeddings with the protein encoder. Finally, we utilize this embedding to rank all enzyme candidates by comparing embedding similarity. For in-context learning, we further add a one-shot demonstration of carboxy-
late reductase before the prompt above to facilitate enzyme mining. The demonstration is:
Indentify the enzymes: Oxidizes aldehydes to the corresponding carboxylic acids with a preference for aromatic aldehydes. <PROT> [mount] </PROT>
where the “[mount]” token is represented by the protein embedding of PaoC (Neumann et al., 2009), a typical carboxylate reductase.
Results In Table 3, we report the recall of active enzymes found in Mak et al. (2015) at top 10, 20, and 50 ranked candidates. It is observed that in-context learning outperforms zero-shot retrieval on 18 out of 24 metrics, which verifies that PROTLLM can learn from a few demonstrations and improve its enzyme mining performance based on such knowledge. To study the top-ranked enzymes by PROTLLM more in-depth, we employ AutoDock Vina (Trott and Olson, 2010) to further screen the top-20 enzymes found by in-context learning and pick the one with the lowest Vina energy for visualization. As shown in Figure 4, the lead enzymes selected in this way are all with good properties, possessing high enzyme activity (i.e., high Kcat/KM and Kcat values measured by Mak et al. (2015)) and low binding energy measured by AutoDock Vina. These results altogether prove the effectiveness of PROTLLM on enzyme mining.

heading: 5 Conclusion
text: In this paper, we present PROTLLM, a versatile LLM designed to tackle both protein-centric and protein-language tasks. Through dynamic protein mounting and protein-as-word modeling, PROTLLM adeptly handles complex interleaved protein-text data, seamlessly unifying a wide array of protein tasks via a natural language interface. Besides, we construct a large-scale protein-language pre-training dataset, called InterPT, which encourages the model to learn from diverse data
sources ranging from structured paired data to unstructured multi-protein scientific articles. Extensive experiments demonstrate that PROTLLM not only achieves competitive performance against specialized baselines across standard protein-centric benchmarks but also paves the way for exploring novel protein-language applications.
Limitations
In this paper, we primarily focus on sequence modeling for protein understanding. Nonetheless, PROTLLM is a general interface for the inputs in other modalities. Future research could further extend PROTLLM to additional modalities, such as protein structures and molecular graphs, by incorporating modality-specific encoders. Besides, we would like to explore more novel applications of PROTLLM such as scientific discovery.
Ethics Statement
To the best of our knowledge, the InterPT dataset has been compiled from publicly available sources, carefully avoiding the inclusion of sensitive or private information. The primary focus of PROTLLM is to enhance protein understanding through various downstream tasks, distinguishing it from applications in dialogue systems. This focus inherently limits the potential for generating harmful content, leading to outputs that are inherently more controllable and safer. Nevertheless, there remains a risk that malicious actors could exploit PROTLLM to spread misinformation or mislead users.

heading: Acknowledgements
text: This work is supported by the National Natural Science Foundation of China (No.U21B2009). We thank Zuobai Zhang, Meng Qu, and Kunlun Zhu for the helpful discussions and comments.

heading: A Experimental Details
text: Pre-training We list the detailed pre-training hyperparameters of PROTLLM in Table 4. PROTLLM is pre-trained on 4 NVIDIA A100 GPUs for 10, 000 steps with batch size 256 on the InterPT dataset described in Table 1. We adopt LoRA for efficient training, applying LoRA to all linear models of LLaMA, including [down_proj, up_proj, q_proj, v_proj, k_proj, o_proj, gate_proj]. Notice that only the LoRA weights and the cross-modal connector modules are updated during training.
Fine-tuning PROTLLM is further fine-tuned on various downstream tasks including EC number prediction, GO term prediction, and PPI prediction. Table 5 presents the fine-tuning hyperparameters. We apply LoRA for efficient tuning of the language model weights. The weights of the protein encoder are frozen for the PPI task, and updated for the other tasks, with a learning rate of 2 × 10−5. The handcrafted prompt templates for each task are shown in Table 6. At each [mount] position, we encode the protein sequences with the protein encoder and feed the resulting protein embedding to the language model. For multilabel classification tasks, i.e., GO and EC, we convert the tasks to binary classification tasks for each label. We fill [name] with the label name, and fill [description] with text descriptions associated with this label. Besides, we utilize a resampling strategy during fine-tuning to ensure a uniform distribution of positive and negative labels.

heading: Future Work
text: Future research could further extend PROTLLM to additional modalities, such as protein structures and molecular graphs, by incorporating modality-specific encoders. Besides, we would like to explore more novel applications of PROTLLM such as scientific discovery