InstructProtein: Aligning Human and Protein Language via Knowledge Instruction (2024)

Abstract: Large Language Models (LLMs) have revolutionized the field of natural language processing, but they fall short in comprehending biological sequences such as proteins. To address this challenge, we propose InstructProtein, an innovative LLM that possesses bidirectional generation capabilities in both human and protein languages: (i) taking a protein sequence as input to predict its textual function description and (ii) using natural language to prompt protein sequence generation. To achieve this, we first pre-train an LLM on both protein and natural language corpora, enabling it to comprehend individual languages. Then supervised instruction tuning is employed to facilitate the alignment of these two distinct languages. Herein, we introduce a knowledge graph-based instruction generation framework to construct a high-quality instruction dataset, addressing the annotation imbalance and the absence of instructional signals in the existing protein-text corpus. In particular, the instructions inherit the structural relations between proteins and annotations in knowledge graphs, which empowers our model to engage in the causal modeling of protein functions, akin to the chain-of-thought processes in natural languages. Extensive experiments on bidirectional protein-text generation tasks show that InstructProtein outperforms state-of-theart LLMs by a large margin. Our code is publicly available at https://github.com/HICAIZJU/InstructProtein.

heading: 1 Introduction
text: The landscape of Natural Language Processing (NLP) research, and indeed the broader Artificial Intelligence (AI) community, has recently been revolutionized by generative Large Language Models (LLMs) (Peters et al., 2018; Devlin et al., 2019; Brown et al., 2020), such as ChatGPT (Ouyang
*Corresponding author.
et al., 2022). The expansion of parameter size and training corpora has empowered these models to acquire versatile, general-purpose data representations that seamlessly transcend linguistic tasks encompassing comprehension and generation in a multitude of languages. Beyond natural languages (a.k.a., human languages), recent investigations have illuminated the potential of these LLMs to serve as a versatile interface for processing multimodal data, including but not limited to images, videos and speech (Chen et al., 2021; Reed et al., 2022; Gong et al., 2023; Huang et al., 2023).
However, general LLMs fall short of capturing the intricate realm of biological sequences, a domain abundant with its own unique linguistic nuances. The biological sequences, particularly proteins, represent a distinctive facet of what could be referred to as “life language”, exerting a significant influence on signal transduction pathways, enzymatic catalysis, and gene regulation (Lee and Yaffe, 2016; Huber, 2001; Durek and Walther, 2008; Luzarowski et al., 2021; Jiang et al., 2022). Existing LLMs like ChatGPT or GPT4 (OpenAI, 2023) fail to accurately model the biological sequences, resulting in limitations on protein understanding and generation (AI4Science and Quantum, 2023).
To unlock the potential within LLMs for deciphering proteins, researchers have put rich efforts into developing protein language models (PLMs) (Alley et al., 2019; Elnaggar et al., 2021; Rives et al., 2021; Rao et al., 2021; Lin et al., 2023). These specialized models are tailored to ingest amino acid sequences as inputs, predict protein functionalities, or even design de novo proteins. Notwithstanding, it is crucial to highlight that while PLMs exhibit competence in comprehending amino acid sequences, they are unable to comprehend the intricacies of human languages. Recent studies (Abdine et al., 2023; Luo et al., 2023) can accept both protein sequences and textual descriptions as input, aiming to enhance protein function
1114
prediction. Nevertheless, these endeavors to align the realms of protein and human languages are unidirectional and remain in their nascent stages; they fall short of being able to generate protein sequences based on textual instructions. In essence, there exists an unaddressed void in the current landscape of LLMs, wherein the ability to swiftly traverse between human and protein languages.
To enable an LLM to adeptly comprehend both human and protein languages, we contend that the limitations imposed by existing models primarily stem from their training corpora. Notably, many existing models are trained on either human languages or protein sequences, rendering them proficient in only one of these linguistic realms. This unilateral training approach is insufficient to imbue an LLM with a comprehensive vocabulary encompassing both languages. Moreover, it is important to recognize that the existing protein-text corpus used in previous studies (Luo et al., 2023; Abdine et al., 2023; Xu et al., 2023; Taylor et al., 2022) has its limitations. (1) The imbalance of annotations: Researchers tend to focus on well-studied proteins, leading to a significant disparity in the availability of annotations (Kustatscher et al., 2022). Training LLMs directly on such a corpus introduces model bias, which ultimately results in suboptimal performance. (2) The absence of instructional signals: Protein-related textual content is primarily comprised of descriptive narratives, often devoid of instructional signals specifically designed for training LLMs. This inherent disparity obstructs a holistic understanding of a wide range of tasks, ultimately resulting in subpar zero-shot performance (Wei et al., 2022a). In short, the fundamental hurdle of current LLMs involves curating an elaborate training corpus that seamlessly bridges the gap between human and protein languages.
In this work, we introduce InstructProtein, a pioneering study that aligns human and protein languages through knowledge instruction, enabling an LLM with bidirectional generation capabilities between these two languages. Specifically, to equip LLMs with the ability to understand protein language, InstructProtein adopts a two-step training approach. It initiates with pre-training on protein and natural language corpora, followed by finetuning with the established protein knowledge instruction dataset. To construct such an instruction dataset, we first transform raw protein-text corpora into a structured knowledge graph (KG). Inspired by the idea of chain-of-thoughts, we enrich KG
Loading [MathJax]/extensions/MathMenu.js
with knowledge causal modeling, which involves establishing causal relationships between triples, indicating causality within annotations. We then propose a debiased sampling strategy to select KG triples, effectively addressing the issue of annotation imbalance. Finally, we mimic KG completion tasks, leverage general LLMs to convert KG triples into instructions, and conduct supervised instruction tuning. Extensive experiments have demonstrated that the introduced protein knowledge instructions significantly improve the performance of LLMs on protein understanding and design tasks. Our contributions can be summarized as follows:
1. We propose InstructProtein, an innovative LLM with bidirectional generation between protein and human languages, effectively filling the gap between the two languages.
2. We introduce a protein instruction generation framework based on knowledge graphs, resulting in the first high-quality protein instruction dataset for tuning LLMs.
3. InstructProtein outperforms state-of-the-art LLMs by a substantial margin, representing a crucial advancement toward text-guided protein function prediction and sequence design.

heading: 2 A Closer Look at Annotation Imbalance
text: Much of life science research is dedicated to unraveling the biological functions of proteins. While certain proteins have undergone extensive investigation, there still exist tens of thousands of proteins remaining categorized as understudied. This phenomenon implies an imbalance in protein function annotation. To clearly illustrate this problem, we take the subcellular location as an example, and show its annotation distribution in Figure 1. The
results reveal a notable concentration of research attention on proteins residing in the cytoplasm, while other subcellular locations lack comprehensive labeling and study. The annotation imbalance has a detrimental effect on the performance of existing LLMs. To demonstrate this, we collect the same number of proteins in each subcellular location category from UniProtKB (Consortium, 2019), resulting in 1,808 proteins in total, and prompt LLMs to predict the subcellular location. The outcomes of LLMs are presented in Table 1, from which one can observe that these LLMs are biased in a certain category, due to the annotation imbalance in the training corpus of LLMs.

heading: 3 InstructProtein
text: This section presents the method of InstructProtein. We first pre-train it in a self-supervised manner on natural language and protein sequence datasets respectively, and then conduct supervised tuning using the created knowledge instruction dataset.

heading: 3.1 Multilingual Pre-Training
text: InstructProtein is designed to comprehend both the protein and human languages. An intuitive approach involves incrementally pre-training an LLM using the protein corpus P and text sequences T . Given X = P ∪ T and {x1, x2, . . . , xn} ∈ X , the training objective of a generative LLM (e.g., OPT (Zhang et al., 2022a)) is defined as
L(X ) = ∑
i
logP (xi|xi−k, . . . , xi−1; θ), (1)
where the prediction of each token depends on previous tokens x<i, k is the context window size, and the conditional probability P is modeled using a neural network parameterized by θ.

heading: 3.2 Instruction Tuning
text: After pre-training, the model acquires an extensive comprehension of both natural language and pro-
tein sequences; however, it still falls short in alignment between these two different languages. We fill this gap through supervised instruction tuning.

heading: 3.2.1 Knowledge Instruction Generation
text: We propose an instruction generation method based on KGs and LLMs, aiming to construct a factual, logical, diverse, and well-balanced protein instruction dataset. Figure 2 illustrates the pipeline of three kinds of instruction generation frameworks. Conventional approaches directly utilize LLMs to generate instruction data from seed tasks or raw documents, which may introduce hallucination and bias from internal knowledge of LLMs. In the proposed method, KGs are incorporated as intermediaries to address these limitations. In specific, a KG encompassed with knowledge causal modeling is constructed to provide factual protein knowledge, based on which a debiased sampling strategy is proposed to pick KG triples. It is worth noting that LLMs simply need to accurately translate the triples into instructional data, without assuming any prior knowledge about proteins.
KG Construction. We use UniProtKB as our data source to construct the protein knowledge graph denoted as G = {P,R, T }. Here, P , R, and T are sets of protein sequences, relations, and textual annotations. Note that the textual description of proteins in UniProtKB is structured, making it easy to transform them into a knowledge graph. In pursuit of a high-quality instruction dataset, we augment KG to provide informative relationships. With chain-of-thoughts (Wei et al., 2022b), we recognize that a logical chain also exists within protein annotations. For example, the biological processes in which a protein can participate are intricately linked to its molecular function and subcellular location, with the molecular function itself being influenced by the protein’s domain. To represent this causal chain of protein knowledge, we introduce a novel concept called Knowledge Causal Modeling (KCM). Specifically, a knowledge causal model comprises multiple interconnected triples organized in a directed acyclic graph, where the edge direction signifies causal relationships. This graph organizes the triples, moving from the micro-level, encompassing characteristics of protein sequences (e.g., domains), to the macro-level, encompassing biological functions. In Figure 3, we show an example of KCM retrieved from InterPro (PaysanLafosse et al., 2023) based on a given triple.
KG Triple Sampling. To generate instruction
data, we need to sample triples from the constructed KG. Considering the annotation imbalance problem in the KG, we propose a debiased sampling strategy as an alternative to uniform sampling. In specific, we first cluster proteins based on their sequence and property similarities, and then uniformly pick triples in each cluster. For sequence similarity, we employ MMseqs2 (Steinegger and Söding, 2017) to calculate the editing distance ds(·, ·) (see Appendix B.1.1). For property similarity, since the protein properties are extensive and many of them remain unexplored, we only consider the known annotations in KG. Specifically, given an annotation t and a relation r, we denote Ct = {p : p ∈ P ∧ (p, r, t) ∈ G} and C/t = {p : p ∈ P ∧ (p, r, t) /∈ G} are the protein set based on the presence or absence of t. The basic idea is to maximize agreement within Ct and minimize agreement between Ct and C/t, via optimizing protein KG embeddings. In practice, we minimize a margin-based ranking criterion over the knowledge graph:
L = [γ + dp(pt, t+ r)− dp(p/t, t+ r)], (2)
where pt ∈ Ct, p/t ∈ C/t, γ is the margin, and dp(·, ·) is a dissimilarity measure of properties, which is implemented as the ℓ2-norm.
We define the identity threshold of sequence and property similarities as δp and δs, respectively. We denote two proteins to be similar p1 ≃ p2 as ds(p1, p2) < δs and dp(p1, p2) < δp. C = {C1, . . . , Cm} represents the aggregation of proteins with m clusters, and the cluster Ci can be formulated as: Ci = {p : p ≃ pcenterCi }, where pcenterCi is the center protein of Ci. Then, the proba-
bility of sampling a triple (p, r, t) is:
P ((p, r, t)) = 1 m × 1||Ci|| × 1||p|| , (3)
where p ∈ Ci, ||Ci|| denotes the size of Ci, and ||p|| are the number of annotations on p.
KG Triple to Instruction. By employing the debiased sampling strategy, a large number of wellbalanced KG triples can be sampled. We then translate these triples into instructions. While the generation of creative tasks requires domain knowledge, the KG completion tasks offer a comprehensive template for proposing domain-specific tasks based on triples. Therefore, we simulate KG completion, and employ general LLMs (e.g., ChatGPT) to transform KG triples with retrieved KCM into
instructions, which contain an instruction describing the protein-related task about and an output result reflecting a correct response of the instruction. Figure 3 shows an example of converting the triple to instructions. The detailed implementation is depicted in Figure 7 and Table 6 in Appendix.

heading: 3.2.2 Tuning LLMs with Instructions.
text: Instruction tuning involves further training LLMs in a supervised manner on an instruction dataset comprising of (instruction, output), bridging the gap between the LLMs’ next-word prediction objective and users’ goal of ensuring adherence to human instructions. With the proposed knowledge instruction dataset I, we finetune the pretrained LLM to align the protein and human languages. Given an instruction Z ∈ I and its tokens {x1, x2, . . . , xn} ∈ Z, the training objective is the same as that defined in Eq.(1).

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: The pre-training corpus contains protein sequences from UniRef100 (Suzek et al., 2015) and sentences from PubMed abstracts. Following the methodology described in Section 3.2.1, We first constructed a protein knowledge graph based on UniProt/SwissProt(Consortium, 2019). Specifically, we select nine property fields: biological process, molecular function, cellular component, family, superfamily, domain, conserved site, active site, and binding site. The resulting KG consists of 464,333 proteins as head entities, 58,725 annotations as tail entities, and 5,207,841 triples. Knowledge causal modeling is sourced from the InterPro (Paysan-Lafosse et al., 2023) and Gene Ontology (Aleksander et al., 2023) database, containing 30,446 causal relationships. We then sample 2.8 million triples and use ChatGPT to convert these triples into instructions. Par-
ticular care is required to prevent potential data contamination between training and evaluation data. We use mmseqs2 to cluster proteins with an identity surpassing the 70% threshold, then remove the clusters containing the proteins in the test set, for a total of 19,455 sequences. Detailed experimental setups and discussion on addressing potential data contamination are described in Appendix B.2.

heading: 4.2 Protein Sequence Understanding
text: Datasets and Metrics. We evaluate LLMs on three widely-used protein property classification tasks: (1) Protein Function Annotation, aiming to predict the correct functions of proteins. We choose Gene Ontology (GO) dataset (Gligorijević et al., 2021), which has three branches: molecular function (MF), biological process (BP), and cellular component (CC). This is a "Held-In" task as the training instructions are derived from the GO dataset. (2) Protein Localization Prediction, which involves the prediction of the subcellular location of a given protein. We address two subproblems from DeepLoc (Almagro Armenteros et al., 2017), the subcellular localization prediction (Abbr., Sub) with 10 location categories and the binary localization prediction (Abbr., Bin) with 2 location categories. This is also a "Hold-In" task since cellular component exist in our instruction data. (3) Metal Ion Binding (MIB) Prediction, a binary classification task where the model needs to determine whether there are metal ion-binding sites in the
protein, which is a "Held-Out" task as the proteins and labels are sourced from Protein Data Bank (PDB) (wwp, 2019). We use the dataset from Hu et al. (2022). Detailed downstream task setting can be found in Appendix C
Baselines. We adopt seven state-of-the-art opensource LLMs as the baselines. OPT (Zhang et al., 2022a) and LLaMA (Touvron et al., 2023) are trained on massive text corpus. Alpaca (Taori et al., 2023) and Mol-Instructions (Fang et al., 2023) refer to LLaMA-based LLMs fine-tuned with other human/protein language datasets. Galactica (Taylor et al., 2022), BioMedGPT (Luo et al., 2023), and BioT5 (Pei et al., 2023) are domain-specific LLMs, which are trained on a large corpus of humanity’s scientific knowledge, such as research papers about proteins and genes. Note that, the training corpora of Mol-Instructions, Galactica, BioMedGPT, BioT5 all contain annotations from Swiss-Prot.
Results. We present the evaluation results in Table 2. Compared with all baselines, InstructProtein achieves new state-of-the-art performance on all tasks. There are two key observations. First, InstructProtein clearly outperforms the LLMs (i.e., ChatGPT, LLaMA, Alpaca) which are stemmed from natural language training corpora. These results demonstrate that training with the corpus where proteins and natural language coexist is beneficial to LLMs, enhancing their proficiency in protein language understanding. Second, InstructProtein performs consistently better than Galactica, BioMedGPT, Mol-Instructions, and BioT5 on HeldIn and Held-Out tasks, proving that the model has the ability to generalize both protein sequences and protein-related tasks. The instruction templates of Mol-Instructions are not adequately diverse, thus unable to understand the tasks in the GO and MIB benchmarks, leading to all negative predictions. It is worth noting that in the protein subcellular localization (Bin) task, there exists a severe bias in LLMs, leading to the classification of all proteins into a single group and resulting in the same accuracy of 57.52%.
Since the closed-source models such as ChatGPT, Claude-2 and GPT4 often refused to follow the instructions as illustrated in Appendix 15, we only consider the occasional cases when they responded to and report their results in Table 3. We discover that these models’ predictions solely encompassed cytoplasm, nucleus, cell membrane, and secreted locations in the protein localization prediction task. This finding underscores that annota-
tion imbalance impacts even these closed-source models, further emphasizing the significance of the high-quality instruction dataset. We provide case studies in Figure 16 in Appendix.

heading: 4.3 Protein Sequence Design
text: Generating proteins following human instructions is a highly exciting area of research. With the incorporation of the protein as part of the language capabilities in LLMs, InstructProtein is capable of generating protein sequences. However, the lack of standardized computational metrics to properly assess the quality of proteins generated by LLMs poses challenges for advancing protein generation models. In this study, we present our endeavor to build a computational evaluation framework.

heading: 4.3.1 Instruction-Protein Pairing
text: Datasets and Metrics. We design an instructionprotein pairing task to assess the consistency between the instruction and the generated protein. Specifically, we employ the dataset proposed by Hou et al. (2018) to provide fold-related instructions and proteins. Given a protein p and the corresponding instruction Z0, we randomly sample other n instructions {Z1, Z2, ..., Zn} (n = 9 in this experiment), and the likelihood L of the protein given the various instructions is computed. The minimization of L(p|Zi) at i = 0 signifies a correct pairing, and vice versa.
Results. Table 4 reports the accuracy of the instruction-protein pairing task. One can observe that InstructProtein surpasses the baselines by a large margin. BioMedGPT focuses solely on converting proteins to texts and lacks protein design capabilities. Galactica exhibits limited zero-shot performance in aligning instructions with proteins, since it is trained with narrative protein corpus. Mol-Instructions lacks pre-training on protein corpora, which makes it difficult for the model to distinguish the nuances of proteins, resulting in poor
Loading [MathJax]/extensions/MathMenu.js
results. These results confirm the superiority of our model in instruction-following for protein design.
4.3.2 Protein Sequence De Novo Design Designing proteins with specified structures. We investigate whether InstructProtein could generate new protein sequences that are individually valid and consistent with instructions. SCOPe (Chandonia et al., 2022) classifies protein structures according to the content and organization of secondary structures, including all α-helix, all β-sheet, and the combination of α-helix and β-sheet. We sample 100 sequences from each class and assess the foldability of individual sequences by predicting their corresponding structures using ColabFold (Mirdita et al., 2022; Jumper et al., 2021) and computing the average predicted local distance difference test (pLDDT) across the whole structure (Figure 4 (a)). pLDDT increases with model scale, suggesting that scaling up the parameter size results in the generation of sequences with fewer intrinsically disordered regions. We leverage ESM2 (Lin et al., 2023) as a feature extractor to obtain the generated all αhelix and all β-sheet protein representations, which are then visualized using multi-dimensional scaling (MDS) algorithm (Kruskal, 1964) (Figure 4 (b)). We observe that the representations are divided into two groups according to instructions, indicating the instruction-following ability of the proposed model. We visualize the predicted structure of the proteins with the highest confidence in each class (Figure 4 (c)). To evaluate the novelty of the generated sequences, we utilize HHblits to search for homologs against the Uniclust30 dataset. Our
analysis revealed that the highest identity of each alignment ranged from 0.313 to 0.880, with a onestandard deviation range of 0.437 to 0.732, demonstrating the generated sequences are not merely based on mutation or the combination of existing sequences but exhibit a degree of novelty. These results demonstrate that InstructProtein establishes a close correlation between natural language and protein language, verifying the effectiveness of protein de novo design based on structure-related instructions.
Designing proteins binding with specified ligands. To verify the ability to follow functionrelated instructions, we employ InstructProtein to design heme binders, which are proteins capable of binding to a specific compound, and visualize 3D structures of three generated proteins. In Figure 5, we present the docking result (docked by DiffDock (Corso et al., 2023)), the binding affinity (predicted by Smina (Koes et al., 2013; Trott and Olson, 2010), the lower the better), and the pLDDT score (predicted by ColabFold; the higher the absolute value, the better). We can observe the resulting proteins exhibit notable binding affinity, confirming the efficacy of InstructProtein in heme binder design. We provide more case studies and comparisons with ChatGPT, Claude-2, and MolInstructions in Appendix E.

heading: 4.4 Ablation Study
text: We conduct ablation studies on the sampling strategy and knowledge causal modeling (KCM) used in our knowledge instruction generation method.
From the results in Table 5, we observe that clustering similar proteins in annotation imbalance-related tasks (Location) can effectively improve model performance. However, for tasks where annotation imbalance is not significant (GO), the clustering method based on sequence alone degrade model performance, which is reasonable because this method reduces the frequency of hard samples (proteins with similar sequences but different functions). This problem can be avoided by considering both sequence and property similarities. We compare property clustering methods based on KGE distance and edit distance, and the results prove that KGE has a stronger ability to model property similarity. We also observe that the causal relationship between annotations introduced by KCM improves the performance.

heading: 5 Related Works
text: Large Language Models (LLMs) have achieved breakthrough performance in NLP (Brown et al., 2020; Rae et al., 2021; Hoffmann et al., 2022; Black et al., 2022; Zhang et al., 2022a; Chowdhery et al., 2022; Touvron et al., 2023). However, these LLMs are primarily tailored for human language comprehension, which limits their utility in decoding protein language. To align these two distinct languages, multimodal approaches (Taylor et al., 2022; Abdine et al., 2023; Luo et al., 2023; Fang et al., 2023) train LLMs on a corpus where natural and protein language co-exist. For example, Luo et al.
(2023) integrate protein encoders into LLMs within an encoder-decoder framework. Notwithstanding, these architectures predominantly exhibit a unidirectional cross-modal capability, focusing solely on converting protein language to textual description. Taylor et al. (2022) treats protein language and human language as a unified modality. However, the lack of instruction signals and annotation imbalance hinder the alignment.
Instruction Tuning is a supervised approach to align language models with user intention (Mishra et al., 2022; Wang et al., 2022; Wei et al., 2021; Ouyang et al., 2022). It is worth noting that acquiring large-scale instruction data can be a resourceintensive and time-consuming endeavor, thereby motivating the exploration of automatic data generation techniques. A prevalent strategy (AnabyTavor et al., 2020; Andreas, 2020; Kaushik et al., 2019) involves augmenting existing datasets. Alternatively, several fully automatic datasets have been proposed to eliminate the need for labeled data. Schick and Schütze (2021); Ye et al. (2022) advocate for leveraging pre-trained language models to generate comprehensive labeled datasets from scratch, tailored to predefined tasks. Honovich et al. (2023a),Wang et al. (2023) and Honovich et al. (2023b) used pre-trained LLMs to automatically construct instructions by a handful of examples. Li et al. (2023) proposes to construct instruction data in a way that a LLM generates tasks based on outputs. However, these methods may introduce hallucination and bias into the instruction data. Fang et al. (2023) construct a template-based instruction dataset, which lacks diversity. Inspired by KG-enhanced LLMs (Sun et al., 2021; Liu et al., 2020; Zhang et al., 2022b), we propose knowledge instruction that can construct factual, logical, and diverse instruction datasets.

heading: 6 Conclusion
text: InstructProtein explores the feasibility of bidirectional generation between human and protein languages within a single large language model. Our approach involved the transformation of a raw protein-text corpus into a structured knowledge graph, from which KG triples were sampled and converted into instructions. This KG-based instruction generation method resulted in a high-quality instruction dataset, facilitating the LLM to align protein language with human language.

heading: 7 Limitations
text: It is important to acknowledge that there are some limitations inherent in our model. One such limitation, shared with large language models, is that InstructProtein encounters challenges with handling numerical values. In the field of protein modeling, a significant proportion is dedicated to quantitative tasks, including the determination of 3D structure, stability assessment, and fitness evaluation. Inadequate quantitative language modeling hinders the understanding of proteins and more granular controllable generation.
In the future, we will incorporate a broader spectrum of instructions, including quantitative descriptions, empowering our model to provide quantitative outputs. These developments will open up new avenues for further advancing the integration of protein and human languages, as well as expanding its practical utility in diverse applications.

heading: Acknowledgements
text: We would like to thank all anonymous reviewers for their insightful and invaluable comments. This work is supported by National Natural Science Foundation of China (62302433, 62301480, U23A20496), Zhejiang Provincial “Jianbing” “Lingyan” Research and Development Program of China (2024C01135), Hangzhou West Lake Pearl Project Leading Innovative Youth Team Project (TD2023017), Zhejiang Provincial Natural Science Foundation of China (LQ24F020007) and CCF-Tencent Rhino-Bird Fund (RAGR20230122).

heading: A Detailed Analysis of Protein Understudying Problems
text: Much of life science research is dedicated to unraveling the biological functions of proteins. While certain proteins, such as the well-studied tumor suppressor p53 (Dolgin, 2017), have undergone extensive investigation, tens of thousands of proteins remain categorized as understudied. This classification implies that their biological functions are poorly elucidated, and they lack comprehensive annotation of their molecular properties.
In Figure 6, we present an analysis conducted on UniProtKB/Swiss-Prot, a highly reputable and manually curated protein knowledge repository. Figure 6 (a) depicts the relationship between the distribution of proteins and their annotation scores. These results emphasize the substantial variation in protein distribution corresponding to different annotations. This variance implies that the annotation of proteins is biased. To illustrate this problem more clearly, we analyze the subcellular location annotation. Figure 6 (b) illustrates the distribution of such annotations. The data reveals a notable concentration of research attention on proteins residing in the cytoplasm, with other subcellular locations significantly lacking in comprehensive labeling.

heading: B Detailed Method
text: B.1 In-Context Examples
Knowledge Instruction relies on examples to teach language models understand how to convert information extracted from the knowledge graph into instruction data. Here we provide our example (Figure 7). We notice that when only two examples of different expressions are provided for each KGC task, the language capabilities of LLMs are activated, generating a variety of instruction data as illustrated in Table 6
B.1.1 Sequence Distance Algorithm
We denote A = a1a2 . . . an and B = b1b2 . . . bn as two sequences to be aligned, where n and m are the lengths of A and B, respectively. Before calculating the editing distance, we have to determine the substitution matrix to calculate the replacement score s(·, ·) ∈ (0, 1] and the gap penalty scheme Wk, where k is the gap length. Then the distance
matrix H can be formulated as:
Hi,j = min{Hi−1,j−1 + s(ai, bj); Hi−k,j −Wk; Hi,j−1 −W1; 1}
(4)
where Hk,0 = H0,l = 0 for 0 ≤ k ≤ n and 0 ≤ l ≤ m. We leverage Hn,m/max(n,m) as the sequence distance between A and B.
B.2 Detailed Experimental Setups To learn protein KE embedding, following the TransE approach, we initiate embeddings for entities and relationships through a random initialization procedure. We employ the SGD optimizer with a learning rate of 1.0. The dimensions of entities and relations’ embeddings are set to 200. After 1000 epochs, the loss eventually converges to 0.168. The ℓ2 distance utilized for clustering proteins is set to 1.4. For sequence similarity, we use mmseqs2 (GPL-3.0 license) with –cov-mode 0 –min-seq-id 0.8 parameter.
We perform incremental training on OPT-1.3b. We wrap the protein sequence with <protein> and </protein> and apply character-based tokenization, treating each amino acid as a single token. For text corpus, we tokenize them using the GPT-2 byte level BPE tokenizer. We utilize Pytorch to conduct experiments with 8 32G V100 GPUs. We use a batch size of 128 and a context length of 1,024 tokens. We adopt the Fully Sharded Data Parallel (FSDP) acceleration strategy alongside the fp16 data format. We adopt the AdamW optimizer with β = (0.9, 0.98). We set the weight decay to 0.01 and the dropout rate to 0.1. The learning rate increases to 1e-4 for the first 5000 warming-up steps and decays linearly to 0 for the rest of the training steps. We pre-train InstructProtein for the first 40,000 steps, and fine-tune it with instruction data in the next 20,000 steps.
B.3 Detailed Data Preprocessing An instruction data consists of a sequence of proteins, as well as textual questions and corresponding answers. We consider the data to be contaminated if the protein sequence appears in both the evaluation set and the training set. To avoid such a problem, we filtered proteins based on sequence identity. Specifically, we first collected a total of 9,373 protein sequences used in the test set for all downstream tasks. Then we clustered the proteins
in the training set based on sequence identity (70% as threshold), and deleted each cluster containing the sequence in the test set (a total of 19,455 protein sequences), which ensures that there is no overlap between the proteins in the training and test sets and avoid the potential for data contamination.

heading: C Downstream Task
text: Similar to reading comprehension problems, we transform all data in protein sequence understanding tasks into a Question&Answer (QA) format where each item consists of a protein sequence, a question about that protein, and a list of possible answers. LLMs are required to predict which answers are true. Following Brown et al. (2020), we use a classification approach where, for example, only two outputs ("yes" and "no") are considered and the higher probability one is taken as the model’s prediction.
We list the detailed definition of downstream tasks. {protein} and {label} are used as placeholders. The licenses of all datasets allow scientific research. Dataset statistics are summarized in Table 7.
Subcellular Localization Prediction is a sequence-level classification task. Each input sequence x is mapped to a label y which represents the subcellular location.
• Prompt template (InstructProtein, OPT, LLaMA, Alpha, BioMedGPT, MolInstructions): {protein} Instruction: What cellular components is the protein located in?
• Prompt template (Galactica): {protein} ## Subcellular Location
Table 6: Example of instructions in the Knowledge Instruction dataset. We leverage {protein} as a placeholder for the corresponing protein sequence.
Task Instruction
Family Generation Instruction: Can you provide me with a protein belonging to the secretoglobin family?Output: Sure, here’s a protein from the secretoglobin family: {protein}.
Superfamily Prediction Instruction: Could you please provide information about the superfamily to which
{protein} belongs? Output: {protein} is likely a member of the rho GTPase activation protein superfamily.
Domain Generation
Instruction: Could you provide me with a protein that contains the nad(p)-binding rossmann-fold domain? Output: Certainly, here is a protein that contains the nad(p)-binding rossmann-fold
domain: {protein}.
Family Prediction Instruction: What family does {protein} belong to? Output: The protein belongs to the alpha-1b-glycoprotein/leukocyteimmunoglobulin-
like receptor family.
Function Prediction Instruction: protein Does the protein have the molecular function of zinc ion binding? Output: Yes, based on the information that the protein belongs to the Retinoid X
receptor/HNF4 family, it indeed has the molecular function of zinc ion binding.
Process Prediction
Instruction: I’m interested in learning about the biological process associated with {protein}. Could you provide information on this? Output: Given its association with the Retinoid X receptor/HNF4 family, {protein} is likely involved in the regulation of DNA-templated transcription.
Binding Site Generation Instruction: Provide me with a protein associated with the IQ motif EF-hand binding site. Output: Sure, the protein that interacts with the IQ motif EF-hand binding site and
exhibits a protein binding molecular function is {protein}.
Family Prediction Instruction: Does {protein} belong to nfat activation molecule 1?Output: Based on the record, the answer is yes.
• Label words (sub): {0: "plasma membrane", 1: "cytoplasm", 2: "endoplasmic reticulum", 3: "golgi", 4: "vacuole", 5: "mitochondrion", 6: "nucleus", 7: "peroxisome", 8: "chloroplast", 9: "extracellular"}
• Label words (bin): {0: ["plasma membrane", "golgi", "vacuole", "endoplasmic reticulum"], 1: ["extracellular", "peroxisome", "nucleus", "cytoplasm", "mitochondrion", "chloroplast"]}
Protein Function Annotation is a sequencelevel classification task to annotation protein with functional labels. Each example consists of a protein, a label. They system must predict whether the label belongs to the protein.
• Prompt template: {protein} Instruction: Does the protein associate with label?
• Label words: {0: "No", 1: "Yes"}
Metal Ion Binding Prediction is a sequencelevel classification task to predict whether a protein can bind to ion.
• Prompt template: {protein} Instruction: Does the protein associate with metal ion binding?
• Label words: {0: "No", 1: "Yes"}
Instruction-Protein Pairing Accuracy probe the insturction-following capabilities in protein generation. Protein are decoded under 10 different instructions (9 randomly sampled instructions and 1 true corresponding instruction). The system must predict which one is the most relevant instruction.
• Prompt template: Instruction: I would like a protein that is in {label}. Output: One of the protein that meets the demand is {protein}"

heading: D Additional Experiments
text: D.1 Generalization ability on free-form questions
In addtion to template-based Q&A, We also experiment with creating question in a free-form manner using ChatGPT for the Subcellular Localization task (e.g., “Where within the cell can the protein be found?” or “Which parts of the cell contain the protein?”). As shown in Table 8, comparative analysis
with instruction-tuned baselines demonstrates the model’s ability to generalize across diverse questions.
D.2 Evaluation of protein representations output by InstructProtein
Because the structure of a protein determines its functions, the accuracy of a model’s structure prediction directly reflects its ability to understand function. We conduct analysis using the contact map prediction task from the TAPE (Rao et al., 2019) benchmark. We compared the performance of our model, InstructProtein, with its base model OPT, as well as with the TAPE Transformer, a wellestablished benchmark model for structure-related tasks. We report precision at L/5 (P@L/5) in Table 9. We observe that InstructProtein significantly outperforms its base model (OPT) in understanding protein structure, demonstrating the efficacy of pretraining and instruction tuning stages in enhancing structural understanding. These findings provide valuable insights into the model’s ability to reason about protein structure.
D.3 Ablation study on pre-training and instruction tuning stages
To substantiate the advantages of pre-training and instruction tuning stages, we are providing additional experimental results in Table 10. The pre-training is conducted with the UniRef100 and Pubmed datasets that respectively contain proteins and biomedical literature, while the instruction tuning is conducted using the proposed Knowledge Instruction approach with aligned natural language and protein language corpora. We can observe that the improvement in performance is mainly due to the fine-tuning phase. However, the lack of pre-training for modeling protein language lead to suboptimal performance.

heading: E More Examples
text: We provide more illustrative examples of protein understanding and protein de novo design to show the superiority of our InstructProtein over existing LLMs.
What superfamily does the protein belong to?
What domain does the protein have?
What is the function of this protein?
Does the protein have proteinase, inhibitor i2, kunitz, conserved site?
Instruction: I would like a protein that has cystatin domain. Output:
GoundTruth: • UniProt ID: B2Z449
Design 2: • Length: 115 residues
Design 3: • Length: 107 residues Design 1: • Length: 165 residues
Instruction: I would like a protein that is in metallothionein family. Output:
GoundTruth: • UniProt ID: A0A024R6T4
Design 2: • Length: 61 residues
Design 3: • Length: 54 residues Design 1: • Length: 61 residues
Instruction: I would like a protein that is in retroviral VpR/VpX protein family. Output:
GoundTruth: • UniProt ID: A0A023HIS7
Design 2: • Length: 96 residues
Design 3: • Length: 96 residues Design 1: • Length: 126 residues
Instruction: I would like a protein that is in SsrA-binding protein family. Output:
GoundTruth: • UniProt ID: A0ALD2
Design 2: • Length: 150 residues
Design 3: • Length: 151 residues Design 1: • Length: 153 residues
Instruction: Instruction: I would like a protein that is in kappa casein family. Output:
GoundTruth: • UniProt ID: P02668
Design 2: • Length: 145 residues
Design 3: • Length: 192 residues Design 1: • Length: 98 residues
Figure 10: More examples of family-instruction-based protein de novo design (colored by pLDDT).
Loading [MathJax]/extensions/MathMenu.js

heading: Future Work
text: In the future, we will incorporate a broader spectrum of instructions, including quantitative descriptions, empowering our model to provide quantitative outputs. These developments will open up new avenues for further advancing the integration of protein and human languages, as well as expanding its practical utility in diverse applications