BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuning (2024)

Abstract: Recent research trends in computational biology have increasingly focused on integrating text and bio-entity modeling, especially in the context of molecules and proteins. However, previous efforts like BioT5 faced challenges in generalizing across diverse tasks and lacked a nuanced understanding of molecular structures, particularly in their textual representations (e.g., IUPAC). This paper introduces BioT5+, an extension of the BioT5 framework, tailored to enhance biological research and drug discovery. BioT5+ incorporates several novel features: integration of IUPAC names for molecular understanding, inclusion of extensive bio-text and molecule data from sources like bioRxiv and PubChem, the multi-task instruction tuning for generality across tasks, and a numerical tokenization technique for improved processing of numerical data. These enhancements allow BioT5+ to bridge the gap between molecular representations and their textual descriptions, providing a more holistic understanding of biological entities, and largely improving the grounded reasoning of bio-text and bio-sequences. The model is pre-trained and fine-tuned with a large number of experiments, including 3 types of problems (classification, regression, generation), 15 kinds of tasks, and 21 total benchmark datasets, demonstrating the remarkable performance and stateof-the-art results in most cases. BioT5+ stands out for its ability to capture intricate relationships in biological data, thereby contributing significantly to bioinformatics and computational biology. Our code is available at https: //github.com/QizhiPei/BioT5. * Corresponding authors: Lijun Wu (apeterswu@gmail. com) and Rui Yan (ruiyan@ruc.edu.cn)

text: Findings of the Association for Computational Linguistics ACL 2024, pages 1216–1240 August 11-16, 2024 ©2024 Association for Computational Linguistics
Recent research trends in computational biology have increasingly focused on integrating text and bio-entity modeling, especially in the context of molecules and proteins. However, previous efforts like BioT5 faced challenges in generalizing across diverse tasks and lacked a nuanced understanding of molecular structures, particularly in their textual representations (e.g., IUPAC). This paper introduces BioT5+, an extension of the BioT5 framework, tailored to enhance biological research and drug discovery. BioT5+ incorporates several novel features: integration of IUPAC names for molecular understanding, inclusion of extensive bio-text and molecule data from sources like bioRxiv and PubChem, the multi-task instruction tuning for generality across tasks, and a numerical tokenization technique for improved processing of numerical data. These enhancements allow BioT5+ to bridge the gap between molecular representations and their textual descriptions, providing a more holistic understanding of biological entities, and largely improving the grounded reasoning of bio-text and bio-sequences. The model is pre-trained and fine-tuned with a large number of experiments, including 3 types of problems (classification, regression, generation), 15 kinds of tasks, and 21 total benchmark datasets, demonstrating the remarkable performance and stateof-the-art results in most cases. BioT5+ stands out for its ability to capture intricate relationships in biological data, thereby contributing significantly to bioinformatics and computational biology. Our code is available at https: //github.com/QizhiPei/BioT5.
* Corresponding authors: Lijun Wu (apeterswu@gmail. com) and Rui Yan (ruiyan@ruc.edu.cn)

heading: 1 Introduction
text: Molecules and proteins are two crucial bio-entities in drug discovery, forming the foundation of biological activities (Dara et al., 2022; AI4Science and Quantum, 2023). A molecule can be represented by its SMILES (Weininger, 1988; Weininger et al., 1989) or SELFIES (Krenn et al., 2020) sequence, and a protein can be described by a FASTA (Lipman and Pearson, 1985; Pearson and Lipman, 1988) sequence. With the advancement of Language Models (LMs), an increasing body of work focuses on understanding the molecules and proteins by modeling their biosequences (Chithrananda et al., 2020; Rives et al., 2021; Lin et al., 2022).
Notably, biological literature (Canese and Weis, 2013; White, 2020) is full of extensive information on molecules and proteins. When a biological entity is mentioned in such literature, its context is predominantly centered around a description of some characteristics of the entity. Consequently, there has been a growing body of work dedicated to the joint modeling of text and biological entities (Pei et al., 2024), such as Galactica (Taylor et al., 2022), MolXPT (Liu et al., 2023c), BioT5 (Pei et al., 2023) and BioMedGPT (Luo et al., 2023c), which are all scientific models trained on text, molecule and protein sequences. Despite their achievements, substantial opportunities for enhancement still remain: (1) Prior works neglect the importance of modeling the textual name of molecules, such as International Union of Pure and Applied Chemistry (IUPAC), which provides a standard and systematic naming method for ensuring uniformity and clarity across the scientific community. Different from SMILES and
1216
SELFIES, IUPAC bears a closer resemblance to natural language that is evident in its widespread adoption within scientific literature (Klinger et al., 2008). (2) Previous models were predominantly specialist models, necessitating the training of a separate model for each downstream task, thereby lacking in generality and increasing the training and developing cost (Liu et al., 2023c; Pei et al., 2023). (3) Most of the previous models based on T5 (Raffel et al., 2020) and GPT (Brown et al., 2020) architectures only focus on the classification tasks since they do not implement specialized tokenization for numerical data, which results in their suboptimal adaptation to regression tasks.
To address the above challenges, in this paper, we introduce BioT5+, an advanced iteration of the BioT5 framework (Pei et al., 2023), designed to augment biological research and drug discovery with enriched data integration, multi-task capabilities, and the ability to solve regression tasks. Shortly speaking, BioT5+ incorporates following significant enhancements:
(1) Enhanced Molecule Understanding: By integrating IUPAC name into BioT5+ framework, the model can achieve a deeper comprehension of molecular structures. This integration allows BioT5+ to interpret chemical names as they commonly appear in scientific literature, bridging the gap between formal molecular representations (such as SELFIES) and their textual descriptions. Consequently, this enhances the understanding of molecules and facilitates more accurate predictions and analyses of molecular properties and activities.
(2) Expanded Bio-text and Molecule Data: Compared to BioT5, BioT5+ includes an extensive corpus of bio-text data from sources like bioRxiv (Sever et al., 2019) and PubMed (Canese and Weis, 2013; White, 2020), alongside highquality molecular data from PubChem (Kim et al., 2019). This expansion not only broadens the knowledge base of the model but also enriches the contextual understanding of biological entities.
(3) Multi-task Instruction Tuning: BioT5+ employs multi-task instruction tuning strategy for downstream tasks rather than the separate specialized model training for each task. By leveraging a unified and multi-task training framework, BioT5+ can seamlessly integrate knowledge from diverse tasks, enhancing its predictive power and generalization capabilities across different biological and chemical domains.
(4) Advanced Numerical Tokenization: To over-
come the limitations of the numerical representations, BioT5+ integrates an advanced characterbased numerical tokenization strategy, drawing inspiration from the Llama (Touvron et al., 2023a) model. This technique allows for a more nuanced and consistent representation of numerical values.
With our designed pre-training and multi-task instruction tuning, the effectiveness of BioT5+ is verified on 3 types of problems (classification, generation, and regression), 15 different tasks, and 21 benchmark datasets, including molecule property prediction, retrosynthesis, molecule description generation, drug-target interaction, and so on. BioT5+ has shown highly competitive results, achieving state-of-the-art performance in most of the tasks. This robust performance underscores the enhanced capability of BioT5+ to capture and analyze the intricate relationships and properties inherent in biological data, marking a significant step forward in computational biology.

heading: 2 Related Work

heading: 2.1 Biological Cross-modal Models
text: Recent advancements in LLMs have led to an increased focus on jointly modeling molecules, proteins, and text, aiming to enhance the understanding of bio-entities through text. Molecule-Text. MolT5 (Edwards et al., 2022) is jointly trained on general text and molecule SMILES using T5 (Raffel et al., 2020) masked span prediction objective. MoMu (Su et al., 2022) employs contrastive learning on molecular graphs and related text, and MolFM (Luo et al., 2023b) further incorporates knowledge graph embedding for molecule representation. MolXPT (Liu et al., 2023c) is jointly trained on molecule SMILES and wrapped text using GPT (Brown et al., 2020) framework. MolCA (Liu et al., 2023d) enhances LMs by integrating 2D molecular graph perception through a cross-modal projector and uni-modal adapter. GIT-Mol (Liu et al., 2023a) is a multi-modal LLM that synergizes graphs, images, SMILES, and molecule captions. Text+Chem T5 (Christofidellis et al., 2023) is a multi-domain, multi-task language model capable of concurrently processing molecules and natural language. Protein-Text. Several notable works focus on jointly modeling proteins and text. ProteinDT (Liu et al., 2023b) presents a text-guided protein design framework. BioTranslator (Xu et al., 2023b) is a cross-modal translation system, which can annotate
1217
various biological instances using textual descriptions. Prot2Text (Abdine et al., 2023) combines GNN and LLM in an encoder-decoder framework to generate protein functions in a free-text style.
In addition to the models mentioned above, there are other models trained in a more diverse range of modalities: DeepEIK (Luo et al., 2023a) is a multi-modal model which integrates features from multi-modal inputs including drugs, proteins, and text. BioT5 (Pei et al., 2023) is a T5-based (Raffel et al., 2020) model that undergoes joint training on text, molecule SELFIES, and protein FASTA sequences, effectively bridging the gap between textual and biological data.
Despite the successes of these models, their focus on single-task training limits their versatility and hinders the development of a more generalized and adaptable approach in computational biology.

heading: 2.2 Instruction Tuning for Biological Tasks
text: Instruction tuning is a popular technique applied to pre-trained LLMs where they are trained with specialized instruction datasets, thus equipping LLMs with the ability to understand task-specific instructions. Recently, there has been a growing interest in exploring instruction tuning for various biological tasks. Notable among these efforts is the development of Mol-Instructions (Fang et al., 2023), a comprehensive instruction dataset specifically designed for the biological domain, which includes molecule-oriented instructions, proteinoriented instructions, and biomolecular text instructions. InstructMol (Cao et al., 2023), a multimodal LLM, employs instruction tuning to align molecular graphs, molecule SELFIES, and natural language. Different from these approaches, our BioT5+ is specifically pre-trained for the biological domain. Further instruction tuning within bi-
ological instructions enables BioT5+ to not only understand bio-entities but also generalize across various biological tasks.

heading: 3 BioT5+ Framework
text: This section introduces BioT5+ framework and an overview is shown in Figure 1. The tasks involved in pre-training are presented in Figure 3.

heading: 3.1 Intuition for IUPAC Integration
text: IUPAC naming system provides a standardized set of rules for naming chemical compounds, which allows for the precise description of molecular structures and their components (functional groups, chains, and rings), making it a cornerstone in chemical nomenclature. Typically, an IUPAC name is constructed from the names of individual molecules’ constituent parts, reflecting their structure. This includes prefixes, infixes, and suffixes that indicate various chemical groups and structural features, providing a comprehensive description of the molecule. For instance, the IUPAC name for Aspirin is “2-acetyloxybenzoic acid”. Here, “2- acetyloxy” refers to an acetoxy group attached to the second carbon of a benzene ring, and “benzoic acid” indicates a benzene ring with a carboxylic acid group. The resemblance of IUPAC names to natural language, coupled with their prevalent use in scientific literature, makes them an ideal candidate for model pre-training. By pre-training the model on literature that includes IUPAC names, BioT5+ can establish a nuanced understanding of the relationship between molecules and various textual descriptions of their chemical properties.

heading: 3.2 Pre-training Corpus
text: As an extension of BioT5, the majority of the pretraining corpus for BioT5+ is identical to BioT5,
1218
hence we will briefly mention the common elements while focusing primarily on the novel aspects introduced in BioT5+.
The pre-training corpus consists of 4 classes: (1) Single-modal data, including molecule SELFIES with IUPAC name from PubChem (Kim et al., 2019), molecule SELFIES from ZINC20 (Irwin et al., 2020), protein FASTA from Uniref50 (Suzek et al., 2007), and general text from “Colossal Clean Crawled Corpus” (C4) (Raffel et al., 2020). For the molecule from PubChem, we concatenate the IUPAC name and SELFIES for pre-training as shown in Figure 3. (2) Wrapped text, where the molecule or gene/protein names are suffixed with corresponding sequence representation. We employ BERN2 (Sung et al., 2022), a neural-based Named Entity Recognition (NER) system in the biological domain, to detect and classify occurrences of molecules and proteins within the abstracts of PubMed (White, 2020) and bioRxiv (Sever et al., 2019). For the molecule name, we first standardize the name to its IUPAC name and then append the corresponding SELFIES. For the gene/protein name, we will directly append its FASTA sequence. For the generation of high-quality wrapped text, we also analyze the confidence score distribution predicted by BERN2 (Sung et al., 2022). Only those entities with higher confidence scores were retained to ensure the accuracy and relevance of the appended sequence data. Further detailed descriptions of this process are provided in Appendix Section C. (3) Bio-text, including PubMed (White, 2020) Central full text articles, and bio-texts from PubMed (White, 2020) abstracts and bioRxiv (Sever et al., 2019) abstracts that do not yield identifiable named entities in (2). (4) Molecule-description pairs and protein-description pairs. The molecule-text data is collected from PubChem (Kim et al., 2019) and we also add the IUPAC name to the text description. All molecules and proteins that exist in the downstream MolInstructions dataset (Fang et al., 2023) and ChEBI20 (Edwards et al., 2022) are excluded to prevent data leakage. The protein-text data is the same as BioT5 (Pei et al., 2023). Remarkable Difference. The primary distinctions between BioT5+ and BioT5 are as follows: (1) BioT5+ integrates IUPAC in the molecular pretraining data, encompassing IUPAC names combined with SELFIES, wrapped text, and moleculetext translation data. More details are in Appendix Section D. (2) BioT5+ incorporates a broader spec-
trum of high-quality data, including IUPAC names and SELFIES from PubChem, as well as comprehensive articles from bioRxiv and PubMed Central.

heading: 3.3 Tokenization
text: BioT5 (Pei et al., 2023) has already demonstrated the advantages of employing separate tokenization and embedding techniques. BioT5+ inherits this advantage to apply specialized tokenization and vocabulary specifically for bio-entities. This method explicitly differentiates between the biological semantic space and the textual semantic space. For molecule SELFIES, each chemically meaningful atom group, naturally distinguished from textual vocabulary due to its bracketed format like [C], is tokenized as an individual token using the inherent token set defined by SELFIES. For protein FASTA sequences, to ensure a clear modal distinction, each amino acid is tokenized into a separate token with the prefix <p>, differentiating them from standard upper-case English letters.
Concurrently, the tokenization of numerical data is worth dedicated consideration and design. The direct application of T5 (Raffel et al., 2020) dictionary derived from nature language using SentencePiece (Kudo and Richardson, 2018) for numerical tokenization can lead to inconsistencies (Liu and Low, 2023). For instance, the number 1024 might be tokenized into “10” and “24”, while “2048” could be split into “2”, “0”, and “48”. This irregular segmentation poses a challenge for the model in consistently mapping embeddings to numbers, especially when the number of digits they represent varies. In contrast, models like Llama (Touvron et al., 2023a,b) and ChatGLM (et.al., 2023) adopt a character-based approach to numerical tokenization, where each digit is tokenized as an individual token. This method has been demonstrated to yield superior results in various arithmetic tasks (Liu and Low, 2023; Nogueira et al., 2021). Accordingly, in BioT5+ we also implement this character-based approach for numerical tokenization without modifying the original dictionary. The efficacy of this method over the original T5 (Raffel et al., 2020) and BioT5 (Pei et al., 2023) to numerical tokenization are shown in Section 4.3, providing empirical evidence of its superior performance in handling numerical data.

heading: 3.4 Model and Training
text: Model architecture. BioT5+ adopts the same architecture as the BioT5 (Pei et al., 2023), which
1219
follows the T5-v1.1-base1 configuration with vocab size 35, 076 and 252M parameters. Pre-training. Based on the pre-training corpus described in Section 3.2, the pre-training for BioT5+ is conducted in a multi-task way with eight tasks that fall into 4 categories: (1) Modality-Specific T5 Objectives: This category involves the application of the T5 objective (masked span prediction) to each modality in isolation, including molecule SELFIES with IUPAC name (Task #1), molecule SELFIES (Task #2), protein FASTA sequences (Task #3), and general textual content (Task #4). (2) T5 Objectives on Wrapped Text: Applying the T5 objective to “wrapped” text extracted from scientific corpora (Task #5). (3) T5 Objectives on Biotext: Applying the T5 objective to text in biological domain (Task #6). (4) Bidirectional Translation Tasks: This involves the bidirectional translation between molecule SELFIES-text pairs (Task #7) and protein FASTA-text pairs (Task #8). Through these strategically structured pre-training tasks, BioT5+ is adept at learning the intricate relationships and characteristics of bio-entities as represented in textual information. Multi-task Instruction-based Fine-tuning. After the comprehensive pre-training phase, BioT5+ undergoes multi-task instruction-based fine-tuning. Unlike BioT5 where each downstream task has a specialized fine-tuned model, we follow Fang et al., 2023 and Cao et al., 2023 to categorize downstream tasks and conduct multi-task instruction tuning, which not only saves the repeated tuning cost but also eases the model deployment for the evaluation of multiple tasks. The relevant groupings and information about benchmark tasks and datasets are illustrated in Figure 1, which is simply split by the domains, e.g., molecule-oriented or protein-oriented tasks. This methodology serves a dual purpose: firstly, it bridges the gap between the pre-training and fine-tuning phases, ensuring a smoother transition and integration of learned capabilities. Secondly, it activates and harnesses the general capabilities of BioT5+ across various tasks, demonstrating its versatility and adaptability in handling diverse biological problems.

heading: 4 Experiments and Results
text: As shown in Figure 1, BioT5+ is extensively evaluated across 21 well-established downstream
1https://huggingface.co/docs/transformers/ model_doc/t5v1.1
benchmark datasets, which can be classified into 7 molecule-oriented tasks and 8 protein-oriented tasks with 3 types of problems: classification, regression, and generation. Following Fang et al., 2023, we categorize downstream tasks into different groups for multi-task instruction tuning in the same way, and details about the downstream datasets and baselines are in Appendix Section G.

heading: 4.1 Molecule-oriented Tasks
text: The molecule-oriented tasks cover different topics. As we incorporate IUPAC name for molecule in the pre-training, we also use IUPAC in some moleculeoriented tasks, such as molecule property prediction and molecule description generation. More details are in the following sections and Appendix.

heading: 4.1.1 Molecule Property Prediction
text: Molecule property prediction is a crucial task in bioinformatics, focusing on the determination of specific properties exhibited by a given molecule. Following Cao et al., 2023, we explore the ability of BioT5+ on MoleculeNet (Wu et al., 2018) benchmark. For classification tasks, we focus on 4 benchmark datasets: BACE, BBBP, HIV, and Clintox. Each sample includes an instruction detailing the property to be predicted and the molecule SELFIES with IUPAC name, with models required to generate a simple “yes” or “no” prediction. For regression tasks, we focus on 3 regression benchmarks from QM9 dataset, which aims to predict quantum mechanical properties of molecules, based on the
1220
molecule SELFIES with IUPAC name, including HUMO, LUMO, and the HUMO-LUMO gap.
Results. The results for classification and regression tasks are shown in Table 1 and Table 3 respectively. BioT5+ demonstrates superior performance over other generalist model baselines. Notably, for classification tasks, BioT5+ surpasses models like Galactica (Taylor et al., 2022), which is extensively trained on a vast corpus of scientific literature. Similarly, InstructMol (Cao et al., 2023), despite its inclusion of 2D graph information and LLMs, BioT5+ outperforms on both classification and regression tasks. This enhanced performance can be attributed to the integration of IUPAC names, wrapped text, bio-text, and molecule-text pairs in BioT5+ pre-training. The presence of molecule property descriptions in the context of these diverse corpora allows the model to acquire a comprehensive understanding of molecular properties. However, when compared to single-task specialist models, BioT5+ showed some gaps. This discrepancy is understandable and can be attributed partly to the ease of tuning inherent in single-task models and partly to the fact that some baselines incorporated additional molecular information, such as 2D and 3D structures.

heading: 4.1.2 Chemical Reaction-related Tasks
text: In computational chemistry, tasks related to chemical reactions are of vital importance as they can speed up development processes. Following Cao et al., 2023, we focus on 3 such tasks: reagent prediction, forward reaction prediction, and retrosynthesis. Results. The main results are presented in Table 2 and full results are in Table 10. While LLMs have been exposed to some molecular data during pretraining, their direct zero-shot testing on chemical reaction-related tasks demonstrated extremely poor performance. Mol-Instructions (Fang et al., 2023) conducts multi-task instruction tuning based on Llama (Touvron et al., 2023a) with moleculeoriented tasks. InstructMol (Cao et al., 2023) introduces a molecule graph encoder to encode 2D molecular graph information for Vicuna (Chiang et al., 2023). Our BioT5+ follows the same training setting with Mol-Instructions (Fang et al., 2023) and shows superior performance across almost all metrics on chemical reaction-related tasks. This outcome demonstrates the effectiveness of joint pre-training on both molecular and textual data.

heading: 4.1.3 Molecule Description Generation
text: The objective of molecule description generation is to generate a detailed and informative description for a given molecule. To be consistent with BioT5+ pre-training, the input here also consists of molecule SELFIES with IUPAC. Unlike molecule property prediction, which often focuses on specific attributes, molecule description generation involves interpreting and conveying a comprehensive narrative of the molecule. This narrative encompasses not only its molecular composition and properties but also its potential applications and roles,
1221
as derived from the integration of SELFIES representations and IUPAC names. We use the same evaluation metrics as Fang et al., 2023.
Results. As presented in Table 4, our BioT5+ outperforms all compared single-task specialist, retrieval-based LLMs, and multi-task generalist baselines. This superior performance can be attributed to the comprehensive learning during the pre-training of BioT5+. The model has effectively assimilated a multi-dimensional and rich textual description of molecules.

heading: 4.1.4 Description-guided Molecule Design
text: Description-guided molecule design is essentially the inverse task of molecule description generation, which requires generating a molecule based on a provided textual description. In BioT5+ setting, we do not include IUPAC name in the textual description of the molecule to prevent the model from learning a simplistic mapping from IUPAC name to its SELFIES representation, thereby ensuring the model does not overlook the other descriptive elements provided in the text. Results. Table 5 presents the results for descriptionguided molecule design task. Our BioT5+ surpasses all the compared baselines. This achievement underscores the efficacy of BioT5+ pretraining, where the model has acquired a profound understanding of molecular knowledge.

heading: 4.2 Protein-oriented Tasks

heading: 4.2.1 Protein Description Generation
text: The task of protein description generation involves deriving relevant textual information from a given protein sequence. Following Fang et al., 2023, we mainly focus on 4 related generation tasks: protein function generation, catalytic activity generation,
1222
Table 7: Ablation of IUPAC and additional data on molecule description generation task with single task setting. B-2 stands for BLEU-2 and R-1 is ROUGE-1.
MODEL B-2↑ B-4↑ R-1↑ R-2↑ R-L↑ METEOR↑ BioT5+(single task) 0.671 0.597 0.715 0.590 0.655 0.687 BioT5+(single task) wo IUPAC 0.661 0.584 0.706 0.578 0.647 0.677 BioT5+(single task) wo additional data 0.666 0.591 0.711 0.586 0.651 0.681
BioT5+ 0.666 0.591 0.710 0.584 0.650 0.681
D
P
C
F 0.06
0.23625
0.72
Protein Function
Functional Description
Catalytic Activity
Domain/Motif
Alpaca Baize ChatGLM Galactica
Llama Mol-Instructions Vicuna BioT5+
Figure 2: Performance (ROUGE-L) comparison on protein description generation tasks.
domain/motif generation, and functional description generation. Results. As shown in the Figure 2, BioT5+ surpasses all compared baselines across 4 tasks. This result highlights the advanced ability of BioT5+ to interpret complex protein sequences into meaningful textual information, indicating that BioT5+ gains a comprehensive understanding of protein structures and functions through pre-training.

heading: 4.2.2 Protein Property Prediction
text: Protein property prediction task involves predicting specific properties of proteins, such as solubility, structure, or function, based on their amino acid sequences. Following BioT5 (Pei et al., 2023), we focus on 2 protein property prediction tasks from PEER (Xu et al., 2022) benchmark, which is specifically designed for protein sequence understanding: (1) Solubility prediction: whether the input protein is soluble or not. (2) Localization prediction: either the input protein is “membrane-bound” or “soluble”. Both of these tasks are binary classification tasks and the model needs to generate a “yes” or “no” prediction. The results are summarized together with the ones in the following Section 4.2.3.

heading: 4.2.3 Protein-related Interaction Prediction
text: In drug discovery, the prediction of interactions between bio-entities is very important, from which
Protein-Protein Interaction (PPI) and Drug-Target Interactions (DTI) are two key examples. These two tasks are essential for understanding biological processes and identifying potential therapeutic targets. To facilitate this, we follow Pei et al., 2023 to incorporate 2 PPI dataset from PEER (Xu et al., 2022) benchmark including Yeast and Human, and 3 DTI datasets including BioSNAP (Zitnik et al., 2018), Human (Liu et al., 2015; Chen et al., 2020), and BindingDB (Liu et al., 2007). Results. As shown in Table 6, in the PEER benchmark, our BioT5+ demonstrates exceptional performance, surpassing other multi-task models in 3 out of the 4 tasks and achieving results comparable to single-task specialist models. Notably, in the Yeast PPI prediction task, BioT5+ exceeded the performance of all baseline models. This is particularly significant considering that the baseline ESM-1b (Rives et al., 2021) was specifically pre-trained on a vast array of protein sequences and possesses more than double the number of parameters compared to BioT5+. Furthermore, BioT5+ also showed superior performance in DTI tasks as in Table 8 (full results in Table 11), consistently outperforming other methods on the BioSNAP and BindingDB datasets. It is noteworthy that many baseline methods involved specialized designs for molecule and protein encoders. These results underscore the effectiveness of the joint pre-training of BioT5+ on bio-text, molecules, and proteins. This comprehensive understanding is evident in the ability of BioT5+ to accurately predict protein properties, interactions, and drug-target interactions, making it a valuable tool in the field of computational biology.

heading: 4.2.4 Description-guided Protein Design
text: For description-guided protein design, the model needs to generate protein amino acid sequences based on specific design requirements, such as protein structures and functions. Due to the absence of a well-established benchmark for this task, we present in Appendix Table 20 a selection of test cases along with their corresponding sequence similarity scores to provide a direct
1223
comparison between our model and existing models like Galactica (Taylor et al., 2022) and MolInstructions (Fang et al., 2023).

heading: 4.3 Ablation Studies
text: In this section, we conduct ablation studies to investigate the effectiveness of our design in BioT5+. Specifically, we focus on the following 3 scenarios: (1) Do not incorporate IUPAC name of the molecule. As shown in Table 7, removing the IUPAC name results in a noticeable performance drop in the molecule description generation task. This decline highlights the significant role that IUPAC name plays in tasks related to molecular understanding. (2) Do not add PubMed Central and bioRxiv data in pre-training. Results in Table 7 and Table 13 indicate that these two datasets play a crucial role in enhancing molecular understanding. The omission of them leads to a slight but noticeable decrease in performance on molecule description generation and descriptionguided molecule design tasks. (3) Use T5 default tokenizer for numbers instead of characterbased tokenizer. The results in Table 9 demonstrate that the character-based approach for tokenizing numbers is more effective than the default T5 tokenizer on regression tasks. We also conduct an ablation study to further contrast single-task and multi-task tuning strategies in Appendix Section F.

heading: 5 Conclusions
text: BioT5+, as an advanced iteration of the BioT5 framework, represents a significant stride in computational biology and drug discovery. By integrating IUPAC names, expanding bio-text and molecular data sources, employing multi-task instruction tuning, and incorporating an advanced numerical tokenization technique, BioT5+ has successfully bridged the gap between molecular representations and their textual descriptions. The enhanced understanding of molecular structures and its ability to process complex biological data have been demonstrated across a wide range of tasks, with BioT5+ achieving state-of-the-art performance in most of them. This success highlights the potential of BioT5+ as a versatile and powerful tool in understanding and analyzing biological entities.

heading: 6 Limitations
text: Despite the significant advancements achieved by BioT5+, there remain certain limitations that need
to be addressed in future work. Firstly, the model faces challenges in generalizing across various biological tasks, a problem that is distinct from common NLP settings. The intricate and unique nature of each biological task makes it difficult to develop a one-size-fits-all solution, highlighting the need for more specialized approaches within this domain. Secondly, the current scale of BioT5+ is somewhat limited and cannot comprehend and integrate information from other modalities, such as images, restricting its applicability in multi-modal biological data analysis. BioT5+ is not equipped to function as a universal chatbot or to answer queries spanning general domain questions outside the specific scope of biology. This constraint highlights the need for developing larger, more versatile models capable of handling a wider range of data types and answering a broader array of questions in both biological and general domains.

heading: 7 Ethical Considerations
text: While BioT5+ presents a significant advancement, its capabilities, particularly in generating molecules based on textual descriptions and predicting chemical reaction products, raise important ethical considerations. One of the concerns is the potential misuse of this technology to generate harmful or dangerous molecules, which could pose risks to public safety and environmental health. Moreover, the ability of BioT5+ to predict and generate novel molecules may also lead to issues surrounding intellectual property rights and patenting. The ease with which new compounds can be designed and synthesized using AI-driven methods could potentially disrupt traditional research practices and raise questions about the ownership of these discoveries.

heading: 8 Acknowledgements
text: This work was supported by the National Natural Science Foundation of China (NSFC Grant No. 62122089), Beijing Outstanding Young Scientist Program NO. BJJWZYJH012019100020098, and Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China, the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China. Qizhi Pei is supported by the Outstanding Innovative Talents Cultivation Funded Programs 2023 of Renmin University of China.
1224

heading: A Additional Results
text: For the molecule description generation task and description-guided molecule design task, we also compare the performance of BioT5+ with baseline methods on Mol-Instruction (Fang et al., 2023) test sets. The results are shown in Table 15 and Table 16. BioT5+ demonstrate superior performance in almost all metrics compared to baseline methods, which underscore BioT5+’s advanced capabilities in understanding complex molecular data.

heading: B Hyper-parameters
text: Pre-training. The pre-training process spans 300K steps and is executed on eight NVIDIA 80GB A100 GPUs with batch size 96 per GPU. To balance the data from different tasks during pre-training, we adopt a batch-level balancing strategy, where each batch evenly includes data from all eight different tasks, ensuring a more balanced and comprehensive pre-training process. For small datasets, such as molecule-text pairs and proteintext pairs, we employ a round-robin strategy to repeat their usage multiple times, compensating for their limited size. The dropout rate is maintained at 0.0 and the maximum input length during pretraining is set at 512. Optimization is performed using the AdamW (Loshchilov and Hutter, 2019) optimizer with Root Mean Square scaling. A cosine annealing learning rate scheduler is employed, with the base rate set at 1e− 2 and the minimum rate at 1e− 5.
Multi-task Fine-tuning. For multi-task finetuning, the dropout rate is searched in [0.0, 0.05, 0.1], and the learning rate is searched in [5e-5, 1e-4, 2e-4, 5e-4]. The total number of steps is 100K and warmup steps is 6% of total steps. The batch size is set to 768 for molecule-oriented tasks and 96 for protein-oriented tasks. The best hyper-parameters for molecule-oriented and protein-oriented task are shown in Table 12.

heading: C NER and Entity Linking
text: In general, our approach adheres to the same Named Entity Recognition (NER) and Entity Linking process as BioT5 (Pei et al., 2023) for bioentity name mentions in biological text using BERN2 (Sung et al., 2022). However, we have implemented some modifications to the process: (1) Upon analyzing the confidence scores of the identified bio-entities, we observed a long-tailed distribution, with the majority of bio-entity confidence scores exceeding 0.9. Based on this empirical finding, we set a threshold of 0.9, retaining only those NER results that surpass this score. (2) With the introduction of IUPAC names into our workflow, we now assess whether a recognized molecular name is an IUPAC name. If it is, we exclusively append the SELFIES representation; if not, we append both the IUPAC name and the SELFIES. This dual approach ensures a more comprehensive and accurate representation of molecular entities in our analysis.
D IUPAC Incorporation
In our downstream tasks of molecule property prediction and molecule description generation, it is effective to enrich molecules with their IUPAC name. One reason is that IUPAC names are more commonly found in bio-text. By explicitly incorporating IUPAC names in the molecular context during pre-training, the model more readily learns relevant molecular knowledge and establishes connections between the molecule and its contextual information. Additionally, IUPAC names inherently contain structural information about the molecule, such as functional groups and other structural components. This information allows the model to better understand the molecular structure, thus predicting molecule properties with higher accuracy and generating more accurate and detailed descriptions.
D.1 Mapping Process
Initially, we normalize the SMILES sequences provided in the dataset and map them to their respective PubChem (Kim et al., 2019) CIDs. Subsequently, these CIDs are used to retrieve the corresponding IUPAC names. However, for some molecules, their SMILES sequences do not correspond to a PubChem CID. In such cases, we employ STOUT (Rajan et al., 2021), a highly accurate SMILES to IUPAC name translator utilizing transformers, to convert these SMILES sequences
1229
into IUPAC names. This multi-step process ensures that each molecule is accurately equipped with its IUPAC name, facilitating more effective prediction and generation tasks.

heading: E Comparison with BioT5
text: Regarding the size of pre-training data and preprocessing techniques, BioT5+ introduces several enhancements over BioT5 (Pei et al., 2023):
• Addition of 28.8M PubChem molecular data, including molecule SELFIES along with their IUPAC names.
• Addition of 28.8M full articles from PubMed.
• Addition of 2.3M bioRxiv abstracts as wrapped biotext.
• In the molecule-text bidirectional translation task, IUPAC representations are included in the text description.
• For wrapped biotext pre-training, detected molecule names in BioT5 are directly replaced with corresponding SELFIES. In
BioT5+, we first determine whether a molecule name is an IUPAC name; if so, it is appended with the corresponding SELFIES. If not, both the IUPAC name and SELFIES are appended. Besides, we further ensure data quality by only processing detected bioentities with a confidence score (from BERN2 (Sung et al., 2022)) above 0.9.
• The numbers that appear in the pre-training corpus are tokenized character by character in BioT5+’s tokenizer.

heading: F Additional Ablation Study
text: To contrast single-task and multi-task tuning strategies, we further fine-tune BioT5+ with a single-task setting on three MoleculeNet (Wu et al., 2018) classification tasks including BACE, BBBP, Clintox, and three QM9 (Fang et al., 2023) regression tasks including HOMO, LUMO, and HOMO-LUMO gap. The consolidated results are shown in Table 14 (we also summarize the results in Table 7 and Table 13 for generation tasks on ChEBI-20 (Edwards et al., 2022) dataset here), covering 3 types of
1230
tasks: MoleculeNet for classification task, ChEBI20 for generation task, and QM9 for regression task. Our findings indicate that: (1) Single-task fine-tuning yields different performance in different tasks. For the tasks such as BACE, BBBP, Clintox, and ChEBI-20, single-task tuned BioT5+ performs closely or slightly better than the multitask-tuned version. (2) Multi-task fine-tuning can still be advantageous for tasks with inherent correlations, such as the prediction of HOMO, LUMO, and the HOMO-LUMO gap. This evidence points to the potential for cross-task generalization, even
if it is not uniformly applicable across all tasks.

heading: G Fine-tuning Details
text: We adopt multi-task instruction tuning on moleculeoriented tasks and protein-oriented tasks. To facilitate a fair comparison with earlier studies, given the wider variety of categories in our fine-tuning dataset compared to the Mol-Instructions (Fang et al., 2023) dataset, we perform multi-task instruction tuning for both molecule-oriented and proteinoriented tasks, using both the Mol-Instruction dataset and an alternative dataset excluding Mol-
1231
Instructions for each sub-domain. Taking into account the varying sizes of the datasets involved, we report the results across differing epoch spans to accommodate these discrepancies. All results are derived from 3 random runs.
G.1 Molecule-oriented Tasks
G.1.1 Molecule Property Prediction Classification We focus on the following four datasets with scaffold splits setting: (1) BACE dataset provides both qualitative binary labels and quantitative IC50 measurements for various inhibitors aimed at human beta-secretase 1 (BACE-1). (2) BBBP (Blood-Brain Barrier Penetration) dataset, designed to assist in predicting and modeling permeability of the blood-brain barrier, consists of compounds classified by binary labels that denote their ability to penetrate the barrier. (3) HIV dataset includes more than 40,000 compounds evaluated for their ability to inhibit HIV replication. They were initially categorized into Confirmed Inactive (CI), Confirmed Active (CA), and Confirmed Moderately Active (CM). Later, CA and CM categories were merged, simplifying the classification into a binary system of inactive (CI) versus active (CA and CM). (4) Clintox dataset differentiates between FDAapproved drugs and those that failed clinical trials due to toxicity. It features two distinct classification tasks with known chemical structures: (i) determining whether they exhibited toxicity in clinical trials, and (ii) assessing their FDA approval status.
We compare our BioT5+ with (1) singletask specialist models including GraphCL (You et al., 2020), GraphMVP-C (Liu et al., 2022), MGSSL (Zhang et al., 2021), MolCLR (Wang
et al., 2022), GEM (Fang et al., 2022), UniMol (Zhou et al., 2023), KV-PLM (Zeng et al., 2022), MoMu (Su et al., 2022), MolFM (Luo et al., 2023b), MolXPT (Liu et al., 2023c), and BioT5 (Pei et al., 2023); (2) LLM-based generalist models including Galactica (Taylor et al., 2022), Vicuna (Chiang et al., 2023), Llama2 (Touvron et al., 2023b), and InstructMol (Cao et al., 2023). The baseline results are mainly derived from original papers, MolFM (Luo et al., 2023b), InstructMol (Cao et al., 2023). The evaluation metric is AUROC, and we follow the same AUROC calculation method with MolXPT (Liu et al., 2023c) and BioT5 (Pei et al., 2023) based on the logits of “yes” and “no” predictions.
Regression For the regression task, we focus on three tasks from the QM9 dataset, which encompasses over 134,000 stable organic molecules with no more than nine heavy atoms, characterized by their geometric, energetic, electronic, and thermodynamic properties. Following MolInstructions (Fang et al., 2023), our focus is on three specific subtasks within QM9: (1) “HOMO” for the highest occupied molecular orbital energy. (2) “LUMO” for the lowest unoccupied molecular orbital energy. (3) “GAP” denoting the energy difference between HOMO and LUMO. All of them are measured in Hartree units. We use the processed QM9 dataset in instruction format and corresponding splits from Mol-Instruction (Fang et al., 2023).
We compare our BioT5+ with LLM-based generalist models, including Llama2 (Touvron et al., 2023b), Vicuna (Chiang et al., 2023), MolInstructions (Fang et al., 2023), and InstructMol (Cao et al., 2023). The baseline results come from InstructMol (Cao et al., 2023). The evaluation
1232
metric is MAE.
G.1.2 Chemical Reaction-related Tasks In computational chemistry, tasks centered around chemical reactions are crucial, as they significantly enhance research and development efficiency and support the advancement of eco-friendly chemistry methods. These tasks typically require understanding the interplay among reactants, reagents, and products, usually represented in the format of “reactant > reagent > product”. In line with Fang et al., 2023 and Cao et al., 2023, we concentrates on three specific tasks: (1) Reagent Prediction: This critical task entails the identification of key components such as catalysts, solvents, or auxiliary substances necessary for executing a chemical reaction. The input for this task comprises the reactants and the intended product, challenging the model to deduce the required reagents for the reaction process. (2) Forward Reaction Prediction: This task focuses on forecasting the potential outcomes of a chemical reaction. Provided with the reactants and reagents, the objective is to accurately predict the products that would result from the chemical reaction, thereby aiding in the planning and optimization of chemical synthesis. (3) Retrosynthesis: An essential task in synthetic chemistry, retrosynthesis
involves working backwards from a target product to identify plausible reactant combinations for its synthesis. The input is a specific product compound, and the challenge lies in determining the most efficient and feasible reactants needed to produce it. All of these chemical reaction-related data in instruction format and corresponding splits are sourced from Mol-Instruction (Fang et al., 2023) dataset.
We compare our BioT5+ with LLM-based generalist models, including Alpaca (Taori et al., 2023), Baize (Xu et al., 2023a), ChatGLM (et.al., 2023), Llama (Touvron et al., 2023a), Vicuna (Chiang et al., 2023), Galactica (Taylor et al., 2022), Text+Chem T5 (Christofidellis et al., 2023), MolInstructions (Fang et al., 2023), and InstructMol (Cao et al., 2023). The baseline results are sourced from InstructMol (Cao et al., 2023). The evaluation metrics are exact match, BLEU (Papineni et al., 2002), Levenshtein distance (Miller et al., 2009), three molecular fingerprints (FTS) similarity scores including MACCS (Durant et al., 2002), RDK (Schneider et al., 2015), and Morgan (Rogers and Hahn, 2010), and validity score (whether the SMILES can be successfully processed by RDKit (Landrum, 2021)).
1233
G.1.3 Molecule Description Generation
Molecule description generation task requires the model to generate a comprehensive description for a given molecule, encompassing its properties, functions, and potential applications. Unlike the straightforward prediction of molecular properties, this task poses a significantly greater challenge. It demands an in-depth and holistic understanding of the molecule from the model, necessitating not only the recognition of its structural and chemical characteristics but also an integration of this knowledge into a coherent, detailed narrative. There are two well-established benchmark datasets for this task: ChEBI-20 (Edwards et al., 2022) and MolInstructions (Fang et al., 2023) with recommended splits, and the corresponding results are presented in Table 4 and Table 15.
We compare our BioT5+ with (1) single-task specialist models, including Transformer (Vaswani et al., 2017), T5 (Raffel et al., 2020), MolT5 (Edwards et al., 2022), MoMu (Su et al., 2022), MolFM (Luo et al., 2023b), MolXPT (Liu et al., 2023c), GIT-Mol (Liu et al., 2023a), Text+Chem T5 (Christofidellis et al., 2023), BioT5 (Pei et al., 2023), and MolCA (Liu et al., 2023d); (2) retrievalbased LLM, including GPT-3.5-turbo (Li et al., 2024) and GPT-4 (OpenAI, 2023). (3) LLM-based generalist models, including GPT-3.5-turbo (Li et al., 2024), BioMedGPT (Luo et al., 2023c), Mol-Instructions (Fang et al., 2023), and InstructMol (Cao et al., 2023). The baseline results are mainly derived from MolXPT (Liu et al., 2023c), BioT5 (Pei et al., 2023), MolCA (Liu et al., 2023d), and InstructMol (Cao et al., 2023). The evaluation metrics are common NLP metrics, including BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and METEOR (Banerjee and Lavie, 2005).
G.1.4 Description-guided Molecule Design
Description-guided molecule design is the inverse task of molecule description generation. This task centers around designing molecules based on detailed textual descriptions. Here, the model is presented with a comprehensive description encompassing various aspects of a desired molecule, such as its intended functions, properties, and potential applications. The challenge lies in accurately interpreting this textual information and translating it into a specific molecular structure. This task is considerably complex as it requires the model to have a deep understanding of the relationship between molecular characteristics and their corresponding textual descriptors. This task is crucial for advancing drug discovery and material design, where precise molecular configurations are often derived from elaborate functional requirements. We use the same two benchmark datasets: ChEBI-20 (Edwards et al., 2022) and Mol-Instructions (Fang et al., 2023) as the molecule description generation task, and the corresponding results are shown in Table 5 and Table 16.
The compared baselines are a subset of that listed in Section G.1.3 except for Llama2 (Touvron et al., 2023b), with results mainly sourced from BioT5 (Pei et al., 2023) and MolReGPT (Li et al., 2024). The evaluation metrics are BLEU (Papineni et al., 2002), exact match, Levenshtein distance (Miller et al., 2009), three molecular fingerprints (FTS) similarity scores including MACCS (Durant et al., 2002), RDK (Schneider et al., 2015), and Morgan (Rogers and Hahn, 2010), FCD score (Preuer et al., 2018), Text2Mol score (Edwards et al., 2021), and validity score.
1234
G.2 Protein-oriented Tasks
G.2.1 Protein Description Generation
In computational biology, the task of protein description generation is of paramount importance, as it entails extracting insightful textual information from protein sequences. Aligning with Fang et al., 2023, our focus is on four intricate generation tasks, each taking a protein sequence as its input: (1) Protein Function Generation: This task aims to produce outputs that consist of Gene Ontology (GO) terms, providing a multifaceted description of the protein’s functions. These GO terms cover three key domains: cellular component, biological process, and molecular function, offering a holistic view of the protein’s role and interactions within a cellular context. (2) Catalytic Activity Generation: Here, the focus is on delineating the specific catalytic activities of the protein, moving beyond merely identifying its Enzyme Commission (EC) number. The output targets a detailed characterization of the chemical reactions facilitated by the protein, capturing its dynamic role in metabolic and biochemical pathways. (3) Domain/Motif Generation: This task involves pinpointing and describing domains or motifs within the protein sequence. Domains and motifs are essential elements, recognized as compact, folded three-dimensional structures that play pivotal roles in the protein’s function and stability. The identification of these features is crucial for understanding protein folding, function, and interactions. (4) Functional Description Generation: The goal here is to generate a comprehensive and detailed textual description that encapsulates a protein’s function, its subcellular localization, and its involvement in various biological processes. This output seeks to provide an extensive narrative, encompassing the diverse functionalities, roles, and significance of the protein within a biological system. These tasks collectively aim to deepen our understanding of proteins, facilitating advancements in fields such as drug discovery, molecular biology, and bioinformatics. All of these data in instruction format and corresponding splits are sourced from Mol-Instruction (Fang et al., 2023) dataset.
We compare our BioT5+ with LLM-based generalist models, including Alpaca (Taori et al., 2023), Baize (Xu et al., 2023a), ChatGLM (et.al., 2023), Galactica (Taylor et al., 2022), Llama (Touvron et al., 2023a), Vicuna (Chiang et al., 2023), and Mol-Instructions (Fang et al., 2023). The baseline results come from Mol-Instructions (Fang et al.,
2023). The evaluation metric is ROUGE-L (Lin, 2004).
G.3 Description-guided Protein Design Similar to description-guided molecule design, in the description-guided protein design task, the primary objective is to generate amino acid sequences of proteins that meet specific user-defined design requirements. This task necessitates a profound and comprehensive understanding of protein structures and functions from the model. It involves the intricate process of translating complex textual descriptions, which may include functional targets, structural characteristics, and desired biological activities, into precise amino acid sequences. As there is no well-established benchmark for this task, we show some cases for this task in Section 20.
G.4 Protein Property Prediction The protein property prediction task plays a pivotal role in computational biology, focusing on the prediction of specific protein attributes such as solubility, structural characteristics, or functional roles, based on their amino acid sequences or structural information. Following Pei et al., 2023, we centered on two key protein property prediction tasks within the PEER (Xu et al., 2022) benchmark. (1) Solubility Prediction: This task involves determining the solubility status of a given protein. It seeks to predict if a protein, when introduced into a solvent, will dissolve or remain insoluble. This property is crucial as it influences the protein’s functionality and its interaction with other biomolecules. (2) Localization Prediction: The second task focuses on identifying the cellular localization of proteins, distinguishing whether a given protein is “membrane-bound” or “soluble”. Membrane-bound proteins are those that are associated with or integrated into the cell membrane, playing key roles in various cellular processes such as signal transduction and transport. In contrast, soluble proteins are those that are not associated with the membrane and are typically involved in various intracellular activities. We use the same data and splitting methods as BioT5 (Pei et al., 2023).
We compare our BioT5+ with (1) singletask specialist models, including DDE (Dipeptide Deviation from Expected Mean) (Saravanan and Gautham, 2015), Moran feature descriptor (Moran correlation) (Feng and Zhang, 2000), LSTM (Hochreiter and Schmidhuber, 1997), Trans-
1235
formers (Vaswani et al., 2017), CNN (O’Shea and Nash, 2015), ResNet (He et al., 2016), ProtBert (Elnaggar et al., 2021), ESM-1b (Rives et al., 2021), and BioT5 (Pei et al., 2023); (2) multi-task generalist models, including CNN (O’Shea and Nash, 2015), Transformers (Vaswani et al., 2017), and ESM-1b (Rives et al., 2021). Note that here the multi-task generalist results are derived from PEER (Xu et al., 2022), where contact prediction, fold classification, and secondary structure prediction are combined with the original task. We report the best results obtained from training each of these tasks in conjunction with the primary task. The baseline results are derived from PEER (Xu et al., 2022). The evaluation metric is Accuracy.
G.5 Protein-Protein Interaction The Protein-Protein Interaction (PPI) task is an essential component in the field of computational biology, focusing on the prediction of interactions between two proteins based on their amino acid sequences. In this task, the input consists of the amino acid sequences of two distinct proteins, and the output is a binary classification: “yes” if the proteins are predicted to interact, and “no” if they are not. This task holds substantial biological significance as protein-protein interactions are fundamental to most biological processes, including signal transduction, cellular metabolism, and immune responses. Following Pei et al., 2023, we use Yeast and Human PPI datasets with corresponding splits from the PEER (Xu et al., 2022) benchmark, which include proteins related to yeast and humans respectively.
The compared baselines are the same with that in Section G.4, with results derived from PEER (Xu et al., 2022). The evaluation metric is Accuracy.
G.6 Drug-Target Interaction The Drug-Target Interaction (DTI) task focuses on the prediction of interactions between a drug molecule and a protein target. This task involves inputting the molecular structure of a drug, encoded as a SELFIES representation, alongside the amino acid sequence of a target protein. The output is a binary decision: “yes” indicates a predicted interaction between the drug and the protein, and “no” suggests no interaction. Understanding and predicting these interactions is crucial for drug discovery and development, offering insights into the mechanism of action of drugs, identifying potential off-target effects, and aiding in the design of novel
therapeutics with improved efficacy and reduced side effects. We use the same data and corresponding splits with BioT5 (Pei et al., 2023).
We compare our BioT5+ with single-task specialist models, including Support Vector Machine (Cortes and Vapnik, 1995) (SVM), Random Forest (Ho, 1995) (RF), DeepConv-DTI (Lee et al., 2019), GraphDTA (Nguyen et al., 2021), MolTrans (Huang et al., 2021), DrugBAN (Bai et al., 2023), and BioT5 (Pei et al., 2023). The baseline results are sourced from DrugBAN (Bai et al., 2023) and BioT5 (Pei et al., 2023). The evaluation metric is AUROC, AUPRC, and Accuracy.

heading: H Case Studies
text: In this section we show some cases for our finetuned BioT5+ model. We only showcase key parts of inputs, intentionally omitting the instruction context due to the varying requirements for instruction context across different baseline methods.
H.1 Molecule-oriented Cases We show some selected cases for molecule-oriented tasks, including molecule description generation in Table 17, description guided molecule design in Table 18, and chemical reaction-related tasks in Table 19.
H.2 Description-guided Protein Design We also show some cases of description-guided protein design task in Table 20. We compute the normalized Smith-Waterman (SW) alignment score (Yamanishi et al., 2008) between the output and ground truth protein FASTA sequence. This method involves comparing segments of variable lengths within the protein sequences and is optimized to maximize the similarity metric, effectively evaluating the correspondence between the model output and the ground truth. In the model output of amino acid sequences, occurrences of more than two consecutive amino acids at the terminus were manually truncated.
1236

heading: Future Work
text: Despite the significant advancements achieved by BioT5+, there remain certain limitations that need
to be addressed in future work. Firstly, the model faces challenges in generalizing across various biological tasks, a problem that is distinct from common NLP settings. The intricate and unique nature of each biological task makes it difficult to develop a one-size-fits-all solution, highlighting the need for more specialized approaches within this domain. Secondly, the current scale of BioT5+ is somewhat limited and cannot comprehend and integrate information from other modalities, such as images, restricting its applicability in multi-modal biological data analysis. BioT5+ is not equipped to function as a universal chatbot or to answer queries spanning general domain questions outside the specific scope of biology. This constraint highlights the need for developing larger, more versatile models capable of handling a wider range of data types and answering a broader array of questions in both biological and general domains