BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations
Authors: Qizhi Pei
Wei Zhang
Jinhua Zhu
Kehan Wu
Kaiyuan Gao
Lijun Wu
Yingce Xia
Rui Yan
Venue: EMNLP 2023 Main (2023)
Keywords: cross-modal
biology
text
molecule
protein

Abstract: Recent advancements in biological research leverage the integration of molecules, proteins, and natural language to enhance drug discovery. However, current models exhibit several limitations, such as the generation of invalid molecular SMILES, underutilization of contextual information, and equal treatment of structured and unstructured knowledge. To address these issues, we propose BioT5, a comprehensive pre-training framework that enriches cross-modal integration in biology with chemical knowledge and natural language associations. BioT5 utilizes SELFIES for 100\% robust molecular representations and extracts knowledge from the surrounding context of bio-entities in unstructured biological literature. Furthermore, BioT5 distinguishes between structured and unstructured knowledge, leading to more effective utilization of information. After fine-tuning, BioT5 shows superior performance across a wide range of tasks, demonstrating its strong capability of capturing underlying relations and properties of bio-entities. Our code is available at https://github.com/QizhiPei/BioT5.

BioT5: Enriching Cross-modal Integration in Biology with Chemical
Knowledge and Natural Language Associations
Qizhi Peils, Wei Zhang?, Jinhua Zhu?, Kehan Wu?, Kaiyuan Gaoe, Lijun Wu**, Yingee Xia**, Rui Yan
'Gaoling School of Artificial Intelligence, Renmin University of China University of Science and Technology of China
Huazhong University of Science and Technology *Microsoft Research Engineering Research Center of Next- Next-Generation Intelligent Search and Recommendation, Ministry of Education
Beijing Key Laboratory of Big Data Management and Analysis Methods {qizhipei, (uiiyanerrcc, edu. cn
(weizhang_cs, teslazhu, wu_2018 (2001)mmmmml........ www. edu. cn im_ mai_kai@hhst.cc edu. cn (lijuw,, www.:yyiiiiiiiiiiioooooo
com
Abstract
Name: Hemoglobin suburit
SMILES: CC CCC=CCCC1-00000
Recent advancements in biological research leverage the integration of molecules, proteins, and natural language to enhance drug discovery. However, current models exhibit several limita- tions, such the generation of invalid molec- ular SMILES, underutilization of contextual information, and equal treatment of structured and unstructured knowledge. To address these issues, we propose Bio'T',, a comprehensive pre- training framework that enriches cross-modal integration in biology with chemical knowl- edge and natural language associations. BioT5 utilizes SELFIES for 100% robust molecular representations and extracts knowledge from the surrounding context of bio-entities un- structured biological literature. Furthermore, BioT5 distinguishes between structured and un- structured knowledge, leading more effective utilization of information. After fine-tuning, BioT5 shows superior performance across wide range of tasks, demonstrating its strong capability of capturing underlying relations and properties of bio-entities. Our code is available https:: / thub. hom//iizhaPii/BiTTS
SELFIE: / [[C]] (eCneenhhCC[[opppeeeee [C]oooooo---0000C0 [-C|A-CC|iing 1 -ojo
FASTA: WWH.TPESSAVTAANGGE
Structure
Structure:
OH
Representations of molecule and protein. Molecule can be represented by its name, bio-sequence (SMILES and SELFIES), and 2D graph structure. Pro- tein can be represented by its name, corresponding gene name, bio-sequence (FASTA), and 3D structure.
Figure 1:
molecules and proteins can be represented using quences. A molecule can be depicted by a SMILES sequence (Weininger, 1988; Weininger et al., 1989), which derived by traversing the molecular graph through depth-first search and applying specific branching rules. A protein can be represented by FASTA sequence (Lipman and Pearson, 1985; Pearson and Lipman, 1988), which outlines the amino acids in a protein. The sequential formats of molecules and proteins facilitate the application of Transformer models (Vaswani al., 2017) and pre- training techniques (Liu et al., 2019; Radford al., 2019) from natural language processing (NLP) to the biomedical field. Chemberta (Chithrananda et al., 2020) and ESM (Rives et al., 2021; Lin et al.. 2022) apply masked language modeling to molecu- lar SMILES and protein FASTA respectively, while MolGPT (Bagal et al., 2022) and ProtGPT2 (Ferruz et al., 2022) leverage GPT-style models for molec- ular and protein generation.
1 Introduction
Molecules and proteins are two essential bio- entities in drug discovery (Dara et al., 2022). Small molecule drugs have been the cornerstone of the pharmaceutical industry for nearly a century, owing to their unique advantages such as oral availability, diverse modes of action, etc (AstraZeneca, 2023). Proteins serve as the foundation of life science, functioning as drug targets or crucial elements in disease pathways. As illustrated in Figure 1, both
*Corresponding authors: Lijun Wu (1i (Lijuwu@microsoft com), Yingee Xia (yinxia@mic crosoft, com), and Rui Yan (ruiyan@ruc. cn)
Scientific literature (Beltagy et al., 2019; Canese and Weis, 2013) and biological databases (Kim

---

al., 2023; Boutet al., 2007) serve as knowledge repositories of molecules and proteins. These re- sources detail properties, experimental results, and interactions between various bio-entities, which cannot be explicitly inferred from molecular or protein sequences alone. Consequently, recent trend involves jointly modeling text along with molecules and proteins, allowing the textual de- scriptions to enhance molecular and protein repre- sentations. MolT5 (Edwards al., 2022) adopts the T5 (Raffel et al., 2020) framework molecular SMILES and biomedical literature. MolXPT (Liu et al., 2023b) and Galactica (Taylor et al., 2022) are GPT models trained on text and bio-entities, such as SMILES and FASTA sequences. Deep- EIK (Luo et al., 2023) fuses the encoded features from multi-modal inputs using attention (Vaswani al., 2017) mechanism. Despite their success, there is still significant room for improvement: (i) Prior work often relies on SMILES to represent molecules. However, addressing the issue of gen- erating invalid SMILES remains a challenge to overcome (Edwards et al., 2022; Li et al., 2023). (ii) The contextual information surrounding molec- ular protein names could offer valuable insights for understanding the interactions and properties of bio-entities. Developing effective methods lever- age this information merits further attention. (iii) Existing research tends to treat structured data (c.g.. molecule-text pairs from databases) and unstruc- tured data (e.g., text sequences in literature) equally. However, structured data could be utilized more effectively to further enhance overall performance.
sequences. Following Liu et al. (2023b), we fer to such data "wrapped" text. Text tokens, FASTA sequences, and SELFIES are tokenized independently (see Section 3.2 for more details). (2) Model training: BioT5 utilizes a shared en- coder and shared decoder to process various modalities. The standard T5 employs the "recover masked spans" objective, wherein each masked span and its corresponding part share the same sen- tinel token. We refer to the aforementioned train- ing objective function as the "T5 objective" for simplicity. There are three types of pre-training tasks: (i) Applying the standard T5 objective to molecule SELFIES, protein FASTA, and general text independently, ensuring that the model pos- sesses capabilities in each modality. (ii) Applying the T5 objective to wrapped text from the biologi- cal domain, where all text, FASTA, and SELFIES tokens can be masked and recovered. (iii) For the structured molecule-text data, we introduce a trans- lation objective. Specifically, BioT5 is trained to translate molecule SELFIES to the corresponding description and vice versa. Likewise, the transla- tion objective is applied protein-text data.
After pre-training, we fine-tune the obtained BioT5 on 15 tasks covering molecule and pro- tein property prediction, drug-target interaction prediction, protein-protein interaction prediction, molecule captioning, and text-based molecule gen- eration. BioT5 achieves state-of-the-ar perfor- mances 10 tasks and exhibits results comparable to domain-specifie large models 5 tasks, demon- strating the superior ability of our proposed method. BioT5 model establishes a promising avenue for the integration of chemical knowledge and natu- ral language associations to augment the current understanding of biological systems.
To address the above challenges, in this paper, we introduce BioTS, comprehensive pre-training framework encompassing text, molecules, and pro- teins. BioT5 leverages SELFIES (Krenn et al., 2020) to represent small molecules since its advan- tage over SMILES is that SELFIES offers a more robust and error-tolerant molecular representation, eliminating issues of illegitimate structures often encountered with SMILES. There are mainly two steps for BioT5 pre-training:
Related Work
In this section, we briefly review related work about cross-modal models in biology and representations of molecule and protein.
2.1 Cross-modal Models in Biology
(1) Data collection & processing: We gather text, molecule, and protein data, as well as existing databases containing molecule-text parallel data and protein-text parallel data. For the text data (PubMed) from the biological domain, we employ named entity recognition and entity linking ex- tract molecular and protein mentions, replacing them with the corresponding SELFIES or FASTA
Language models in the biology field have gained considerable attention. Among these, BioBERT (Lee et al., 2020) and BioGPT (Luo et al., 2022), which are pre-trained on scientific corpora, have been particularly successful in effec- tively understanding scientific texts. More recently, cross-modal models focusing on jointly modeling

---

text with bio-sequences have emerged. They can be categorized into the following three groups. Cross Text-molecule Modalities MolT5 (Ed- wards et al., 2022) is T5 (Raffel et al., 2020)- based model, which is jointly trained on molecule SMILES and general text corpus. MoSu (Su al.. 2022) is trained on molecular graphs and related textual data using contrastive learning. MolXPT (Liu et al., 2023b) is a GPT (Radford et al., 201 2018)-based model pre-trained on molecule SMILES, biomedical text, and wrapped text. Dif- ferent from BioT5, these models all use SMILES to represent molecules, which leads to validity issues when generating molecules.
Weininger et al., 1989), InChl (Heller et al., 2013), DeepSMILES (O Boyle and Dalke, 2018), SELF- IES (Krenn et al., 2020), 2D molecular graph, etc. SMILES (Simplified Molecular-Input Line- Entry System), a compact and textual representa- tion of the molecular structure, is the most common method. It employs a sequence of characters to en- code atoms, bonds, and other molecular features. However, SMILES has several drawbacks (Krenn et al., 2022), such as the lack of syntactic and se- mantic robustness, which significantly affects the validity of molecules generated by deep learning models (Edwards et al., 2022). To address this is- sue, SELFIES (Self-referencing Limbedded Strings) is introduced as a 100% robust molecular string rep- resentation (Krenn et al., 2020). Every permutation of symbols within the SELFIES alphabet invariably generates a chemically valid molecular structure, ensuring that each SELFIES corresponds to a valid molecule. Unlike existing works introduced Sec- tion 2.1 that use SMILES for molecule representa- tion, we employ SELFIES with separate encoding in BioT5 to achieve 100% validity in downstream molecule generation tasks.
Cross Text-protein Modalities ProteinDT (Liu et al., 2023a) is a multi-modal framework that text for protein design.
uses semantically Bronsaanrttorr (Xu et al., 2023a) is cross-modal translation system specifically designed for anno- tating biological instances, such gene expression vectors, protein networks, and protein sequences, based on writtwritten text.
Cross Three or More Biology Modalities Galac- tica (Taylor et al., 2022) is general GPT-based large language model trained on various scien- tific domains, including scientific paper corpus, knowledge bases (e.g., PubChem (Kim al., 2023) molecules, UniProt (uni, 2023) protein), codes, and other sources. DeepElK (Luo et al., 2023) fuses the feature from multi-modal imputs (drugs, proteins, and text). Then attention (Vaswani et al., 2017) mechanism is adopted to do textual information denoising and heterogeneous features integration.
Protein Representation Protein can also be repre- sented in various ways, such as by its name, cor- responding gene name, FASTA format, or 3D ge- ometric structure. The FASTA format a com- mon choice for encoding protein sequences, which uses single-lctter codes to represent the 20 different amino acids. In BioTS, we also employ FASTA format for protein representation.
Unlike Edwards et al. (2022) and Taylor et al. (2022) that share the dictionary between bio- sequence tokens and nature language tokens, BioT5 uses a separate dictionary and biology-specific tok- enization to explicitly distinguish biological modal- ities. We give further analysis of this in Section 3.2.
Our work differs from previous studies in sev- eral ways: (1) we primarily focus on two biological modalities-moleculee protein-with text serving knowledge base and bridge to enrich the un- derlying relations and properties in the molecule and protein domains; (2) we use multi-task pre- training to model the connections between these three modalities a more comprehensive manner. (3) we use SELFIES instead of SMILES to repre- sent molecules, which is more robust and resolves the validity issue in molecule generation tasks.
3 BioT5
The overview of the BioT5 pre-training is illus- trated in Figure 2. We combine data from different modalities to perform multi-task pre-training.
3.1 Pre-training Corpus
2.2 Representations of Molecule and Protein
As shown in Figure 2, the pre-training corpus of BioT5 is categorized into three classes: (1) Single- modal data, including molecule SELFIES, protein FASTA, and general text. For small molecules, we use the ZINC20 (Irwin et al., 2020) dataset and convert SMILES to SELFIES. For protein
Molecule Representation The representation and modeling of molecules have long been challenge in bioinformaties. There are many methods to represent a molecule: name, fingerprint (Rogers and Hahn, 2010a), SMILES (Weininger, 1988;

---

Task Baom:CUliiiIOOMMM-ICC((iiieeeeee
Modality
Molecule
ancht]j[]]eeomo
#2
Frotein FASTA
addition, of <M1> involved <M2> folding pethway.
chaperones, immunaphilnss and other proteins eMiz+Hip00- mediated protein
General Text
Effect -oo--boo-CCi [[][[])M-MMM-
<eoms on cultured fibroblasts: <M2> related
<MI>[N] (MMa-Nh/BBanchhl <M2 release of lysosomal hydrolases MaB.--p--pp---paa
sequence
BioT5
Wrapped Sentences
Is <bopocc-MM Masps-pp-ssp-ppppppppppp spplsp.......oo. <cop>
and inhibition of their uptake.
Chloroform.
DESCRIPTION: Chloraform is colorless pleasant, nonimitating and a slightly sweat tasto..
Molecule
Pair
FUNCTION: Acts nuclear transcription factor... Binds heat
#6 shock promoter elements (HSE). SUBCELLULAR LOCATION: Nucieus, PROTEIN FAMILIES: FAM170 family
Protein-text
Pair
Figure 2: Overview of Bio'T5 pre- pre-training. The solid line refers to the "T5 objective", which aims to reconstruct the original unmasked input. Each consecutive span of masked tokens is replaced with a sentinel token, depicted as <M1>, <M2>, and <M3>, We apply this objective molecule SELFIES (task #1), protein FASTA (task #2), general text (task #3), and wrapped text (task f44). The dashed line represents the bidirectional translation between bio-sequences and structured text description (task #5 and #6).
FASTA, we randomly sample proteins from the Uniref50 (Suzek et al., 2007) dataset, filtering out proteins exceeding a specified length, result- ing in collection of 27M proteins For general text, the "Colossal Clean Crawled Corpus" (C4) dataset (Raffel et al., 2020). (2) Wrapped text, where molecule names are replaced with their corresponding SELFIES and gene names are ap- pended with related protein FASTA. We use 33M PubMed articles (Canese and Weis, 2013) and ap- ply BERN2 (Sung et al., 2022) for named entity recognition. The scientific sentences which are not replaced or appended by bio-sequences remained supplement to general text. The detailed process depicted in Figure 4 and dis- cussed Appendix B. (3) Molecule-descriptinn pairs and
Input
MofTS
Ours
Ground Truth
SMILES: [B-]][Lis]
moleculs a metal tetraborate,
molecule
molacule Ithlum sat which the counteeion
metal bromide salt with a UC-) counterion, non. has to rereitzor.
SELFIES:
metal ion and a ino onion. is conjugate scid of ditr onooiiiiiuu.
The unbydraux torms cublic crystals simllar common salt. it and
Structure:
inorgonic bromade salt and a Iitiium
Figure 3: Case for tokenization. MolT5 processes "Br"(bromine atom) as "B" (boron atom) and "r", re- sulting in incorrect descriptions including tetraborate (related "B"). BioT5 retains the chemically meaning- ful group "[Br-1]" as a complete token, thereby produc- ing the correct output.
sufficient attention to detail. MolT5 (Edwards et al., 2022) employs the same dictionary the original T5, it starts pre-training from the original T5 checkpoint. The original T5 dictionary is derived from nature language using SentencePiece (Kudo and Richardson, 2018). However, directly utilizing this dictionary for molecule SMILES is subopti- mal, as some chemically meaningful tokens, such as functional groups complete atoms, will be to- kenized inaccurately. For example, in the molecule depicted in Figure 3, the bromine atom, symbolized as "Br" in SMILES, is tokenized as "B" (a boron atom) and "r" by MolT5. Consequently. MolT5 incorrectly characterizes this molecule as both di- bromolit (related to "Br") and tetraborate (related to "B"). The character-based tokenization of Galac-
pairs. For molecule- text data, we collect 339K molecule SELFIES along with their corresponding names and descrip- tions from PubChem (Kim et al., 2019), excluding all molecules present in the downstream ChEBI- 20 dataset (Edwards et al., 2022) avoid poten- tial data leakage. For protein-text data, we obtain 569K protein FASTA-description pairs from Swiss- Prot (Boutet et al., 2007), which contains high- quality annotations of various protein properties. Details are left in Appendix E.I.
3.2 Separate Tokenization and Embedding
most previous works, the representation of molecules and proteins has not been modeled with

---

tica (Taylor al., 2022) suffers the same issue. In addition to the tokenization method, shar- ing token embeddings for different modalities (Fd- wards et al., 2022; Taylor et al., 2022) is also ques- tionable. In multilingual tasks, shared embeddings allow models to accurately represent the mean- ings of borrowed words and cognates, which retain their original meanings across languages. How- ever, molecules, proteins, and text represent en- tirely distinct languages. The same token within these three different modalities carries different semantic meanings. For example, the token "C" signifies character C in nature language, the car- bon atom in molecules, and cysteine (one of the 20 amino acids) in proteins. Studies by Beltagy et al. (2019) and Gu al. (2021) further emphasize the significance of domain- -sprciffe vocabulary.
Applying T5 objective wrapped text from scien- tific corpus (task #4). (3) Bidirectional translation for the molecule SELFIES-text pairs (task #5) and protein FASTA -tet pairs (task #6). By effectively learning the underlying connections and properties of bio-entities from textual information through these pre-training tasks, BioT5 allows for a holistic understanding of the biological domain, thereby fa- cilitating enhanced prediction and generation abili- ties in various biological tasks.
Fine-tuning BioT5 can be fine-tuned on various downstream tasks involving molecules, proteins, and text. To unify different downstream tasks and reduce the gap between pre-training and fine- tuning (Brown et al., 2020) stage, we adopt the prompt-based fine-tuning (Gao et al., 2021) ap- proach, which facilitates various task formats into sequence generation format.
To address the issues mentioned above, we employ separate vocabularies for molecule, pro- tein, and text. In BioT5, molecule is repre- sented by SELFIES string, where each chemi- cal meaningful atom group is enclosed within brackets and tokenized SELFIES token. For example, []][=]][Br-xCCC]CCe
4 Experiments and Results
We evaluate BioT5 on 15 well-established down- stream tasks, which can be categorized into three types: single-instance prediction, multi-instance prediction, and cross-modal generation. We in- clude details regarding fine-tuning datasets, base- lines, and prompts in Appendix F.
[Br]. For protein, to differentiate amino acids with capi- tal letters in text, we introduce a special pre- fix <p> for each amino acid. For example, <p>M<p><<<>R ><p>M <p>K, <p>R. For text, we use the same dictionary as the original T5, Through this, we explicitly distinguish the semantic space of different modalities, which maintains the inherent integrity of each unique modality and prevents the model from conflating meanings across modalities.
For the downstream binary classification tasks presented in Section 4.1 and 4.2, the calculation of evaluation metrics such as AUROC and AUPRC ne- cessitates the soft probability of the predicted label. As we use the prompt-based fine-tuning method, the output is either Yes for the positive label or No for the negative label. To obtain appropriate label distribution, following Liu et al. (2023b), we first extract the probabilities of Yes and No tokens (denoted as Ppos and Pneg respectively) and nor- malize them. The resulting probability for positive label is Pans and negative label is Pring
3.3 Model and Training
Model architecture BioT5 employs the same ar- chitecture T5 models (Raffel et al., 2020). We follow the configuration used in -5-ve.l-base! The vocabulary size of BioT5 35, 073, differing from the default configuration as we incorporate separate vocabulary for molecule SELFIES and protein amino acids. In total, the BioT5 model comprises 252M parameters.
Ppon PPang
4.1 Single-instance Prediction
4.1.1 Molecule Property Prediction
Molecule property prediction aims to determine whether a given molecule exhibits specific prop- erties. MoleculeNet (Wu et al., 2018) is a widely used benchmark for molecule property prediction, encompassing diverse datascts that cover numer- ous molecular aspects, such quantum mechanies, physical chemistry, biophysics, etc. In line with Liu et al. (2023b), we conduct experiments on six bi- nary classification tasks, including BBBP, Tox21, Toi, HIV, BACE, and SIDER. Following (Fang
Pre-training During the pre-training phase, the model is trained in multi-task way on six tasks that can be classified into three types: (1) Apply- ing T5 objective to each single modality including molecule SELFIES (task #1), protein FASTA (task #2), and general text (task #3) independently. (2)
'https: //agng/////ggigaaaee ace. wtt.c///oo///toa/oorrrooo model_doc/t5v1._

---

Datasct
BBBP
Tox21
Cli
HIV
BACE
SIDER
Avg
#Molecules
2039
7831
1478
41127
1513
1427
#Tasks
G-Contextual
1.6
7..2:1..3
$9.918.2
75.9109
79.2.10.3
58.4:00.6
69.8
G-Motif
66.4+13.4
73.2400.8
77.8+22.0
73.8+11.4
73.4+:4.0
60.6-t1111
these results, we can see that BioT5 surpasses base- lines on most downstream tasks in MoleculeNet. BioT5 exhibits superior performance compared GROVER MaoVERbase base
70.0=10.1
74.3+0.11
81.2++300
62.5-0.9
82.61007
64.810066
72.6
GROVER jarge
69.5++0.1
73.5+00.1
76.2+
68.2
81.0+114
65.4+0.11
72.3
GraphMVP
72,4+1.6
75.9+00.5
79.1+-2.8
77.0=1122
812++0,9
63.9+1122
74.9
MGSSL
70.5
76.5:10.3
80.712.11
79.
79.7
61881088
74.8
MolCLR
72.242.11
75.01:0.2
91.2.13.55
7..1
82.410.9
58.91144
76.3
GEM
72.4+0.4
78.1+0011
90.1+1133
80.6 +
85.6+1.1
67.24-0.4
79.0
KV-PLM
74,6+00.9
72.7+-0.6
74,0++1.2
61.5+155
Galactica
66.1
68.9
82.6
74.5
61.7
63.2
69.5
MoMu
70.5::2.0
75,6-1033
79.914.11
76.2:1.99
17.1
60.51:00
73.3
MolXPT
80.0
77.1+0.2
0.2
78.1
88.4 +
71.7 + 0.2
81.9
BioT5
77.7++0.6
77.9+-022
95.4+-0.5
81.0=0.1
89,4+0.33
73.2++0.2
82.4
Table I: Performance comparison on MoleculeNet (Best, Second Best). The evaluation metric is AUROC. The baseline results are mainly soureed from MolXPT (Liu al., 2023b).
Model
#Params.
Solubility
Localization
DDE
205.3K
59.77 +: 1.21
77.43 0.42 55.63
Moran
123.4K
57.73
1.33
LSTM
26.7M
70.18 + 0.63
88.11
0.14
to GNN basclines that are pre-trained on 2D/3D molecular data, underscoring the effectiveness of knowledge in text. Furthermore, BioT5 outper- forms other language model baselines, which may be attributed to the presence molecule property descriptions in scientific contextual text or existing biological database entries.
Transformer
21.3M
70.12 0.31
75.74 1:744:00 0.74
CNN
5.4M
64.43 + 0.25
82.67
+ 0.32
ResNet
11.0M
67.33 + 1.46
78.99 + 4.41
ProtBert
419.9M
68.15 0.92
91.32 0.89
ProtBert*
419.9M
59.17 + 0.21
81.54 0.09
ESM-1b
652.4M
70.23 + 0.75
92.40 0.35
LSM-1b*
652.4M
67.02 1 0.40
91.61 0.10
BioT5
252.1M
74.65 + 0.49
91.69 + 0.05
4.1.2 Protein Property Prediction
Table 2: Performance comparison of different methods on solubility and localization prediction tasks (Best, Second Best). The evaluation metric is accuracy. * represents only tuning the prediction head. The baseline results are sourced from PEER (Xu et al., 2022).
Protein property prediction is crucial as it provides critical insights into the behavior and functions of proteins. We concentrate two protein property prediction tasks on PEER benchmark (Xu et al., 2022): protein solubility prediction, which aims predict whether the given protein is soluble, and protein localization prediction, which is to classify proteins as either "membrane-bound" or "soluble". Baselines We compare BioT5 with three types of baselines provided in PEER benchmark: (1) feature engineers, including two protein sequence feature descriptors: Dipeptide Deviation from Expected Mean (DDE) (Saravanan and Gautham, 2015) and Moran correlation (Moran) (Feng and Zhang, 2000); (2) protein sequence encoders, including LSTM (Hochreiter and Schmidhuber, 1997), Trans- formers (Vaswani et al., 2017), CNN (O Shea and Nash, 2015) and ResNet (He al., 2016); (3) pre-trained protein language models, which are pre-trained using extensive collections of protein FASTA sequences, including ProtBert (Elnaggar et al., 2021) and ESM-Ib (Rives et al,, 2021). Both
et al., 2022), we adopt the scaffold splitting, which is more challenging compared to random splitting. Baselines We compare BioT5 with two types of baselines: (1) pre-trained Graph Neural Network (GNN) using molecular graph as input, which are G-Contextual (Rong et al., 2020), G-Motif (Rong et al., 2020), GROVER (Rong et al., 2020), GROVERiage (Rong et al., 2020), GraphMVP (Liu et al., 2022), MGSSL (Zhang et al., 2021) Mol- CLR (Wang et al., 2022) and GEM (Fang et al., 2022); (2) pre-trained language model basclines, which are KV-PLM (Zeng al., 2022), Galac- tica (Taylor et al., 2022), MoMu (Su al., 2022) and MolXPT (Liu et al., 2023b).
Results The results are presented in Table with all statistics derived from three random runs. From

---

BioSNAP
Human
BindingDB
Method
AUROC
AUPRC
Accuracy
AUROC
AUPRC
AUROC
AUPRC
Accuracy
SVM
0.862:0.007
0.864=0.004
0.777=0.011
0,940:10.006
0.920=0.009
0,939=0.001
0.928::0002
0.825=000004
RF
0,860:10.005
0.886=0.005
0.804:00005
0,952::0.011
0,953=0.010
0,942=00011
0.921:00016
0.880:0.012
DeepConv-DII
0,886=20.006
0.890=0.006
0.805=00009
0,980::0.002
0.981=0.002
0.945=0.002
0.925=00005
0.882=00.07
GraphDTTA
0,887=0.008
0.890=0.007
0.800=0.007
0,981=0.001
0,982=0.002
0.951=0.002
0.93420.002
0.888=00.055
MolTrans
0,895:+0.004
0.897=0.005
0.825=00.100
0,980+0.002
0.978=00003
0.952=0.002
0.936::00001
0,887=10.006
DrugBAN
0.903=20.005
0.902=0.004
0.834:0.008
0.982:10.002
0,980=0.003
0.960=0.001
0.948:00002
0.901=0.004
BioTS
0.937=-0.001
0.937=0.004
0.874:00.01
0,989100001
0.985=00002
0.963=0.001
0.952:10.001
0.907:10.003
Table 3: Performance comparison on the BindingDB, Human and BioSNAP datasets. (Best, Second Best), The baseline results derive from DrugBAN (Bai al., 2023).
Model
#Params.
Yeast
Human
(protein) can interact with each other. We se- lect three widely-used DTI datasets with a binary classification setting, which are BioSNAP (Zitnik et al., 2018), BindingDB (Liu et al., 2007) and
DDE
205.3K
55.83 t 3.13
62.77 +: 2.30
Moran
123.4K
53.00 + 0.50
54.67 + 4.43
LSTM
26.7M
53.62 +: 2.72
63.75
5.12
Transformer
21.3M
54.12 + 1.27
59.58 + 2.09
Human (Liu et al., 2015; Chen et al., 2020).
CNN
5.4M
55.07 + 0.02
62.60 + 1.67
ResNet
11.0M
48.91 +: 1.78
68.61 + 3.78
Baselines We compare BioT5 with two types of baselines: (1) traditional machine learning meth- ods including SVM (Cortes and Vapnik, 1995)
ProtBert
419.9M
63.72 + 2.80
77.32 1.10
ProtBent*
419.9M
53.87 + 0.38
83.61 1.34
ESM-1b
652.4M
$7.00
6.38
78.17
and Random Forest (RF) (Ho, 1995); (2) deep learning methods including DeepConv-DDe (Lee et al., 2019), GraphDTA (Nguyen et al., 2021), MolTrans (Huang et al., 2021) and DrugBAN (Bai et al., 2023), in which drug and target feature are firstly extracted by well-design drug encoder and protein encoder then fused for prediction.
ESM- lb*
652.4M
66.07
0.58
88.06
0.24
BioTS
252.1M
64.89 +: 0.43
86.2 + 0.53
Table 4: Performance comparison Yeast and Human datasets (Best, Second Best). The evaluation metric is accuracy. represents only tuning the prediction head. The baseline results derive from PEER (Xu et al., 2022).
Results The results on BioSNAP, Human, and BindingDB datasets are presented in Table 3. All statistics are obtained from five random runs. On BioSNAP and BindingDB datasets, BioT5 consis- tently outperforms other methods in various perfor- mance metrics, including AUROC, AUPRC, and accuracy. For the Human dataset, although deep learning-based models generally exhibit strong per- formance, the BioT5 model demonstrates a slight advantage over the baseline models. is worth noting that, in contrast to most deep learning-based baselines, our BioT5 does not rely on a specific de- sign tailored for molecules or proteins. A possible explanation for the superior performance of BioT5 is that the SELFIES and FASTA representations ef- fectively capture the intricate structure and function of molecules and proteins, and the interaction in- formation between them may be well-described in the contextual scientific literature or corresponding text entries in databases.
ProtBert and Eb are studied with two settings (i) freezing the protein language model parame- and only training the prediction head; (ii) fine- tuning all model parameters.
Results The results are displayed in Table 2, with all statistics derived from three random runs. In the protein solubility prediction task, BioT5 out- performs all baselines in PEER (Xu al., 2022) benchmark. In the protein localization prediction task, BioT5 the second best among all methods. Notably, ProtBert and ESM-1b are both pre-trained on a large corpus of protein sequences, which is comparable to or even larger than ours. Moreover, these models are two to three times larger than BioT5. These demonstrate the potential of BioT5 for enhanced predictive capabilities in protein prop- erty prediction by integrating textual information.
4.2
Multi-instance Prediction
4.2.1
Drug-target Interaction Prediction
4.2.2 Protein-protein Interaction Prediction
Drug-target interaction (DTI) prediction plays a crucial role in drug discovery, as it aims to pre- dict whether a given drug (molecule) and target
Protein- ppretinn interaction (PPI) prediction plays a vital role in understanding protein functions and structures, as it aims to determine the potential

---

Model
#arams.
BLEU-2
BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
Tex12Mol
RNN
56M
0.251
0.176
0.450
0.278
0.394
0.363
0.426
Transformer
76M
0.061
0.027
0.204
0.087
0,186
0,114
0.057
T5-small
77M
0,501
0.415
0.602
0.446
0.545
0,532
0.526
TS-base
248M
0.511
0,423
0.607
0.451
0.550
0.539
0.523
T5-lurge
783M
0.558
0.467
0.630
0.478
0.569
0.586
0.563
TS-small
77M
0.501
0.415
0.602
0,446
0.545
0.532
0.526
MotIS-small
77M
0.519
0.436
0.620
0.469
0.563
0,551
0.540
TS-buse
0,423
0.607
0.451
0.523
MofTS-base
248M
0,540
0.457
0.634
0.485
0.578
0.569
0.547
TS-lurge
783M
0.558
0.467
0.630
0.478
0.569
0.586
0.563
MolTS-large
783M
0,594
0.508
0.654
0,510
0.594
0.614
0.582
GPT-3.5-turbo (zero-shot)
>175B
0,050
0.261
0.088
0.204
0,161
0.352
GPT-3.5-turb (I0-shot MolReGPT)
>175B
0,565
0.482
0.623
0.450
0.543
0,585
0.560
MoIXPT
350M
0.594
0.505
0.660
0.511
0.597
0.626
0.594
BioTS
252M
0.635
0.556
0.692
0.559
0.633
0.656
0.603
Table 5: Performance comparison molecule captioning task (Best, Second Best). Rouge scores are F1 values. The Tex score between ground truth molecule and corresponding text description is 0.609. The baseline results derive from MolT5 (Edwards al., 2022). MolXPT (Liu al., 2023b), and MolReGPT (Li al., 2023).
Model
#Parums.
BLEU"
Esact?
Levenshtcing
MACCS FTS1
RDK FTS:
Morgan FTS1
FCDL
Tex2Mett
Validity
56M
0.652
0.005
0.362
4.55
Transformer
76M
0.499
0.000
$7.66
0.480
0.320
0.217
11.32
0.277
0.906
T5-small
7MM
0.741
27.203
0.704
T5 base
248M
0.762
0.069
24,950
0.731
0.805
0.545
2.48
0.499
0.660
T5 15-large
783M
13-small
77M
0.741
0.064
27.703
0.704
0.578
0.525
2.89
0.479
0.608
0.755
T5 base
248M
0.762
0.089
24.950
0.731
0.605
0.545
2.48
0.499
0.660
248M
T5- T5-lage
783M
0.854
0.279
16.721
0.823
0.731
0.670
1.22
0.552
0.902
MolTS-
0.311
GPT-3.5-turbo (zero-shut)
>175B
0.189
0.019
$2.13
0.705
0.462
0.367
2.05
0.479
0.802
GPT-3.5-turbo MolReGPT)
>1750
0,790
0.139
MoXXPT
350M
0.215
0.859
0.757
0.667
0.45
0.578
0.983
252M
0.867
0.886
0.526
Table 6: Performance comparison text-based molecule generation task (Best, Second Best). Following Edwards et al. (2022), BLEU, Exact, Levenshtein, and Validity are computed on all generated molecues while other metrics are computed only on syntactically valid molecules. The Text2Mol score for ground truth is 0.609. The baseline results derive from MolT5 (Edwards et al., 2022), MolXPT (Liu et al., 2023b), and MolReGPT (Li et al., 2023).
interactions between pairs of proteins. Following PEER (Xu et al., 2022) benchmark, we perform fine-tuning on two PPI datasets: Yeast (Guo et al., 2008) and Human (Pan et al., 2010).
information embedded in scientific text and struc- tured information from biological databases, encap- sulating the comprehensive knowledge of proteins in their varying contexts.
Baselines The baselines for comparison are the same as that in Section 4.1.2.
4.3 Cross-modal Generation
In this section, we evaluate the performance of BioT5 on the cross-modal generation task. Specifi- cally, we fine-tune BioT5 on molecule captioning and text-based molecule generation tasks. These two tasks are proposed by MolT5 (Edwards et al., 2022) and both use the ChEB1-20 dataset (Edwards et al., 2021). The evaluation metries and some inter- esting cases are introduced in Appendix and G.
Results The results are shown Table 4. All statistics are over three random runs. On two PPI datasets, BioT5 shows superior performance compared to almost all baseline models. Remark- ably, BioT5 outperforms both ProtBert and ESM- lb (with full parameters fine-tuned). This result strongly highlights the crucial role of incorporat- ing textual information during the pre-training of BioT5, which effectively establishes profound con- nections between proteins. Our model, despite being smaller, is able to harness the unstructured
4.3.1 Molecule Captioning
For the given molccule, the goal of molecule cap- tioning task is to provide a description of the given

---

molecule. As we use SELFIES sequences to repre- sent molecules, this task can be formulated as an exotic eqquence-t--squeenee translation task. Baselines The baselines include: RNN (Medsker and Jain, 2001), Transformer (Vaswani et al.. 2017), T5 (Raffel et al., 2020), MolT5 (Edwards et al., 2022), GPT-3.5-turbo with zero-shot and 10-shot MolReGPT (Li et al., 2023) settings, and MolXPT (Liu al., 2023b).
integration in biology with chemical knowledge and natural language associations, demonstrating notable improvements in various tasks.
For future work, we aim to further enrich our model by incorporating additional biological data types, such as genomics or transcriptomics data, to create more holistic biological pre-training framework. Additionally, we plan evaluate the interpretability of BioT5 predictions, aiming to pro- vide more insights into the biological systems un- der study. Thus, we foresee our work sparking further innovation in the use of AT models the field of computational biology, ultimately leading to a deeper understanding of biological systems and facilitating more efficient drug discovery.
Results The results are shown in Table 5. BioT5 only has nearly the same number of parameters as MolT5-base, but outperforms all baseline mod- els in all metrics, including those that have more parameters. The Texr2Mol score is 0.603, which is very close to the Text2Mol score of 0.609 be- tween the ground truth molecule and the corre- sponding description. We can attribute this superior performance to the unstructured contextual knowl- edge and structured database knowledge induced in BioT5 pre-training, which helps the model learn the intricate relationship between text and molecules.
Limitations
One limitation of BioT5 is conducting full- parameter fine-tuning on each downstream task. This is done because we do not observe general- ization ability among different downstream tasks using instruction-tuning (Wei et al., 2022) method. Another reason is that combining data from dif- ferent tasks using instructions results in data leak- age. For example, have noticed overlaps between the training set of BindingDB and the test sets of BioSNAP and Human. Additionally, we only demonstrate the ability of BioT5 in text, molecule, and protein modalities. Numerous other biological modalities, such as DNA/RNA sequences and cells, exist, and there are many other tasks within a single modality or across multiple modalities. Moreover, BioT5 primarily focuses on the sequence format of bio-entities, yet other formats, such 2D or 3D structures, also hold significant importance. We leave further exploration of these future work.
4.3.2 Text-Based Molecule Generation
This a reverse task of molecule captioning. Given the nature language description of the intended molecule, the goal is to generate the molecule that fits the description.
Baselines The compared baselines are the same as baselines in Section 4.3.
Results The results are presented in Table 6. BioT5 only uses parameters similar to MolTS-base yet de- livers superior performance across nearly all met- rics. Notably, the exact match score of BioT5 sur- passes the -orge by 32.8% while maintain- ing validity of 1.0. This indicates that BioTS not only generates more relevant molecules correspond- ing the given text descriptions, but also ensures a 100% validity for the generated molecules. The overall enhanced performance of BioT5 can be at- tributed to the incorporation of both contextual and database knowledge, as well as the utilization of SELFIES for molecular representation.
Acknowledgements
This work was supported by the National Key Research and Development Program of China (No. D020YFB1406700 National Natural Science Foundation of China (NSFC Grant No. 62122089), Beijing Outstanding Young Scientist Program NO. BJWZYYHDD10:9110000000 gent Social Governance Platform, Major Innova- tion & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China, the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China.
5 Conclusions and Future Work
and Intelli-
In this paper, we propose BioT5, a comprehensive pre-training framework capable of capturing the underlying relations and properties of bio-entities by leveraging both structured and unstructured data sources with 100% robust molecular representa- tion. Our method effectively enriches cross-modal

---

References
Bioinformatics, Darko Butina. 1999. Unsupervised data base clustering based on daylight's fingerprint and tanimoto similar- ity: A fast and automated way to cluster small and large data sets. J. Chem. Inf. Comput. Sci., 39(4):747- 750.
2023. Uniprot: the universal protein knowledgebase in
2023. Nucleic Acidy Research, 51(DI):D533-DSS) DS31.
José Juan Almagro Armenteros, Casper Kaae Søn- derby, Søren Kaae Sønderby, Henrik Nielsen, and Ole Winther. 2017. Deeploc: prediction of protein subcellular localization using deep learning. Bioin- form., 33(1)3333233338333 3395.
Kathi Canese and Sarah Weis. 2013. Pubmed: the bibli- ographic database. The NCBI handbook, 2(1).
Dong- Cao, Qing-Song Xu, and Yi-Zeng Liang, 2013. propy: a tool to generate various modes of chou's 's pseaac. Bioinform. 29(7):960-965
AstraZeneca. 2023. A big future for small molecules: targeting the undruggable.
Lifan Chen, Xiaoqin Tan, Dingyan Wang, Feisheng Zhong, Xiaohong Liu, Tianbiao Yang, Xiaomin Luo, Kaixian Chen, Hualiang Jiang, and Mingyue Zheng. 2020. Transformerepi: improving compound- protein interaction prediction by sequence-based deep learning with self-attention mechanism and label reversal experiments. 36( 16):44444444
Viraj Bagal, Rishal Aggarwal, P. K. Vinod, and U. Deva Priyakumar. 2022. Molgpt: Molecular generation using a transformer-decoder model. J. Chem. Inf. Model., 6269)2206.
Peizhen Bai, Filip Miljković, Yan Ge, Nigel Greene, Bino John, and Haiping Lu. 2021. Hierarchical clus- tering split for low-bias evaluation of drug-target in- teraction prediction.
2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). pages 141-644. IEEE.
Seyone Chithramanda, Gabriel Grand, and Bharath Ramsundar. 2020. Chemberta: Large-scale self- supervised pretraining for molecular property pre- diction. arXiv preprint 2rX1v:2010.09885.
Peizhen Bai, Filip Miljković, Bino John, and Haiping Lu. 2023. Interpretable bilinear attention network with domain adaptation improves drug- target predic- tion. Nature Machine Intelligence, 5(2):126
Corinna Cortes and Vladimir Vapnik. 1995. Support- vector networks. Machine learning, 20(3):273-297.
Suresh Dara, Swetha Dhamercherla, Surender Singh Jadav, CH Madhu Babu, and Mohamed Jawed Ahsan. 2022. Machine learning in drug discovery: a review. Artificial Intelligence Review, (5(3):1947-199)
Satanjeev Banerjee and Alon Lavie. 2005, METEOR: am automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the Workshop on Intrinsic and Extrinsic Evalua- tion Measures for Machine Translation and/or Sum- marization @ ACI. 2005, Ann Arbor, Michigan, USA, June 29, 2005. pages 65 55-72. Association for Compu- tational Linguistics.
Joseph L Durant, Burton A Leland, Douglas R Henry. and James G Nourse. 2002. Reoptimization of mdl keys for use in drug discovery. Journal of chemi- cal information and computer sciences, 42(6): 1273- 1280.
Iz Beltagy, Kyle Lo. and Arman Cohan. 2019. Scib- ert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joini Conference on Natural Language Processing, EMNIIP-ICNNPP 2019, Hong Kong, China, November 3-7, 2019, pages 3613-3618. Association for Computational Linguistics.
Carl Edwards, Tuan Manh Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. 2022. Trans- lation between molecules and natural language. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 375-413. Association for Compu- tational Linguistics.
Emmanuel Boutet, Damien Lieherherr, Michael Tog- nolli, Michel Schneider, and Amos Bairoch. 2007.
Carl Edwards, ChengXiang Zhai, and Heng Ji. 2021. Text2mol: Cross-modal molecule retrieval with nat- ural language queries. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 595-607, Association for Computational Linguistics.
the manually annotated sec- tion of the uniprot knowledgebase. Plant bioinfor- matics: methods and protocols, pages 89-112.
Uniprotkb/swiss-pret
Rodney Brister, Danso Ako-Adjei, Yiming Bao, and Olga Blinkova. 2015. Nebi viral genomes resource. Nucleic acids research, 43(D)) ((DD)):D51--SS))
Ahmed Elnaggar. Michael Heinzinger, Christian Dal- lago, Ghalia Rehawi, Yu Wang. Llion Jones, Tom Gibbs. Tamas Feher, Christoph Angerer, Martin Steinegger, et al. 2021. Prottrans: Toward under- standing the language of life through self-supervised learning. IEEE transactions on patter analysis and machine intelligence, 44(10):71 12-7127.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 1877-11-1

---

Xiaomin Fang, Lihang Liu, Jieqiong Lei. Donglong He, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. 2022. Geometry-enhanced molecular representation learning for property pre- diction. Nature Machine Intelligence, 4(2):127-134,
Kexin Huang, Cao Xiao, Lucas Glass, and Jimeng Sun. 2021. MolTrans: Molecular interaction transformer for drug-target interaction prediction. Bioinformat- ics, 37:830- -836.
John J Irwin, Khanh G Tang, Jennifer Young, Chin- zorig Dandarchuluun, Benjamin R Wong, Munkhzul Khurelbaatar, Yurii S Moroz, John Mayfield, and Roger A Sayle. 2020. Zinc20-a free ultralarge-scale chemical database for ligand discovery. Journal of chemical information and modeling, 60(12):6065 6073.
Zhi-Ping Feng and Chun-Ting Zhang. 2000. Prediction of membrane protein types based on the hydrophobic index of amino acids. Journal of protein chemistry, 19:269-275.
Noelia Ferruz, Steffen Schmidt, and Birte Höcker. 2022. Protgpt2 is deep unsupervised language model for protein design. Nature communications, 13(1):4348.
Sameer Khurana, Reda Rawi, Khalid Kunji, Gwo- Yu Chuang. Halima Bensmail, and Raghvendra Mall. 2018. Deepsol: deep learning framework for sequence-based protein solubility prediction. Bioin- form.. 34(15):2605-2611
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Compulational Linguisties and the 11th International Juint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers). Virtual Event, August 1-6, 2021, pages 3816 3830. Association for Computational Linguis- tics.
Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindu- lyte, Jia Hc, Siqian Hc, Qingliang Li, Benjamin A Shoemaker, Paul A Thiessen, Bo Yu, et al. 2023. Pubchem 2023 update. Nucleic Acids Research, 51(D1):D1333-D132
Sunghwan Kim, Paul A Thiessen, Tiejun Cheng, Jian Zhang, Asta Gindulyte, and Evan E Bolton. 2019. Pug-view; programmatic access to chemical annota- tions integrated in pubchem. Journal of cheminfor- matics, 11(1)11-1.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-specific lan- guage model pretraining for biomedical natural lan- guage processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1-23.
Thomas N. Kipf and Max Welling. 2017. Semi- supervised classification with graph convolutional networks. In Sth International Conference on Learn- ing Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings. OpenReview.net.
Yanzhi Guo, Lezheng Yu, Zhining Wen, and Menglong Li. 2008. Using support vector machine combined with auto covariance predict protein-protein in- teractions from protein sequences. Nucleic acids research, 302))3300-
Mario Krenn, Qianxiang Ai, Senja Barthel, Nessa Carson, Angelo Frei, Nathan C Frey, Pascal Friederich. Théophile Gaudin, Alberto Alexander Gayle, Kevin Maik Jablonka, et al. 2022. Selfies and the future of molecular string representations. Parterns, 3000)8
Janna Hastings, Gareth Owen, Adriano Dekker, Mar- cus Ennis, Namrata Kale, Venkatesh Muthukrishnan, Steve Turner, Neil Swainston, Pedro Mendes, and Christoph Steinbeck. 2016. Chebi in 2016: Improved services and an expanding collection of metabolites. Nucleic acids research, 44D1))D1214-D1219
Mario Kremn, Florian Häse, AkshatKumar Nigam, Pas- cal Friederich, and Alan Aspuru-Guzik. 2020. Self- referencing embedded strings (selfies); A 100% ro- bust molecular string representation. Machine Learn- ing: Science and Technology, 1(4):045024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778.
Taku Kudo and John Richardson. 2018. Sentencepiece:
A simple and language independent subword tok- enizer and detokenizer for neural text processing. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 November 4, 2018, pages 66-71. Asso- ciation for Computational Linguistics.
Stephen Heller, Alan McNaught, Stephen Stein, Dmitrii Tchekhovskoi, and Igor Pletnev. 2013. Inchi-the worldwide chemical structure identifier standard. Journal of cheminformatics, 5(1):1-9.
Tin Kam Ho. 1995. Random decision forests. In Pro- ceedings of 3rd International Conference on Doc- ument Analysis and Recognition, volume 1, pages 278-282.
Greg Landrum. 2021. Rdkit: Open-source cheminfor- matics software. GitHub release.
Ingoo Lee,, Jongsoo Keum, and Hojung Nam. 2019. DeepConv-DTI: Prediction of drug-target interac- tions via deep learning with convolution on protein sequences. PLoS Computational Biology, 15.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 98):1735- 1780.

---

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics,
Linguistics (Volume 2: Short Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 1606-1616. Association for Computational Linguisties.
1240.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Jiatong Li, Yunqing Liu, Wenqi Fan, Xiao-Yong Wei, Hui Liu, Jiliang Tang, and Qing Li. 2023. Empower- ing molecule discovery for molecule-caption transla- tion with large language models: A chatgpt perspec- tive. arXiv preprini arXiv:2306.06615
Rengian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics. 23(6).
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 81.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al. 2022. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv.
Yizhen Luo, Kui Huang, Massimo Hong, Kai Yang. Jiahuan Zhang. Yushuai Wu, and Zaiqin Nie. 2023. Empowering ai drug discovery with explicit and im- plicit knowledge. arXiv preprint arXiv:2305.01523.
Larry R Medsker and LC Jain. 2001. Recurrent neural networks, Design and Applications, 37.
David J Lipman and William R Pearson. 1985. Rapid and sensitive protein similarity searches. Science, 227(4693): 1435-993):135-1144
Frederic P Miller, Agnes F Vandome, and John McBrew- ster. 2009. Levenshtein distance: Information theory, computer science, string (computer science), string metric, damerau? levenshtein distance, spell checker, hamming distance.
Carolyn E Lipscomb. 2000. Medical subject headings (mesh). Bulletin of the Medical Library Association, 88(3):265.
Hui Liu, Jianjiang Sun, Jihong Guan, Jie Zheng, and Shuigeng Zhou. 2015. Improving campound-protehi interaction prediction by building up highly credible negative samples. Bioinformatics, 31 (1)1222221-
Piotr Nawrot. 2023. n5
Thin Nguyen, Hang Le, T. Quinn, Tri Minh Nguyen, Thuc Duy Le. and Svetha Venkatesh. 2021. GraphDTA: Predicting drug-target binding affin- ity with graph neural networks. Bioinformatics. 37(8):1140-114 1147.
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. 2022. Pre- training molecular graph representation with 3d ge- ometry. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Noel O' Boyle and Andrew Dalke. 2018. Deepsmiles: adaptation of smiles for use in machine- of chemical structures.
Shengehao Liu, Yutao Zhu, Jiarui Lu, Zhao Xu, Weili Nie, Anthony Gitter, Chaowei Xiao, Jian Tang, Hongyu Guo, and Anima Anandkumar. 2023a. A text-guided protein design framework. arXiv preprint arXiv:2302.04611
Keiron O' O'Sh and Ryan Nash. 2015. introduction to convolutional neural networks. arXiv preprint arXiv:1511.08455
Xiao-Yong Yong Pan, Ya-Nan Zhang, and Hong-Bin Shen. 2010. Large-scale prediction of human protein- pro- tein interactions from amino acid sequence based on latent topic features. Journal of proteome research, 9 499-4-9000
Tiqing Liu, Yuhmei Lin, Xin Wen, Rohert N Jorissen, and Michael K Gilson, 2007. Bindingdb: a web- accessible database of experimentally determined pigtein-ligand binding affinities. Nucleic acids re-
search. 35(uppl 1):D198 D201.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, July 6-12, 2002, Philadelphia. PA, USA, pages 311-318. ACL.
Zequn Liu, Wei Zhang, Yingee Xia, Lijun Wu, Shu- fang Xie, Tao Qin, Ming Zhang, and Tie-Yan Liu. 2023b. Molxpt: Wrapping molecules with text for generative pre-training In Proceedings of the 61 Annual Meeting of the Association for Computational
William R Pearson and David J Lipman. 1988. Im- proved tools for biological sequence comparison. Proceedings of the National Academy of Sciences. 85(8):2444 2448.

---

Suraj Peri, J. Daniel Navarro, Ramars Amanchy, Troels Z Kristiansen, Chandra Kiran Jonnalagadda, Vineeth Surendranath, Vidya Niranjan, Babylak- shmi Muthusamy, TKB Gandhi, Mads Gronborg, et al. 2003. Development of human protein refer- ence database as an initial platform for approach- ing systems biology in humans. Genome research, 13(10):2363-333
Nadine Schneider, Roger Sayle, and Gregory Lan- drum. 2015. Get your atoms in order an open- source implementation of novel and robust molecu- lar canonicalization algorithm. J. Chem. Inf. Model., 55(10):211
Martin Steinegger and Johannes Söding. 2018. Cluster- ing huge protein sequence sets in linear time. Nature communications, 911:2552
Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp
Teague Sterling and John J. Irwin. 2015. ZINC 15 ligand discovery for everyone. J. Chem. Inf. Model., 55 55(11):2324-233 2337.
Hochreiter, and Günter Klambauer. 2018. Fréchet chemmet distance: A metric for generative models for molecules in drug discovery. J. Chem. Inf.
Bing Su, Dazhao Du Zhao Yang. Yujie Zhou, Jiang- meng Li, Anyi Rao, Hao Sun, Zhiwu Lu, and Ji- Rong Wen. 2022. A molecular multimodal founda- tion model associating molecule graphs with natural language. arXiv preprint arXiv:2209.0548)
Alec Radford, Karthik Narasimhan, Tim Salimans, Ily Sutskever, et al. 2018. Improving language under- standing by generative pre-training.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 18):9
Mujeen Sung, Minbyul Jeong, Yonghwa Choi, Donghyeon Kim, Jinhyuk Lee, and Jaewoo Kang. 2022. Bern2: an advanced neural biomedical named entity recognition and normalization tool. Bioinfor- matics, 58(20):4837-488
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21()):5455-551.
Baris Suzek, Hongzhan Huang. Peter McGarvey, Raja Mazumder, and Cathy Wu. 2007. Uniref: compre- hensive and non-redundant uniprot reference clusters. Bioinformatics, 12110)1122221 1288.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint wrXiv:2211.00000 1.00855.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, Lawrence Zitnick, Jerry Ma, et al. 2021. Biologi- cal structure and function emerge from scaling unsu- pervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, (18(1)::2016239118
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Stephen F. Robertson and Hugo Zaragoza. 2009, The probabilistic relevance framework: BM25 and be- yond. Found. Trends Inf. Retr. 3(4)3333-389
Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. 2022. Molecular contrastive learn- ing of representations via graph neural networks. Na- ture Machine Intelligence, 4(3):279 -277.
David Rogers and Mathew Hahn. 2010a. Extended- connectivity fingerprints. Journal of chemical infor- mation and modeling. 5055):742-754.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quac V. Le. 2022. Finetuned language models zero-shot learners. In The Tenth International Conference on Learning Representu- tions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
David Rogers and Mathew Hahn. 2010b. Extended- connectivity fingerprints. J. Chem. Inf. Model., 50K5):742-754.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. 2020. Self-supervised graph transformer on large- scale molecular data. Advances in Neural Informa- tion Processing Systems, 33:12559-1123 12571.
David Weininger. 1988. Smiles, a chemical language and information system. 1. intrrdduction to methodol- ogy and encoding rules. Journal of chemical infor- mation and computer sciences, 28(1):31-33
Vijayakumar Saravanan and Namasivayam Gautham. 2015. Harnessing computational biology for exact linear b-cell epitope prediction: novel amino acid composition-ben based feature descriptor. Omics: a jour- nal of integrative biology, 19(10)644-665)
David Weininger, Arthur Weininger, and Joseph L. Weininger. 1989 Smiles. 2. algorithm for genera- tion of unique smiles notation. Journal of chemical information and computer sciences, 2102)

---

David Wishart, Yannick D Feunang, An Guo, Elvis Lo, Ana Marcu, Jason R Grant, Tanvir Sajed, Daniel Johnson, Carin Li, Zinat Sayeeda, et al. 2018. Drug- bank 5.0: major update the drugbank database for 2018. Nucleic acids research, 46(D1):D1074- D1082.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu. Karl Leswing, and Vijay Pande. 2018. Moleculenet: a benchmark for molecular machine learning. Chem- ical science, 9(2):513-530
Hanwen Xu, Addie Woicik, Hoifung Poon, Russ Alt- man, and Sheng Wang. 2023a. Multilingual trans- lation for zero-shot biomedical classification using biotranslator. Nature Communications, 14(1):738.
Minghao Xu, Xinyu Yuan, Santiago Miret, and Jian Tang. 2023b. Protst: Muliimodalitty learning of pro- tein sequences and biomedical texts. arXiv preprint arXiv:2301.12040.
Minghao Xu, Zuobai Zhang, Jiarui Lu, Zhaocheng Zhu, Yangtian Zhang, Ma Chang, Runcheng Liu, and Jian Tang. 2022. Peer: comprehensive and multi-task benchmark protein sequence understanding. vances in Neural Information Processing Systems, 35:35156-355 -31733
Zheni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2022. deep-learning system bridging molccule structure and biomedical text with comprehension comparable human professionals. Nature commu- nications, 136))862.
7aixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. 2021. Motif-based graph self- supervised learning for molecular property predic- tion. Advances in Neural Information Processing Systems, 34:15870- 15882.
Marinka Zitnik, Rok Sosic, and Jure Leskovec. 2018. Biosnap datasets: Stanford biomedical network dataset collection. Note: http://smap. stanford. edu/biodata Cited by. 5(1).

---

A Reproducibility
amino acid and C Dictionary and SELFIES Conversion
The codes for our BioT5 are available at https: / //g.ttub...cmm//22//e/////10
For molecule-related datasets, when only SMILES is provided, we utilize selfies² package to convert SMILES into SELFIES.
B NER and Entity Linking Process
D Molecule-Text Generation Metrics
We follow KV-PLM (Zeng al., 2022) and MolXPT (Liu et al., 2023b) to conduct Named Entity Recognition (NER) and Entity Linking for the bio-entity names appearing in the scientific text. More specifically, we firstly utilize BERN2 (Sung et al., 2022), an advanced neural Named Entity Recognition (NER) tool in biomedical fields, to identify all instances of molecule protein men- tions. Subsequently, we map them correspond- ing entities within publicly accessible knowledge bases. For molecule, we use ChEBI (Hastings et al., 2016) and MeSH (Lipscomb, 2000) database, and for protein we use NCBI Gene (Brister et al.. 2015) database. Then we can get the correspond- ing molecule SELFIES and protein FASTA for the matched entities. As shown in Figure 4, for
We follow Edwards et al. (2022) to use the same evaluation metrics for molecule captioning and text- based molccule generation tasks. To ensure fair comparison, convert the molecule SEIFLES SMILES before calculating these metrics.
D.1 Molecule Captioning Metrics
In the molecule caption task, NLP metrics like BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and METEOR (Banerjee and Lavie, 2005) are uti- lized to evaluate the closeness the generated de- scription to the ground truth description. We also adopt Texr2Mol metric, which is proposed by Ed- wards et al. (2021) and employ pre-trained mod- els measure the similarity between the descrip- tion and ground truth molecule. Higher similar- ity means that the given text description is more relevant to the molecule, and the Text2Mol score between the ground truth description and molecule is also computed for comparison.
Effect chloraquine cultured fibrablasts: release of Iysasomal hydrol asesand of uptake.
Entity Linking
chloraquin idSH D002738 + SELFIES: BancIIMIIBancch1RRRinnICCCC
lysosomal hydrolases Gene id: 3988
FASTA: MKMRFFG...
D.2 Text-based Molecule Generation Metries
Associate compound & protein with SELFIES & FASTA
Since molecules can be represented in bio- sequence structure, NLP metrics like BLEU (Pa- pineni et al., 2002) and Exact Match scores be- tween generated and ground truth SMILES are di- rectly applied for evaluation. Additionally, we also report performance on molecule-specific metrics: three molecule fingerprints (FTS) similarity scores- MACCS (Durant et al., 2002), RDK (Schneider et al., 2015), and Morgan (Rogers and Hahn, 2010a); Levenshtein distance (Miller et al., 2009); FCD score (Preuer et al., 2018), which measures molecule similarities according to biological infor- mation based pre-trained "ChemNet": validity, which is the percentage of the valid SMILES that can be processed by RDKit (Landrum, 2021). The Tex metric also used to measure the simi- larity between the molecule SMILES and ground truth description.
Effect of (enmm:CCCICINIIOroon////illllll ceom on cultured fibroblasts: release of hysesomalyydroleee sequence is Inhibition uptake.
Figure 4: Wrapped text matching and mapping process.
molecule, directly replace all the detected names with its SELFIES string: for protein, due to the length limitation, if sentence consists of more than one protein entity, we only randomly choose one to append the protein FASTA to the name. The motivation for appending protein FASTA in- stead of replacing is that the genes are transcribed and translated to generate proteins. Therefore, un- like the molccule names directly representing the molecule, the relation between gene names and pro- tein FASTA is indirect. Note that the replacement or appendage will not happen every sentence. Only those with detected bio-entities will done the above process.
htttp:: thub. cww.aspuru-aaaaeaacaoeee grruik//grpp///// selfies

---

E Pre-training Details
F.1 Single-instance Prediction
F.1.1 Molecule Property Prediction
E.1 Special Tokens
All the datasets are split using an 8 : 1 ratio for train, validation, and test, respectively. We use the scaffold splitting method, in which molecules are categorized according the Bemis-Murcko scaffold representation.
In the pre-training of BioTS, we conduct translation tasks on molecule-text pairs and protein-text pairs extracted from PubChem (Kim et al., 2023) and Swiss-Prot (Boutet al., 2007) separately. We format the text description from these database entries using special tokens, which serve as an- chors for embedding scientific context and struc- ture. For molecule, we use MOLECULE NAME and DESCRIPTION to represent its name and de- scription including properties, functions, etc. For protein, similar to Xu et al. (2023b), we use PRO- TEIN NAME, FUNCTION, SUBCELLULAR LO- CATION, and PROTEIN FAMILIES represent its name, functions, location and topology in the cell, and families it belongs to. A complete text description created by concatenating these fields sequentially, omitting any missing fields. Through special tokens, we can effectively encode the intri- cate information associated with each bio-entity.
Datasets
(I) The BBBP (Blood-Brait Barrier Penctration) is curated with the intention of aiding the modeling and forecasting of barrier permeability. It com- prises compounds that are categorized using binary labels, indicating whether they can penetrate the blood-brain barrier.
(2) The Tox21 ("Toxicology in the 21 Century") initiative established a publicly accessible database that quantifies the toxicity levels of various com- pounds. The dataset encompasses qualitative toxic- ity assessments (binary labels) for approximately 8,000 compounds, targeting 12 distinct biological pathways such as nuclear receptors and stress re- sponse mechanisms.
E.2 Hyper-parameters
(3) The ClinTox dataset contrasts FDA-approvee drugs with those that have been unsuccessful in clinical trials owing to toxicity issues. This dataset incorporates two classification objectives for 1,491 drug compounds with established chemical struc- tures: (i) Presence or absence of toxicity in clinical
We use the codebase nano75 (Nawrot, 2023) for BioT5 pre-training. We pre-train BioT5 for 350K steps on eight NVIDIA 80GB A100 GPUs. The batch size is 96 per GPU, in which a batch includes six types of data. The "translation" directions for molecule-text and protein-text pair are randomly selected for each sample with probability of 0.5. We use AdamW (Loshchilov and Hutter, 2019) with Root Mean Square (RMS) scaling Optimizer for optimization. The learning rate scheduler is cosine annealing with the base learning rate set to 1e and the minimum learning rate set 1e 5. The number of warm-up steps is 10,000 and the dropout rate is 0.0. The maximum input length for pre-training is 512. Unlike absolute position encodings, T5 (Raffel et al., 2020) use relative po- sition encodings. This makes the model flexible inputs of different lengths, which is helpful for downstream fine-tuning.
trials; (ii) approved or unapproved by FDA.
(4) The HIV dataset assesses the inhibitory poten- tial of over 40,000 compounds on HIV replication. The screening outcomes were classified into three categories: Confirmed Inactive (CI), Confirmed Ac- tive (CA), and Confirmed Moderately Active (CM). Subsequently, the latter two labels were combined, transforming the task into a binary classification between inactive (CI) and active (CA and CM) cat- egories.
(5) The BACE dataset presents quantitative IC50 values and qualitative binary labels for collec- tion of inhibitors targeting human beta-sccretase (BACE-1).
(6) The SIDER (Side Effect Resource) a compre- hensive database that consists of marketed drugs and their corresponding adverse drug reactions (ADR). The drug side effects in SIDER are or- ganized into 27 system organ classes, adhering to the MedDRA classifications. This dataset encom- passes data for 1,427 approved drugs. Baselines
F Fine-tuning Details
In this section, provide details about down- stream tasks, including datasets, compared base- lines, and prompts. Some statistics about down- stream tasks are shown in Table When displaying prompts, (SELFIES) refers to the molecule SELF- IES, and (FASTA) refers to the protein FASTA.
(1) GROVER (Rong et al., 2020) incorporates Mes-

---

Task/Dataset
Task Type
#Train
#Validation
#Test
Molecule Property Prediction
ВВВР
Molecule-wise Classification
1,631
204
204
Tox21
Molecule-wise Classification
6,264
783
784
Coi
Molecule-wise Classification
1,181
148
148
HIV
Moleculo-wise Classification
32,901
4,113
4,113
BACE
Molecule-wise Classification
1,210
151
152
SIDER
Molecule-wise Classification
1,141
143
143
Protein Property Prediction
Solubility prediction
Protein-wise Classification 62,478 5,184
1,999
1,999
Localization prediction
Protein-wiss wise Classification
1,749
1,749
Drug-targe Interaction Prediction
BioSNAP
Molecule-protein Classification
19,224
2,747
5,493
Human
Molecule-protein Classification
4,197
600
1,200
BindingDB
Molecule-protele Classification
50,149
5,604
5,505
protein-protein Interaction Prediction
Yeast
Protein-pate Classification 4,945
394
394
Human
Molecule-pair Classification
35,669
237
237
Molecule Captioning and Text-based Molecule Generation
ChEBI-20
Molecule-text Translation 26,407
3,301
3,300
Table 7: Downstream task descriptions, including task dataset name, type, and the size of each split.
sage Passing Networks within a Transformer-style architccture and pre-trained on large-scale molec- ular dataset without any supervision. G-Contextual and G-Motif are two variants of GROVER, which are pre-trained on contextual property prediction task and motif prediction task, respectively.
combination of molecular names and SMILES se- quences allows the model to capture both textual and structural information, thereby enhancing its performance in various downstream tasks.
(7) Galactica (Taylor et al., 2022) is a large GPT- based language model which is pre-trained on var- ious corpus like papers, codes, SMILES, protein sequences, etc.
(2) GraphMVP (Liu et al., 2022) employs self- supervised learning by capitalizing on the corre- spondence and consistency between molecule 2D topological structures and 3D geometric views.
(8) MoMu (Su et al., 2022) is pre-trained using molecular graphs and their semantically related textual data through contrastive learning.
(3) MGSSL (Zhang et al., 2021) incorporates a novel self-supervised motif generation framework for Graph Neural Networks.
(9) MolXPT (Liu et al., 2023b) is a unified GPT- based language model for text and molecules pre- trained on "wrapped" text, where molecule names are replaced with corresponding SMILES.
(4) MolCLR (Wang et al., 2022) is self-supervised learning framework that capitalizes on substantial unlabelled unique molecules (approximately 10 million)
Prompts
For the six MoleculeNet datasets mentioned above. the prompts only differ in the Task Definition. Therefore,
(5) GEM (Fang et al., 2022) features a specially de- signed geometry-
graph neural network archi- tecture and several dedicated geometry-level self- supervised learning strategies to capture molecular geometry knowledge effectively.
will only provide the Instruction and Output for the first dataset, and the remaining datasets will follow the same format.
(1) BBBP
(6) KV-PLM (Zeng al., 2022) is BERT-based model designed for molecular representation learn- ing, in which molecule SMILES are appended af- ter its name during the praitigg process. This
Task Definition: Definition: Molecule property prediction task (a binary elassification task) for the BBBP dataset. The blood brain barrier pene- tration (BBBP) dataset is designed for the model-

---

ing and prediction of barrier permeability. If the given molecule can penetrate the Ahood-brain bar- rier, indicate via "Yes". Otherwise, response via "No".
"Yes". Otherwise, response via "No". where (side effect) refers to the corresponding side effects for each subtask.
F.1.2 Protein Property Prediction
Now complete the following example- Imput: Molecule: (bom) (SELFIES) (eom) Output:
Instruction:
Datasets
Output: Yes for inhibitor and No instead.
(1) Solubility prediction to predict whether a pro- tein is soluble or not. We follow the same splitting method with DeepSol (Khurana et al., 2018).
(2) Tox21
Task Definition: Definition: Molecule property prediction task (a binary classification task) for for the Tox21 dataset. The Tox21 dataset contains qualita- live loxicity measurements for 8k compounds on 12 different targets, including nuclear receptors and stress response pathways. the given molecule can activate/change/affe (target), indicate via "Yes". Otherwise, response via "No". where (target) represents the corresponding receptor, domain, ele- ment, gene, potential, or pathway for each subtask. (3) ClinTox
(2) Localization prediction aims predict whether "soluble", which simple version of subcellular localization pre- diction task. We follow the same splitting method with DeepLoc (Armenteros et al., 2017)-
protein is tnembrane-bounde
Baselines
(1) Feature engineers. The DDE (Dipeptide Devi- ation from Expected Mean) (Saravanan and Gau- tham, 2015) feature descriptor, consisting of 400 dimensions, is based on the dipeptide frequency within a protein sequence. The Moran feature descriptor (Moran correlation) (Feng and Zhang, 2000), with 240 dimensions, characterizes the dis- tribution of amino acid properties within a protein sequence.
Task Definition: Definition: Molecule property prediction task (a binary classification task) for the ClinTox dataset. The ClinTox dataset com- pares drugs approved by the FDA and drugs that have failed clinical trials for toxicity reasons. If the given molecule is (Subtask). indicate via "Yes". Otherwise, response via "No". where the (Subtask) is either toxic or FDA approved.
(2) Protein sequence encoders, including LSTM (Hochreiter and Schmidhuber, 1997), Transformers (Vaswani et al., 2017), CNN (O'Shea and Nash, 2015) and ResNet (He et al., 2016). The amino acid features in the last layer are aggregated for final prediction.
(4) HIV
Task Definition: Definition: Molecule property prediction task (a binary classification task) for the HIV dataset. The HIV dataset introduced by the Drug Therapeutics Program (DTP) AIDS Antiviral Screen, which tested the ability to inhibit HIV replication for over 40, ,00 compounds. If the given molecule can inhibit HIV replication, indi- cate via "Yes". Otherwise, response via "No".
(3) Pre-trained protein language models. Prot- Bert (Elnaggar et al., 2021) and ESM-1b (Rives et al., 2021) are both pre-trained on a massive dataset of protein sequences using the masked lan- guage modeling (MLM) objective. Specifically, ProtBert is pre-trained on 2.1 billion protein se- quences obtained from the BFD database (Steineg- ger and Söding, 2018), while ESM-1b is pre- trained on a smaller dataset of 24 million protein sequences sourced from UniRef50 (Suzek et al., 2007).
(5) BACE
Task Definition: Definition: Molecule property prediction task (a binary classification task) for the BACE dataset. The BACE dataset provides qualitative (binary label) binding results for a set of inhibitors of human beta-secrelase (BACE-I). If the given molecule can inhibit BACE-1, indicate via "Yes". Otherwise, response via "No".
Prompts
(1) Solubility prediction
Task Definition: Protein solubility prediction task (a binary classification task) for the solubility dataset. If the given protein is soluble, indicate via "Yes". Otherwise, response via "No".
(6) SIDER
Task Definition: Definition: Molecule property prediction task (a binary classification task) for the SIDER dataset. The Side Effect Resource (SIDER) is a dataset of marketed drugs and adverse drug re- actions (ADR). If the given molecule can cause the side effect of (side effect), indicate via
Instruction Now complete the following example- Input: Protein: (bom) (FASTA) (eom) Output:.
Output: Yes for soluble protein No instead. (2) Localization prediction

---

Task Definition: Protein subcellular localization task (a binary classification task). If the given pro- tein is membrane-bound indicate via "Yes". Other- wise (the protein is soluble), response via "No". Instruction Now complete the following example
fed into fully connected neural network for final prediction.
(4) GraphDTA (Nguyen et al., 2021) uses graph neural networks (GNNs) for the encoding of drug molecular graphs, and a CNN is used for the encod- ing of protein sequences. The derived vectors of the drig and protein representation are concatenated for interaction prediction.
Input: Protein: (bom) (FASTA)(eom) Outpu:. Output: Yes for membrane-bound protein or No for soluble protein.
(5) MolTrans (Huang et al., 2021) uses transformer architecture to encode drug and protein. Then a CNN-based interaction module used to capture their interactions.
F.2 Multi-instance Prediction
F.2.1
Drug-target Interaction Prediction
Datasets
(6) DrugBAN (Bai et al., 2023) use Graph Convo- lution Network (GCN) (Kipf and Welling, 2017) and ID CNN encode drug and protein sequences. Then bilinear attention network adopted to learn pairwise local interactions between drug and protein. The resulting joint representation is de- coded by a fully connected neural network.
(I) BioSNAP (Zitnik et al., 2018) is derived from the DrugBank database (Wishart et al., 2018) and created by Huang et al. (2021) and Zitnik et al. (2018). It consists of 4,510 drugs and 2,181 pro- teins. This dataset balanced, containing both validated positive interactions and an equal number of randomly selected negative samples from unseen pairs
Prompts
Task Definition: Desinition: Drug target interac- tion prediction task (a binary classification task) for the (Dataset) dataset. If the given molecule and protein interact with each other, indicate via "Yes". Otherwise, response via "No". where (Dataset) is one of the three DTI datasets men- tioned above.
(2) BindingDB (Liu al., 2007) is an accessible online database that contains experimentally val- idated binding affimities. Its main focus is on the interactions between small drug-like molecules and proteins. We follow Bai ct al. (2023) to use a modi- fied version of the BindingDB dataset, which was previously constructed by Bai et al. (2021) with reduced bias.
Instruction: Now complete the following example- Input: Molecule: (bom) (SELFIES)(eon)) Protein: (bom) (FASTA) (eom) Output: Output: Yes for positive label or No instead.
(3) Human (Liu al., 2015; Chen al., 2020) is constructed with the inclusion of highly credible negative samples. Following Bai et al. (2023), we also use a balanced version of the Human dataset, which contains an equal number of positive and negative samples.
F.2.2 Protein-protein Interaction Prediction
Datasets
(1) Yeast (Guo et al., 2008) involves determining whether two yeast proteins interact or not. The negative pairs are derived from distinct subcellular locations. Following (Xu et al., 2022), the dataset is split and removed redundancy according protein sequences similarity, which allows for the evalu- ation of generalization across dissimilar protein sequences.
Baselines We compare the performance of BioT5 with the following six models on DTI task.
(I) Support Vector Machine (Cortes and Vap- nik, 1995) (SVM) on the concatenated finger- print ECFP4 (Rogers and Hahn, 2010b) (extended connectivity fingerprint, up to four bonds) and PSC (Cao et al., 2013) (pseudo-amino acid compo- sition) features.
(2) Human (Pan et al., 2010) involves determin- ing whether two human proteins interact or not. It comprises positive protein pairs sourced from the Human Protein Reference Database (HPRD) (Peri et al., 2003) and negative pairs derived from dif- ferent subcellular locations. The dataset splitting scheme is similar to that of Yeast PPI prediction with an 8 : ratio for train/validation/tess.. Baselines The compared baselines are the same as the protein property prediction task Sec-
(2) Random Forest (Ho, 1995) (RF) on the concate- nated fingerprint ECFP4 and PSC features.
(3) DeepConv-DTI (Lee et al., 2019) uses fully connected neural network to encode the ECFP4 drug fingerprint and a Convolutional Neural Net- work (CNN) along with global max-pooling layer to extract features from protein sequences. Then the drug and protein features are concatenated and

---

tion F.1.2.
by BM25 (Robertson and Zaragoza, 2009) for text- based molecule generation task.
Prompts
Task Definition: Protein protein interaction pre- diction task (a binary classification task) for the (Dataset) dataset. If the given two yeast proteins (Protein_A A and Protein_Bi can interact with each other, indicate via "Yes". Otherwise, response via "No". where (Dataset) either yeast or human. Instruction: Now complete the following example- Input: Protein_A :: (bom) (FASTA) (eom) Protein__ B: (bom) (FASTA) (eom) Output:
(6) MolXPT (Liu al., 2023b) jointly trained on molecule SMILES from PubChem (Kim et al., 2023), biomedical text from PubMed (Canese and Weis, 2013), and "wrapped" text in which molecule names are replaced with corresponding SMILES.
Prompts
Different from the classification task in which the ground truth output either Yes or No, the output for molecule captioning task is text sequence.
Output: Yes for positive label or No instead.
Task Definition: Definition: You are given a molecule SELFIES. Your job is to generate the molecule description in English that fits the molecule SELFIES.
F.3
Cross-modal Generation
F33.1 Molecule Captioning
Instruction: Now complete the following example-
Datasets
<eom> Input: <bom> < Output: Output: (Text Description)
We use ChEBI-20 dataset created by Tex12mol (Ed- wards et al., 2021), which consists of 33, 010 molecule-text pairs and 20 means each text descrip- tion has more than 20 words. The dataset is split into 8 1: 1 for train, validation, and test.
F.3.2 Text-based molecule generation
This is the reverse task of molecule captioning. The input is the text description of the desired molecule and the output the corresponding molecule SELF- IES. The datasets and compared baselines are the same with molecule captioning in Section F.3.1 so will only provide the prompts here.
Baselines
(I) RNN (Medsker and Jain, 2001) with 4-layer bidirectional encoder is trained from scratch on ChEBI-20 dataset.
(2) Transformer (Vaswani et al., 2017) containing 6 encoder and decoder layers is trained from scratch on ChEBI-20 dataset.
Prompts
Task Definition: Definition: You are given a molecule description in English. Your job is generate the molecule SELFIES that fits the de- scription.
(3) T5 (Raffel et al., 2020) is directly fine-tume tuned on ChEBI-20 dataset from public checkpoints with three different model sizes: small, base and large. Note that no molecule domain knowledge is introduced in the original T5 pre-training.
Instruction: Now complete the following example-
Input: (Text Description) Output: Output: (SoFF>(EELFIEE)
(4) MolT5 (Edwards al., 2022) is jointly trained on molecule SMILES from ZINC-15 dataset (Ster- ling and Irwin, 2015) and general text from C4 dataset (Raffel et al., 2020) so that MolT5 has prior knowledge about these two domains. It also con- tains three different sizes: small, base and large. Then they are further fine-tuned on ChEB1-20 dataset.
G Case Study
In this section, we show several example outputs from different models in molecule captioning and text-based molecule generation tasks. Figure 5 shows the cases for the molecule captioning task. In example (I), the description of BioT5 matches the ground truth best, successfully localizing the position of the substituent group and "member of pyridines and an aryl thiol". In example (2), MolT5 mistakenly describes that the molecule contained boron, while BioT5's description is more accurate. In example (3), while MolT5 generates repetitive output, BioT5 and T5 generate semantically coher- ent output, and BioT5's output matches better with ground truth. For a complex molecule in example (4), the output of BioT5 is more holistic and accu- rate. Notably, only BioT5 deseribes this molecule
(5) GPT-3.5-turbo (Li et al., 2023) used by di- rectly call OpenAI API without further fine-tuning. The input includes five parts as Li et al. (2023): role identification, task description, examples, out- put instruction, and user input prompt. The exam- ples are retrieved by Morgan Fingerprint (Butina, 1999) similarity for molecule captioning task and
'https://github httt::///ggttubb.mm/ggggllee reeeeech
text to- to-texx-ttrassfeeee trwasferrttraattee former httmmrrrbloomain// clleased_checkpoitts md#t511

---

an inhibitor of SARS coronavirus main pro- teinase, which may come from our integration with protein knowledge. Figure shows the cases for the text-based molecule generation task. From the cases, we have several findings: (i) BioT5 more likely to generate molecules that exactly match the ground truth. (ii) By using SELFIES, BioT5 will not generate invalid molecules, especially for com- plex and longer molecules shown in examples (3) and (4). (ii) Some molecules are actually short proteins. Example (3) shows a molecule that is 33-membered polypeptide, which consists of 33 amino acid residues joined in sequence. Therefore, the boundary between proteins and molecules may not always be distinct, and leveraging information from both can provide reciprocal benefits.

---

Input
MofT5
Ours
Ground Truth
The molenule merrrtrraatttammi rouuling from the forma condeneette Usocyane asdd benuert o, iis member of thhccrtomiccaeee and munethopaateem credeer
sene-sustttiig sistituted pontior nuttery
The mumusaolle syridine subatiiused anC suifanytarun thane note as tlureeranna quencter and on allrgonn. 1 monierr of pyldines andonoryfttia.
posiiton ha metaboldss hiis thiok ond momber bontenos. In derivis from u Irytnde of thas..
hasa rote comcaioninnita inniittto and allegen is memroc of preit ree and angithiil
Thomolecuie a metel holide, a mutal cation and timium ondby, huso
The molecule metol metal munutal iam ummn unnalannnt inonganicarion. compugaano
The melacuia is mettl bramiide nainwtt couumeioon inorganic rose us
Themoleouie lnhlimmott countarion beamn simsleri
csmotye
Thomoloouie lea quotemary ammoniunn well thas anmmuriait
The molacala is the iodowoudeoeeat surbo-
The molacuia is suareemneryy um crmmnnmmaaaiie thatis the monotosicaeett
H2C
H2C
J
groucs roplaced by motrol
the hydingens arehe
iedi iodio voco usto todko sell--odd--
tnde--tdda-iict iet---at-edddeeeeeeeeeeeeee iodo
vodo lodo
chyyaiinchyaammmmnn quaternars ammonium saltt on whyphimsdhyammmmennn
ammentum sairwhest baslounh comptiast
CH
lodo dorvetthv todo iodo odto
etty mttyrrmmttytmmmmn cation and an indide wnion itts guaeemeryy ommorium satiandioo oddide solt.
quotemary
well containoe bosumach secumnchhlmmmoriin
The motecasi swaondwrry co corboxain de resutting noom the furmal concamattion the cartuuy group or intonoooe carboglle primary proup
Thomelooute socondary corbosa mide rosulting from the fommel condersttional the cartacy group of i-linalle carbowyli acld methmmetimm peipimary aminngeous cyelshool
Thumolansls is becapaptizle compused vapponte phenyllllniie beo proline Laapartic ptamylalariie aoppoolin.
centopoptide curstatinget gor cradeeoyymmnnyyy phammaaaninn L----
ioprifiyttontssiin melouse joined in secuieoos it hosa min sa metabolite it polapeptioo, member naprttullnns ands po stapoptiie.
slanine
compynalidi-- Papppmaatt----llppppoeeee peppplr-ollpponnnnngg
(18))
9.4 [SARS ca coronavius visusmiaiin pioeelnase) immiitee and
0.72 22mhffi. Ithuu
replicatioon culture (BC50
andirnibns SARS
EC
SAlts wonus mainprotennne
cell cuban (ECS0 n. mumt) twe nol as ne F..3..5
andiveicata EARtS-Ca Coll nast catton
(SARS
ontoorcnaahologgen a
coronae ru main pioteneaet intibttr andien antizarcnsst
aecondary centocemete membrr
somulaan---mme digopaptidee
ageent indoderastuccmmd pyroolsie
Figure 5: Molecule captioning cases.
Input
MolTs
Ours
Ground Truth
nusieotide
bsaniing wadiauure brame subetituene on postton on tegethwiing... An acthuan nydlle AHP dopendent prosein kinsoo but stant copaadationnly gydle prosphodiestrroee
OH
aderyliibouudion db dialviss from S-opcciic A0M
moleeaale sanaminngpptiiiit i mettyloyyliidin 2.4 diaminin in omicctine of hydnngens the mothyl grop heen reptared hya P-cyclopeeryt- dimethony chromen memberr amtrappetmiatme member eyetecro.anne
Invelid
My
andio
HD
The molecaaa 37--memmeend pikepaptice consiating of Oiy. City Sec App, The Aun, Lou, Als, fan, byy Law,
Invalid
gluzagonelcn pepridedd mosetor agoniatused forthe oraament shon bowel syndicnee. hos vitte un villccagen-ite expticus2 monetor agnnrie, metabollin, untionddant protective ugant.
The molcoll ise CyS andon organic peactlcsst sack hasa rolo as fuorochromo, containss UAC1 Scs][1r).
Invalid
Invalid
Figure 6: Text-based molecule generation cases.


Reviews:
[{"review_id": "ogJzdufqct", "reviewer": "Reviewer_BcEQ", "summary": "", "questions": "A: Do you use the official data splits of train, dev, test? \n\nB: Do you use the original reported results of the baselines or the models are tested here, for in some cases the results are different, like MolTrans.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the valuable comments. Below we would like to provide some explanations to address your concerns.\n\n> **It is not clear where the improvement of results come from, from SELFIES, FASTA, or general text representations?**\n\nOur results primarily stem from several aspects, including not only the SELFIES, FASTA, and general text representations but also their implicit and explicit combinations. By applying the standard T5 [1] pre-training to molecule SELFIES, protein FASTA, and general text individually, we ensure the model's proficiency in each modality. Pre-training on wrapped biological text allows the model to implicitly learn the properties of bio-entities and capture potential connections between different bio-entities. Performing bi-directional translation pre-training tasks on structured molecule-text data and protein-text data makes BioT5 explicitly capture the connection between bio-entities and text. The tailored tokenization and dictionary for each modality ensure their integrity, particularly for molecule SELFIES and protein FASTA.\n\nTo further demonstrate the effectiveness of each part of our BioT5, we also conduct an additional ablation study. Specifically, we focus on the following 5 scenarios:\n\n- (a) Retaining only single-modal data, excluding wrapped text, molecule-text pairs, and protein-text pairs.\n- (b) Keeping single-modal data, molecule-text pairs, and protein-text pairs, excluding wrapped text.\n- (c) Keeping single-modal data and wrapped text, excluding molecule-text pairs and protein-text pairs.\n- (d) Shared vocabulary among three modalities: using the text tokenizer and vocabulary from original T5 [1] for molecule SELFIES tokens and protein amino acid tokens.\n- (e) Fine-tuning on downstream tasks from scratch.\n\nThen we fine-tune the above pre-trained models on molecule property prediction task with BACE and BBBP datasets and drug-target interaction (DTI) task with BioSNAP dataset. The results are shown in the following tables (\"—\" means removing the corresponding part).\n\n| Method/Dataset | BACE | BBBP |\n| --- | --- | --- |\n| BioT5 | 0.894 (0.003) | 0.777 (0.006) |\n| (a) —wrapped text —molecule text pairs —protein text pairs | 0.864 (0.008) | 0.740 (0.001) |\n| (b) —wrapped text | 0.887 (0.001) | 0.753 (0.009) |\n| (c) —molecule text pairs —protein text pairs | 0.876 (0.006) | 0.759 (0.013) |\n| (d) —separate chemical dictionary  | 0.862 (0.007) | 0.751 (0.007) |\n| (e) from scratch  | 0.836 （0.007） | 0.698 (0.001) |\n\nTable 1: Ablation study on molecule property prediction with BACE and BBBP dataset. The evaluation metric is AUROC.\n\n| Method | AUROC | AUPRC | Accuracy |\n| --- | --- | --- | --- |\n| BioT5 | 0.937 (0.001) | 0.937 (0.004) | 0.874 (0.004) |\n| (a) —wrapped text —molecule text pairs —protein text pairs | 0.920 (0.002) | 0.919 (0.003) | 0.855 (0.002) |\n| (b) —wrapped text | 0.928 (0.001) | 0.927 (0.002) | 0.864 (0.002) |\n| (c) —molecule text pairs —protein text pairs | 0.930 (0.002) | 0.929 (0.003) | 0.865 (0.005) |\n| (d) —separate chemical dictionary | 0.902 (0.009) | 0.908 (0.008) | 0.827 (0.015) |\n| (e) from scratch | 0.845 (0.004) | 0.862 (0.002) | 0.752 (0.018) |\n\nTable 2: Ablation study on drug-target interaction prediction with BioSNAP dataset.\n\nFrom the tables, we have the following observations.\n\n1. **Importance of multimodal data**: The complete BioT5 model, which presumably encompasses all data modalities, outperforms all the ablated models. Scenario (a) demonstrates the significant drop in performance when removing wrapped text, molecule-text pairs, and protein-text pairs, thereby underscoring the value of these components in contributing to the model's performance.\n2. **Role of wrapped text and molecule/protein-text pairs**: In scenario (b), when only wrapped text is excluded, there is a decline in performance relative to the full BioT5 model but is not as pronounced as when all the modalities are removed (as in scenario (a)). The same conclusion is also observed in scenario (c). This indicates that both wrapped text and molecule/protein-text pairs are important in capturing the properties and relationships of molecules and proteins\n3. **Tokenization strategy**: Scenario (d) sheds light on the importance of the separate tokenization and dictionary for molecule SELFIES tokens, protein amino acid tokens, and text tokens. Utilizing the text tokenizer and vocabulary from the original T5 for bio-sequences yields lower performance, suggesting that our domain-specific tokenization strategy is beneficial for biological tasks.\n4. **Value of pre-training**: The most drastic performance reduction is observed in scenario (e), where models are fine-tuned from scratch. This emphasizes the immense value of pre-training, even in domain-specific tasks. \n\nWe hope the above explanation and ablation study can address your question and will add these in the revised version.\n\n> **I understand space is limited, but it would be useful to perform error analysis.**\n> \n\nWe acknowledge the importance of error analysis. Here we conduct an analysis of drug-target interaction task with Human dataset. Following the DrugBAN [5], we fine-tune BioT5 on Human dataset with 5 random seeds and compare our results with the baseline method DrugBAN (the results of its five random runs are derived in its supplementary material).\n\n| Methods/Random Experiments | 1 | 2 | 3 | 4 | 5 | Mean±Std | p-value of t-test between BioT5 and DrugBAN |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| AUROC (BioT5) | 0.9899 | 0.9872 | 0.9886 | 0.9890 | 0.9882 | 0.989±0.001 | 0.0007 |\n| AUROC (DrugBAN) | 0.9836 | 0.9801 | 0.9849 | 0.9832 | 0.9790 | 0.982±0.002 |  |\n| AUPRC (BioT5) | 0.9873 | 0.9849 | 0.9833 | 0.9873 | 0.9843 | 0.985±0.002 | 0.0143 |\n| AUPRC (DrugBAN) | 0.9796 | 0.9762 | 0.9842 | 0.9828 | 0.9780 | 0.980±0.003 |  |\n\nFrom the table, we can observe the following outcomes:\n\n1. For the AUROC, BioT5 achieved an AUROC score of 0.989±0.001, while DrugBAN attained an AUROC of 0.982±0.002. The t-test comparing the two results yielded a highly significant p-value of 0.0007.\n2. For the AUPRC, BioT5 demonstrated an AUPRC score of 0.985±0.002. In contrast, DrugBAN recorded an AUPRC of 0.980±0.003. A subsequent t-test between these scores revealed a p-value of 0.0143. \n\nBoth the AUROC and AUPRC results show statistically significant differences between BioT5 and DrugBAN, as indicated by their respective p-values being below the 0.05 threshold.\nBased on the observed results, BioT5 outperforms DrugBAN in both AUROC and AUPRC metrics, indicating superior classification performance and precision-recall trade-off for BioT5.\n\nWe will incorporate more error analyses in the revised version of our manuscript.\n\n> **More details are needed regarding how the baseline results are computed. Q-B: Do you use the original reported results of the baselines or the models are tested here, for in some cases the results are different, like MolTrans.**\n \n\nApologies for the oversight. Our results primarily derive from the original paper and use the same split method as the compared benchmark. Specifically,\n\n- For molecule property prediction experiments, we follow the commonly used scaffold split for MoleculeNet [2] benchmark, with results mainly sourced from MolXPT [3].\n- For protein property prediction and protein-protein interaction experiments, we adhere to the PEER [4] benchmark's split setting, with results also primarily from PEER.\n- For drug-target interaction experiments, our split method and results mainly follow DrugBAN [5]. For instance, the MolTrans [6] results are from the DrugBAN paper, with discrepancies arising from DrugBAN's reproduction of these baselines.\n- For molecule captioning and text-based molecule generation experiments, we follow the default split setting of the ChEBI-20 dataset (first proposed by Text2Mol [9]), with results primarily from MolT5 [7], MolXPT [3], and MolReGPT [8].\n\nWe will provide a detailed description in the revised version of our manuscript.\n\n> **Q-A: Do you use the official data splits of train, dev, test?**\n> \n\nYes, we use the same official split method as the compared benchmark.\n\n- For molecule property prediction experiments, we use the official DeepChem python package from [2] to get data splits.\n- For protein property prediction and protein-protein interaction experiments, we adhere to the PEER [4] benchmark's split setting as described in Table 1 of [4].\n- For drug-target interaction experiments, we follow DrugBAN [5] as described in Section “Evaluation strategies and metrics” of [5].\n- For molecule captioning and text-based molecule generation experiments, we use the default split setting of the CheBI-20 dataset. This dataset was first proposed by Text2Mol [9] and the default splitting was also used in MolT5 [7], MolXPT [3], etc.\n\nWe will provide a detailed description in the revised version of our manuscript.\n\n> **The dataset of molecule captioning (ChEBI-20) needs to be mentioned in the main text of article**\n\nThank you for pointing out the oversight regarding the dataset of molecule captioning (ChEBI-20). Due to space limitations, we briefly mentioned it at the beginning of Section 4.3. We will ensure to provide a more detailed description in the revised version of our manuscript.\n\n********************References********************\n\n[1] Raffel, Colin, et al. \"Exploring the limits of transfer learning with a unified text-to-text transformer.\" *The Journal of Machine Learning Research* 21.1 (2020): 5485-5551.\n\n[2] Wu, Zhenqin, et al. \"MoleculeNet: a benchmark for molecular machine learning.\" *Chemical science* 9.2 (2018): 513-530.\n\n[3] Liu, Zequn, et al. \"MolXPT: Wrapping Molecules with Text for Generative Pre-training.\" *arXiv preprint arXiv:2305.10688* (2023).\n\n[4] Xu, Minghao, et al. \"Peer: a comprehensive and multi-task benchmark for protein sequence understanding.\" *Advances in Neural Information Processing Systems* 35 (2022): 35156-35173.\n\n[5] Bai, Peizhen, et al. \"Interpretable bilinear attention network with domain adaptation improves drug–target prediction.\" *Nature Machine Intelligence* 5.2 (2023): 126-136.\n\n[6] Huang, Kexin, et al. \"MolTrans: molecular interaction transformer for drug–target interaction prediction.\" *Bioinformatics* 37.6 (2021): 830-836.\n\n[7] Edwards, Carl, et al. \"Translation between molecules and natural language.\" *arXiv preprint arXiv:2204.11817* (2022).\n\n[8] Li, Jiatong, et al. \"Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective.\" *arXiv preprint arXiv:2306.06615* (2023).\n\n[9] Edwards, Carl, ChengXiang Zhai, and Heng Ji. \"Text2mol: Cross-modal molecule retrieval with natural language queries.\" *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a method to represent multiple modalities.\n\n- The paper performs an extensive comparison with some the available methods on different datasets for various tasks.", "reasons_to_reject": "- It is not clear where the improvement of results come from, from SELFIES, FASTA, or general text representations? \n\n- I understand space is limited, but it would be useful to perform error analysis.\n\n- More details are needed regarding how the baseline results are computed.", "questions_to_authors": "A: Do you use the official data splits of train, dev, test? \n\nB: Do you use the original reported results of the baselines or the models are tested here, for in some cases the results are different, like MolTrans.", "paper_topic_and_main_contributions": "The paper presents a method to represent molecules, proteins and their associations from the text. The model is compared with various baselines on different datasets for various tasks and achieves state-of-the-art performance in some of them.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The dataset of molecule captioning (ChEBI-20) needs to be mentioned in the main text of article."}, "extra_text": {}}, {"review_id": "slqwlRpXnM", "reviewer": "Reviewer_fMEa", "summary": "", "questions": "1. What is the performance of the  ablation studies on multi-task pretraining?\n2. What  is the  baseline with shared vocabularies and joint training?\n3. The authors should provide  learning curves.\n4. Is there any advantage of using \"SELFIES\" over \"SMILES\" representation of chemical in this work?\n5. For MoleculeNet sets, the regression task is not considered. Why?\n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the positive feedback and constructive suggestions. Below we give detailed responses to your comments.\n\n> **The method proposed by the authors is interesting, but none of the parts used in the workflow is novel.**\n> \n\nThanks for raising this concern. We agree that some components are borrowed from existing ways. However, we would like to emphasize the following points:\n\n1. **Unique combination and adaptation:** While individual components might have been explored in other contexts, the unique combination and adaptation of these components in the realm of bioinformatics is novel. Our BioT5, is not just a mere assembly of existing parts but a synergistic integration that achieves superior performance. Specifically, we have integrated both structured and unstructured data to enhance the interaction among different modalities. The large-scale unstructured wrapped text data from biological literature provides a wealth of knowledge about the properties of bio-entities and their interactions. Meanwhile, the structured sequence-text data derived from databases offers a comprehensive understanding of bio-entities. This integration, to the best of our knowledge, has not been previously explored in this manner, offering a fresh perspective on the problem.\n2. **Separate tokenization of bio-sequences:** We have highlighted the issues other methods face when tokenizing bio-sequences (in paper Figure 3). Our additional ablation study (see below) underscores the importance of using specialized and separate tokenization and dictionaries for bio-sequences, providing valuable insights for future work.\n3. **Robust molecular representation:** By adopting a 100% robust molecular representation, SELFIES, we ensure that the generated molecules are valid. This has been validated in downstream text-based molecule generation tasks.\n\nWe hope these details can help clarify the innovations and insights of our work. Thank you for your feedback, and we will better illustrate these points in the revised version.\n\n> **Q1: What is the performance of the ablation studies on multi-task pre-training? Q2: What is the baseline with shared vocabularies and joint training?**\n \nWe apologize for the oversight. Following your suggestion, we conduct the ablation study with the following 5 scenarios:\n\n- (a) Retaining only single-modal data, excluding wrapped text, molecule-text pairs, and protein-text pairs.\n- (b) Keeping single-modal data, molecule-text pairs, and protein-text pairs, excluding wrapped text.\n- (c) Keeping single-modal data and wrapped text, excluding molecule-text pairs and protein-text pairs.\n- (d) Shared vocabulary among three modalities: using the text tokenizer and vocabulary from original T5 [1] for molecule SELFIES tokens and protein amino acid tokens.\n- (e) Fine-tuning on downstream tasks from scratch.\n\nThen we fine-tune the above pre-trained models on molecule property prediction task with BACE and BBBP datasets and drug-target interaction (DTI) task with BioSNAP dataset. The results are shown in the following tables (\"—\" means removing the corresponding part).\n\n| Method/Dataset | BACE | BBBP |\n| --- | --- | --- |\n| BioT5 | 0.894 (0.003) | 0.777 (0.006) |\n| (a) —wrapped text —molecule text pairs —protein text pairs | 0.864 (0.008) | 0.740 (0.001) |\n| (b) —wrapped text | 0.887 (0.001) | 0.753 (0.009) |\n| (c) —molecule text pairs —protein text pairs | 0.876 (0.006) | 0.759 (0.013) |\n| (d) —separate chemical dictionary  | 0.862 (0.007) | 0.751 (0.007) |\n| (e) from scratch  | 0.836 （0.007） | 0.698 (0.001) |\n\nTable 1: Ablation study on molecule property prediction with BACE and BBBP dataset. The evaluation metric is AUROC.\n\n| Method | AUROC | AUPRC | Accuracy |\n| --- | --- | --- | --- |\n| BioT5 | 0.937 (0.001) | 0.937 (0.004) | 0.874 (0.004) |\n| (a) —wrapped text —molecule text pairs —protein text pairs | 0.920 (0.002) | 0.919 (0.003) | 0.855 (0.002) |\n| (b) —wrapped text | 0.928 (0.001) | 0.927 (0.002) | 0.864 (0.002) |\n| (c) —molecule text pairs —protein text pairs | 0.930 (0.002) | 0.929 (0.003) | 0.865 (0.005) |\n| (d) —separate chemical dictionary | 0.902 (0.009) | 0.908 (0.008) | 0.827 (0.015) |\n| (e) from scratch | 0.845 (0.004) | 0.862 (0.002) | 0.752 (0.018) |\n\nTable 2: Ablation study on drug-target interaction prediction with BioSNAP dataset.\n\nFrom the tables, we have the following observations.\n\n1. **Importance of multimodal data**: The complete BioT5 model, which presumably encompasses all data modalities, outperforms all the ablated models. Scenario (a) demonstrates the significant drop in performance when removing wrapped text, molecule-text pairs, and protein-text pairs, thereby underscoring the value of these components in contributing to the model's performance.\n2. **Role of wrapped text and molecule/protein-text pairs**: In scenario (b), when only wrapped text is excluded, there is a decline in performance relative to the full BioT5 model but is not as pronounced as when all the modalities are removed (as in scenario (a)). The same conclusion is also observed in scenario (c). This indicates that both wrapped text and molecule/protein-text pairs are important in capturing the properties and relationships of molecules and proteins\n3. **Tokenization strategy**: Scenario (d) sheds light on the importance of the separate tokenization and dictionary for molecule SELFIES tokens, protein amino acid tokens, and text tokens. Utilizing the text tokenizer and vocabulary from the original T5 for bio-sequences yields lower performance, suggesting that our domain-specific tokenization strategy is beneficial for biological tasks.\n4. **Value of pre-training**: The most drastic performance reduction is observed in scenario (e), where models are fine-tuned from scratch. This emphasizes the immense value of pre-training, even in domain-specific tasks. \n\nThanks for your suggestions and we will include the ablation study in the revised version.\n\n> **Q3: The authors should provide learning curves.**\n> \n\nWe appreciate your suggestion regarding the inclusion of learning curves. Due to the constraints of the rebuttal, we provide the learning curves for pre-training in tabular form below.\n\n| Step | Loss |\n| --- | --- |\n| 25k | 1.150 |\n| 50k | 0.828 |\n| 75k | 0.743 |\n| 100k | 0.732 |\n| 125k | 0.693 |\n| 150k | 0.694 |\n| 175k | 0.657 |\n| 200k | 0.676 |\n| 225k | 0.678 |\n| 250k | 0.652 |\n| 275k | 0.615 |\n| 300k | 0.579 |\n| 325k | 0.552 |\n| 350k | 0.541 |\n\nTable 3: Pre-training loss, including denoising loss and translation loss, which are jointly optimized. The loss is averaged every 25k steps.\n\nWe can observe from the table that the overall pre-training loss exhibits a progressive and consistent decrease, indicating effective convergence of the model.\n\nWe recognize the importance of visualizing the learning process and will include detailed learning curves for both pre-training and fine-tuning in the revised version of our manuscript. \n\n> **Q4: Is there any advantage of using \"SELFIES\" over \"SMILES\" representation of chemical in this work?**\n> \n\nBoth SELFIES and SMILES are sequence representations of molecules. Their expressive power for molecules is comparable, as evidenced by experiments in ChemBERTa [2] on the molecule property prediction task. However, SELFIES has a distinct advantage in molecular generation tasks. Specifically, every valid SELFIES corresponds to a valid molecule. As shown in Table 6 of the paper, in the task of text-based molecule generation, by employing SELFIES, our model BioT5 achieved a 100% validity rate. In contrast, all baseline methods that utilized SMILES for molecular modeling often resulted in the generation of invalid molecules.\n\n> **Q5: For MoleculeNet sets, the regression task is not considered. Why?**\n> \n\nRegression tasks are inherently more challenging than classification tasks as they require predicting specific numerical values. While classification labels can be directly translated into natural language, regression tasks often necessitate specialized designs for encoding and tokenizing numerical values [3]. Our primary focus in this work is to validate the advantages of our multi-task pre-training framework, so we initially focused on classification downstream tasks that don't require additional design. We plan to address regression tasks in future work.\n\n> **Missing drug-drug interactions comparison**\n> \n\nThank you for pointing out the missing comparison on drug-drug interactions (DDI). In the current version of our paper, we do not focus on the DDI downstream task based on the observation that the results for this task are already quite promising (with an AUROC of 0.9994 [4]). We hypothesize that such impressive results might not necessitate additional pre-training for enhancement. However, we acknowledge the importance of discussing and incorporating DDI-related work and results and will add them in the revised version.\n\n********************References********************\n\n[1] Raffel, Colin, et al. \"Exploring the limits of transfer learning with a unified text-to-text transformer.\" *The Journal of Machine Learning Research* 21.1 (2020): 5485-5551.\n\n[2] Chithrananda, Seyone, Gabriel Grand, and Bharath Ramsundar. \"ChemBERTa: large-scale self-supervised pretraining for molecular property prediction.\" *arXiv preprint arXiv:2010.09885* (2020).\n\n[3] Born, Jannis, and Matteo Manica. \"Regression Transformer enables concurrent sequence regression and generation for molecular language modelling.\" *Nature Machine Intelligence* 5.4 (2023): 432-444.\n\n[4] Chen, Yujie, et al. \"MUFFIN: multi-scale feature fusion for drug–drug interaction prediction.\" *Bioinformatics* 37.17 (2021): 2651-2658.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The introduction and related works sections are well-written, explaining the overall functional parts of the model and gaps in the existing literature.\n2. The proposed method shows competitive performances compared to the existing baselines.\n3. Multi-task pre-training to model the connections between the three modalities.\n4. An appropriate tokenization method is used to represent the chemicals and protein sequences that maintain each modality's integrity.\n5. Prompt-based finetuning strategy to unify all the downstream tasks.\n", "reasons_to_reject": "1. The method proposed by the authors is interesting, but none of the parts used in the workflow is novel. ", "questions_to_authors": "1. What is the performance of the  ablation studies on multi-task pretraining?\n2. What  is the  baseline with shared vocabularies and joint training?\n3. The authors should provide  learning curves.\n4. Is there any advantage of using \"SELFIES\" over \"SMILES\" representation of chemical in this work?\n5. For MoleculeNet sets, the regression task is not considered. Why?\n\n", "paper_topic_and_main_contributions": "\nThe paper proposes a transformer-based language model named BioT5, which integrates information about chemical and biological entities with natural language associations. The T5 model uses SELFIES for representing molecules, FASTA for proteins and text tokens for unstructured literature. The model is pretrained in a multi-task way on individual text modalities, text pairs and wrapped text. The authors claim that the language model outperforms baselines across a range of single-instance prediction, multi-instance prediction and cross-modal generation tasks when fine-tuned using a prompt-based fine tuning strategy.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "For Drug-drug interactions comparison. https://academic.oup.com/bioinformatics/article/37/17/2651/6171181"}, "extra_text": {}}, {"review_id": "aADlPLCLZd", "reviewer": "Reviewer_AqiU", "summary": "", "questions": "A. What is the justification for choosing T5 as opposed to other architectures?\nB. Other than \"reduce the gap between pre-training and fine-tuning\", is there a justification for adopting prompt-based fine-tuning? Would more task oriented fine-tuning approaches (e.g. standard classification setting)  be more well suited for some of the tasks?\nC. Line 102: What does \"text data from the biological domain\" refers to? Is this referring to the scientific literature?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the positive feedback and valuable suggestions. Below we would like to give responses to your comments.\n\n> **The use of prompt-based fine tuning, which is only lightly justified.** \n**Q-B: Other than \"reduce the gap between pre-training and fine-tuning\", is there a justification for adopting prompt-based fine-tuning? Would more task oriented fine-tuning approaches (e.g. standard classification setting) be more well suited for some of the tasks?**\n> \n\nThank you for this insightful question. Beyond the aim to reduce the gap between pre-training and fine-tuning, we have several reasons for adopting prompt-based fine-tuning:\n\n1. **Simpleness and efficiency**: Compared to task-oriented fine-tuning approaches like standard classification settings, prompt-based fine-tuning doesn't require the introduction of additional layers, heads, or parameters. This makes the process more efficient and streamlined.\n2. **Flexibility and extensibility**: Current GPT-based model training predominantly uses instruction tuning [3, 4, 5]. This approach is more extensible and can easily support various types of prompts, making it more user-friendly.\n\n> **The lack of an ablation study, which can be understandable given the already extensive work and paper.**\n> \n\nThanks for this insightful comment. Following your suggestion, we conduct the ablation study with the following 5 scenarios:\n\n- (a) Retaining only single-modal data, excluding wrapped text, molecule-text pairs, and protein-text pairs.\n- (b) Keeping single-modal data, molecule-text pairs, and protein-text pairs, excluding wrapped text.\n- (c) Keeping single-modal data and wrapped text, excluding molecule-text pairs and protein-text pairs.\n- (d) Shared vocabulary among three modalities: using the text tokenizer and vocabulary from original T5 [1] for molecule SELFIES tokens and protein amino acid tokens.\n- (e) Fine-tuning on downstream tasks from scratch.\n\nThen we fine-tune the above pre-trained models on molecule property prediction task with BACE and BBBP datasets and drug-target interaction (DTI) task with BioSNAP dataset. The results are shown in the following tables (\"—\" means removing the corresponding part).\n\n| Method/Dataset | BACE | BBBP |\n| --- | --- | --- |\n| BioT5 | 0.894 (0.003) | 0.777 (0.006) |\n| (a) —wrapped text —molecule text pairs —protein text pairs | 0.864 (0.008) | 0.740 (0.001) |\n| (b) —wrapped text | 0.887 (0.001) | 0.753 (0.009) |\n| (c) —molecule text pairs —protein text pairs | 0.876 (0.006) | 0.759 (0.013) |\n| (d) —separate chemical dictionary  | 0.862 (0.007) | 0.751 (0.007) |\n| (e) from scratch  | 0.836 （0.007） | 0.698 (0.001) |\n\nTable 1: Ablation study on molecule property prediction with BACE and BBBP dataset. The evaluation metric is AUROC.\n\n| Method | AUROC | AUPRC | Accuracy |\n| --- | --- | --- | --- |\n| BioT5 | 0.937 (0.001) | 0.937 (0.004) | 0.874 (0.004) |\n| (a) —wrapped text —molecule text pairs —protein text pairs | 0.920 (0.002) | 0.919 (0.003) | 0.855 (0.002) |\n| (b) —wrapped text | 0.928 (0.001) | 0.927 (0.002) | 0.864 (0.002) |\n| (c) —molecule text pairs —protein text pairs | 0.930 (0.002) | 0.929 (0.003) | 0.865 (0.005) |\n| (d) —separate chemical dictionary | 0.902 (0.009) | 0.908 (0.008) | 0.827 (0.015) |\n| (e) from scratch | 0.845 (0.004) | 0.862 (0.002) | 0.752 (0.018) |\n\nTable 2: Ablation study on drug-target interaction prediction with BioSNAP dataset.\n\nFrom the tables, we have the following observations.\n\n1. **Importance of multimodal data**: The complete BioT5 model, which presumably encompasses all data modalities, outperforms all the ablated models. Scenario (a) demonstrates the significant drop in performance when removing wrapped text, molecule-text pairs, and protein-text pairs, thereby underscoring the value of these components in contributing to the model's performance.\n2. **Role of wrapped text and molecule/protein-text pairs**: In scenario (b), when only wrapped text is excluded, there is a decline in performance relative to the full BioT5 model but is not as pronounced as when all the modalities are removed (as in scenario (a)). The same conclusion is also observed in scenario (c). This indicates that both wrapped text and molecule/protein-text pairs are important in capturing the properties and relationships of molecules and proteins\n3. **Tokenization strategy**: Scenario (d) sheds light on the importance of the separate tokenization and dictionary for molecule SELFIES tokens, protein amino acid tokens, and text tokens. Utilizing the text tokenizer and vocabulary from the original T5 for bio-sequences yields lower performance, suggesting that our domain-specific tokenization strategy is beneficial for biological tasks.\n4. **Value of pre-training**: The most drastic performance reduction is observed in scenario (e), where models are fine-tuned from scratch. This emphasizes the immense value of pre-training, even in domain-specific tasks. \n\nWe appreciate your suggestion and will include this ablation study in the revised version.\n\n> **Q-A: What is the justification for choosing T5 as opposed to other architectures?**\n> \n\nThank you for raising this question. The choice of the T5 architecture for our BioT5 model was based on several considerations:\n\n1. We don’t use encoder-only architecture since our downstream tasks include generative tasks such as molecule captioning and text-based molecule generation. This necessitated the use of an architecture that wasn't encoder-only.\n2. Given that our BioT5 model (\\~250M parameters) is not yet at the super scale of current large language models [6, 7], the bidirectional encoding capabilities of the T5 encoder are potentially beneficial. There is a comparison between encoder-decoder and decoder-only architectures in T5 [1] paper's Section 3.2 that encoder-decoder seems to be a better one at that scale (\\~250M). We will explore the decoder-only model and scale up BioT5 in future work.\n\n> **Q-C: Line 102: What does \"text data from the biological domain\" refers to? Is this referring to the scientific literature?**\n> \n\nYes. The \"text data from the biological domain\" refers to the biological articles/literature from PubMed [2]. PubMed is a biological database consisting of much literature on biomedical topics and life sciences. We will add a more detailed description in the revised version.\n\n> **Typos**\n> \n\nWe appreciate your keen observation of the typos and will revise them in the later version of our manuscript.\n\n**References**\n\n[1] Raffel, Colin, et al. \"Exploring the limits of transfer learning with a unified text-to-text transformer.\" *The Journal of Machine Learning Research* 21.1 (2020): 5485-5551.\n\n[2] Canese, Kathi, and Sarah Weis. \"PubMed: the bibliographic database.\" *The NCBI handbook* 2.1 (2013).\n\n[3] Wei, Jason, et al. \"Finetuned language models are zero-shot learners.\" *arXiv preprint arXiv:2109.01652* (2021).\n\n[4] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" *Advances in Neural Information Processing Systems* 35 (2022): 27730-27744.\n\n[5] Peng, Baolin, et al. \"Instruction tuning with gpt-4.\" arXiv preprint arXiv:2304.03277 (2023).\n\n[6] OpenAI. \"GPT-4 Technical Report.\" arXiv preprint arXiv:2303.08774 (2023)\n\n[7] Touvron, Hugo, et al. \"Llama 2: Open foundation and fine-tuned chat models.\" arXiv preprint arXiv:2307.09288 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The methodology proposed achieves very positive results, which demonstrates the ability of the model to integrate knowledge from the different sources.\nThe use of separate tokenisation and embedding.\nThe work is clearly described and with good level of detail.", "reasons_to_reject": "The use of prompt-based fine tuning, which is only lightly justified.\nThe lack of an ablation study, which can be understandable given the already extensive work and paper.", "questions_to_authors": "A. What is the justification for choosing T5 as opposed to other architectures?\nB. Other than \"reduce the gap between pre-training and fine-tuning\", is there a justification for adopting prompt-based fine-tuning? Would more task oriented fine-tuning approaches (e.g. standard classification setting)  be more well suited for some of the tasks?\nC. Line 102: What does \"text data from the biological domain\" refers to? Is this referring to the scientific literature?\n", "paper_topic_and_main_contributions": "In this paper the authors propose BioT5, a T5 based model that integrates knowledge from chemical and protein textual representations, scientific literature, and from domain databases. The paper describes the model rationale, pre-training and fine-tuning settings and evaluates the model in several down-stream tasks. The paper is well organised and clear; previous works are presented in minimal detail; the proposed model and approach is described in sufficient detail. \nThe described pre-training framework results in very positive results, overcoming the compared works in most tasks, which demonstrates the ability of the model to integrate knowledge from the different sources.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\nline 264: sequneces\nline 310: nature\nline 549: those have\n"}, "extra_text": {}}, {"review_id": "jgVQCl4Nru", "reviewer": "Reviewer_tZxv", "summary": "", "questions": "A). Is the tokenisation ( separate vocabulary for general text. vs protein / chemical names) that is causing the boost in performance? or is it the set of  pretraining tasks ?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. We appreciate your time and effort and would like to address your concerns.\n\n> **Ablation study / analysis of what aspects are crucial to the performance gains would be helpful. Q-A: Is the tokenisation ( separate vocabulary for general text. vs protein / chemical names) that is causing the boost in performance? or is it the set of pretraining tasks ?**\n\nSorry for the lack of an ablation study, which is important to understand the effectiveness of each part of BioT5. Following your suggestion, we conduct the ablation study with the following 5 scenarios:\n\n- (a) Retaining only single-modal data, excluding wrapped text, molecule-text pairs, and protein-text pairs.\n- (b) Keeping single-modal data, molecule-text pairs, and protein-text pairs, excluding wrapped text.\n- (c) Keeping single-modal data and wrapped text, excluding molecule-text pairs and protein-text pairs.\n- (d) Shared vocabulary among three modalities: using the text tokenizer and vocabulary from original T5 [1] for molecule SELFIES tokens and protein amino acid tokens.\n- (e) Fine-tuning on downstream tasks from scratch.\n\nThen we fine-tune the above pre-trained models on molecule property prediction task with BACE and BBBP datasets and drug-target interaction (DTI) task with BioSNAP dataset. The results are shown in the following tables (\"—\" means removing the corresponding part).\n\n| Method/Dataset | BACE | BBBP |\n| --- | --- | --- |\n| BioT5 | 0.894 (0.003) | 0.777 (0.006) |\n| (a) —wrapped text —molecule text pairs —protein text pairs | 0.864 (0.008) | 0.740 (0.001) |\n| (b) —wrapped text | 0.887 (0.001) | 0.753 (0.009) |\n| (c) —molecule text pairs —protein text pairs | 0.876 (0.006) | 0.759 (0.013) |\n| (d) —separate chemical dictionary  | 0.862 (0.007) | 0.751 (0.007) |\n| (e) from scratch  | 0.836 （0.007） | 0.698 (0.001) |\n\nTable 1: Ablation study on molecule property prediction with BACE and BBBP dataset. The evaluation metric is AUROC.\n\n| Method | AUROC | AUPRC | Accuracy |\n| --- | --- | --- | --- |\n| BioT5 | 0.937 (0.001) | 0.937 (0.004) | 0.874 (0.004) |\n| (a) —wrapped text —molecule text pairs —protein text pairs | 0.920 (0.002) | 0.919 (0.003) | 0.855 (0.002) |\n| (b) —wrapped text | 0.928 (0.001) | 0.927 (0.002) | 0.864 (0.002) |\n| (c) —molecule text pairs —protein text pairs | 0.930 (0.002) | 0.929 (0.003) | 0.865 (0.005) |\n| (d) —separate chemical dictionary | 0.902 (0.009) | 0.908 (0.008) | 0.827 (0.015) |\n| (e) from scratch | 0.845 (0.004) | 0.862 (0.002) | 0.752 (0.018) |\n\nTable 2: Ablation study on drug-target interaction prediction with BioSNAP dataset.\n\nFrom the tables, we have the following observations.\n\n1. **Importance of multimodal data**: The complete BioT5 model, which presumably encompasses all data modalities, outperforms all the ablated models. Scenario (a) demonstrates the significant drop in performance when removing wrapped text, molecule-text pairs, and protein-text pairs, thereby underscoring the value of these components in contributing to the model's performance.\n2. **Role of wrapped text and molecule/protein-text pairs**: In scenario (b), when only wrapped text is excluded, there is a decline in performance relative to the full BioT5 model but is not as pronounced as when all the modalities are removed (as in scenario (a)). The same conclusion is also observed in scenario (c). This indicates that both wrapped text and molecule/protein-text pairs are important in capturing the properties and relationships of molecules and proteins\n3. **Tokenization strategy**: Scenario (d) sheds light on the importance of the separate tokenization and dictionary for molecule SELFIES tokens, protein amino acid tokens, and text tokens. Utilizing the text tokenizer and vocabulary from the original T5 for bio-sequences yields lower performance, suggesting that our domain-specific tokenization strategy is beneficial for biological tasks.\n4. **Value of pre-training**: The most drastic performance reduction is observed in scenario (e), where models are fine-tuned from scratch. This emphasizes the immense value of pre-training, even in domain-specific tasks. \n\nWe appreciate your suggestion and will include the ablation study in the revised version.\n\n> **Typos**\n> \n\nThank you for pointing out the potential typo and it should be “cross-modal”. We will revise it in the revised version of our manuscript.\n\n********************References********************\n\n[1] Raffel, Colin, et al. \"Exploring the limits of transfer learning with a unified text-to-text transformer.\" *The Journal of Machine Learning Research* 21.1 (2020): 5485-5551.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Finetuned BioT5 seems to achieve SOTA performance on most of the tasks and datasets listed in the paper.\n2. The experiments and results are well documented.", "reasons_to_reject": "1. Ablation study / analysis of what aspects are crucial to the performance gains would be helpful.", "questions_to_authors": "A). Is the tokenisation ( separate vocabulary for general text. vs protein / chemical names) that is causing the boost in performance? or is it the set of  pretraining tasks ?", "paper_topic_and_main_contributions": "The paper develops a new model \"BioT5\" based on T5 for tasks involving  molecule / proteins combined with text. Finetuned BioT5 seems to achieve SOTA performance on most of the tasks and datasets listed in the paper.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Is the title meant to \"cross-model\" or \"cross-modal\" ?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a method to represent multiple modalities, perform an extensive comparison with existing methods, and demonstrate competitive performance. However, this paper has limitations in a lack of clarity on the source of result improvements, the absence of error analysis, the need for more details on baseline results computation, and a perceived lack of novelty in the components used in the workflow.
Decision: Accept-Main