BioBridge: Bridging Biomedical Foundation Models via Knowledge Graphs
Authors: Zifeng Wang
Zichen Wang
Balasubramaniam Srinivasan
Vassilis N. Ioannidis
Huzefa Rangwala
RISHITA ANUBHAI
Venue: ICLR 2024 poster (2024)
Keywords: drug discovery
foundation model
multi-modal learning
knowledge graph
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We developed a parameter-efficient method to bridge multiple unimodal biomedical foundation models using a knowledge graph to enable multi-modal functionalities.

Abstract: Foundation models (FMs) learn from large volumes of unlabeled data to demonstrate superior performance across a wide range of tasks. However, FMs developed for biomedical domains have largely remained unimodal, i.e., independently trained and used for tasks on protein sequences alone, small molecule structures alone, or clinical data alone.
To overcome this limitation, we present BioBridge, a parameter-efficient learning framework, to bridge independently trained unimodal FMs to establish multimodal behavior. BioBridge achieves it by utilizing Knowledge Graphs (KG) to learn transformations between one unimodal FM and another without fine-tuning any underlying unimodal FMs.
Our results demonstrate that BioBridge can
beat the best baseline KG embedding methods (on average by ~ 76.3%) in cross-modal retrieval tasks. We also identify BioBridge demonstrates out-of-domain generalization ability by extrapolating to unseen modalities or relations. Additionally, we also show that BioBridge presents itself as a general-purpose retriever that can aid biomedical multimodal question answering as well as enhance the guided generation of novel drugs. Code is at https://github.com/RyanWangZf/BioBridge.

Published as a conference paper at ICLR 2024
BIOBRIDGE: BRIDGING BIOMEDICAL FOUNDATION MODELS VIA KNOWLEDGE GRAPHS
Zifeng Wang
Zichen Wang
University of Illinois Urbana- Champaign i.fengw28illinee edu
Amazon AWS AI
zichewan@amazon. com
Balasubramaniam Srinivasan
Vassilis N. Ioannidis
Amazon AWS AI srbalasu@ amazon. com
Amazon Search
ivasilei memiieee@mmaann...mm com
Huzefa Rangwala f
Rishita Anubhai
Amazon AWS AI
Amazon AWS AI
thuzefa@emazon.com com
ranubhai amazamnnno com
ABSTRACT
Foundation models (FMs) learn from large volumes of unlabeled data to demon- strate superior performance across wide range of tasks. However, FMs de- veloped for biomedical domains have largely remained unimodal, i.e., indepen- dently trained and used for tasks protein sequences alone, small molecule structures alone, or clinical data alone. To overcome this limitation, we present BicBRIDGE, a cerrammttr--fffcciee learning framework, to bridge independently trained unimodal FMs to establish multimodal behavior. Bi BioBRIDGE achieves it by utilizing Knowledge Graphs (KG) to learn transformations between one uni- modal FM and another without fine-tuning any underlying unimodal FMs. Our results demonstrate that BiOBRIDGE can beat the best baseline KG embedding methods (on average by 76.3%) in cross-modal retrieval tasks. We also identify BicBRIDGE demonstrates out-of-domain generalization ability by extrapolating unseen modalities relations. Additionally, presents itself as a general- purpose retriever that can aid biomedical multimodal question answering as well as enhance the guided generation of novel drugs."
also show that BioBR IDGE
INTRODUCTION
Foundation models (Bommasani et al., 2021) trained on large volumes of data can be leveraged and adapted for different domains. In biomedicine, FMs are trained to ingest text corpora (Gu et al., 2021) from scientific literature, protein data in sequences and 3D-structures (Jumper et al., 2021), molecule in graphs and SMILES strings (Fabian et al., 2020) and protein-interaction data in the form of relational graphs. These pre-trained biomedical FMs have achieved a significant gain in comparison to previous methods trained on smaller datasets (Qiu et al., 2023). Introducing multimodal data in training further boosts the performance of FMs, especially in few-shot/zero-shot prediction settings (Radford et al., 2021). In the biomedical domain, for drug-text (Edwards et al., 2022), protein-text (Liu et al., 2023), and prug-rrttein data (Huang et al., 2021: 1: loannidis et al., 2020), multimodal data was leveraged by the joint optimization unimodal encoders. However, this idea encounters key issues when scaling beyond two modalities:
Computational Cost. These approaches require many unimodal encoders with approximately sim- ilar sizes to avoid impeding each other. This setup can cause size explosion of the bundled models
This work completed while the author was intern at Amazon. Huzefa Rangwala is on LOA as Professor of Computer Science at George Mason University. This paper describes work performed at Amazon.
Code is https: / thub. com cowm//yawwagggg//////oooo/mo//// Bi o8ri iiridge.

---

Published as a conference paper at ICLR 2024
in magnitude of the number of modalities, thus rendering computational burden for performing joint optimization.
Data Scarcity. They require pairwise cross-modal datasets of similar size to ensure stable training.
The dataset quantity increases exponentially in combinatorial order of (k) where K represents the number of modalities, inevitably leading to data scarcity.
Contrastive Learning
ImageBind
Bridge (Ours]
drug
bological process
bislogiczal
depti
Interact with
enzyye
imiege
ponse
drig
target
video
aesocirted with
contraindicaton
onatomy
phenotype
discase
Ind cationion
Combination Patred
Contralized
modall Knowledge Graph
transformation
Model
Model
Model
Joint optimization via paired contrastive learning
align to central modality
frozen and leam the transformation
Figure 1: The conceptual comparison between our methods and previous methods. Left: multimodal contrastive learning, e-g., CLIP, learns from a combination of paired data, updating all unimodal encoders; Middle: ImageBind aligns all modalities with the central modality, with only the central model frozen; Right: B OBRIDGE learns the transformation across modalities from a multi-modal KG, keeping all FMs frozen.
Unlike ImageBind (Girdhar et al., 2023), which sets the image the central modality and aligns all the other encoders with the image via fine-tuning, the proposed Bi oBRIDGE keeps all the uni- modal FMs fixed and learns bridge these unimodal FMs. The conceptual demonstration is shown in Figure 1. Specifically, BioBRIDGE learns the cross-modality transformation from biomedical knowledge graphs (KGs). This approach modeled by leveraging the following insights:
Data Sufficiency. It is usually easier to collect unimodal data than to collect paired data from two modalities. For instance, close to 250M protein sequences (Rives et al., 2021) and 1.5B molecule structures (Sterling& & Irwin, 2015) are available to perform self-supervised pre-training, while only 441K protein-text pairs are one of the largest of biological multimodal datasets (Liu et al., 2023). As such, compared to the joint training of multimodal encoders, bridging independent models that were trained on unimodal data at scale enjoys the merits of data sufficiency and efficiency.
Structural Transformation. Multimodal biomedical KG contains the structure information repre- sented by the triplets of head and tail biomedical entities and their relationships. It covers rich set of modalities such as protein, molecule, and disease (Chandak et al., 2023), which enables compre- hensive biomedical analytics and ML. We align the embedding space of unimodal FMs through a cross-modal transformation model utilizing the rich structure in KG triplets.
In summary, BioBRIDGE aims to create a universal bridging mechanism capable of efficiently connecting the representations of any pairs of modalities. Technically, the bridge modules pervised by the rich structure information from knowledge graphs, while the unimodal FMs are kept frozen to advance the parameter and computation efficiency, Experiment shows that:
The bridged unimodal FMs are competitive in diverse cross-modal prediction tasks.
BioBRIDGE can extrapolate to nodes that are not present the training KG with comparable performance the supervised basclines.
BioBRIDGE generalizes to relationships that do not exist in the training KG, and the performance be enhanced with further training.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORK
Foundation models have sparked remarkable breakthroughs natural language processing (Brown et al., 2020) and computer vision (Kirillov et al., 2023). In biomedicine, FMs are trained with masked language modeling for text (Gu et al., 2021), proteins (Rives et al., 2021; Lin et al., 2023), drug molecules (Wang et al., 2019a), with generative modeling for text (Taylor et al., 2022), protein (Madani et al., 2023), molecule (Bagal et al., 2021). They are trained on unimodal data and used feature extractors in supervised prediction tasks such protein-protein interaction (Wang et al., 201 9b; Hallee & Gleghorn, 2023), protein function prediction (Gligorijević et al., 2021; Wang al., 2022a) and drug-target interaction (Sledzieski al., 2022; Kalakoti et al., 2022).
In the literature, multimodal biomedical FMs leverage contrastive learning on the pairs of image-text (Wang et al., 2022b), drug-proteds (Huang et al., 2021), drug-text (Edwards et al., 2022) and protein- text (Liu et al., 2023; Xu et al., 2023b;a). Nonetheless, extending these approaches beyond two modalities challenging due to the need for large volumes of pairs of multimodal datasets, recent effort (Girdhar et al., 2023) proposed to combine pre-trained models from diverse modalities by fine- tuning all unimodal encoders to be aligned with the image encoder. By contrast, BioBRIDGE does not involve the base FMs in training but trains the transformation bridge modules. In addition, the rich structure information in KG is leveraged to relax the central modality assumption and allows for more controlled cross coos-mtd transformation. We have also discussed the related literature in knowledge graph learning and multimodal learning in Appendix H.
METHOD
Training: Learning the Bridge Module
protei
drig
balogical
dinezon
assodiatddwith
enzyym
protein
protsin
Reaad
Bridge Mosule
Tr sampingg
(A
associated with
sorget
containdication
protein
target
drug
Gus drup
prottie
larget
L- yort e -Ea COA
Cructie
(Rfass ciee
Bridge
Indication
Module
Prediction: Cross Modality Similarity Search
Generation: Multi modal Prompting for Generation
prodeis
Crust protein
dlessse
associated with
Isputs: proteit
protein
Guai disease
assoclated with
di032s
protese
ssiodited with
Smillantyy
Seidge Medu
Bi idge
Rus
(Parad-Faas)
prompingg
Kor
UFF
Coail
Inputi: question
6
input
associated with carodid
utheroscleoodd?
Figure 2: The overall workflow of BiOBRIDGE: (1) top: we train bridge module that transforms
the head node embedding to the tail node space with contrastive learning. (2) bottom left: the
trained bridge module enables cross- cross-modal prediction through the similarity search. (3) bottom right: The bridge module enables multimodal prompting for retrievallaugmented generation.
The study aims to bridge embeddings across multiple modalities supervised by KG. The overall workflow of BioBRIDGE is illustrated in Figure
Knowledge Graph (KG), KG consists of nodes V and edges E as G = {V,E). 3. A node in the graph, namely v; (x", "x",c") where c the node modality and protein as node in the KG is with c "protein" and x" = the protein's sequence. An edge & that connects two nodes V; and v, is called a triple in the context of KG, as t; = (v,, v,, where rij indicates the relation between the head and tail nodes. KGs present the relations across modalities in a distributed way so BioBRIDGE does not need to set a central modality. Although traditional KG embedding (KGE) methods also enable mrosl prediction via link prediction, they do not extrapolate to nodes not in the training KG. Instead, Bio BioBRIDGE learns to transform the head-modality FM embeddings the tail's space. KG is not needed in the testing phase.
the node feature. For example,
3

---

Published as a conference paper at ICLR 2024
Bridge Modalities. With foundation models (FMs) pre-trained unimodal data, target to bridge the unimodal FMs to fulfill multimodal tasks without fine-tuning the FMs. For two nodes from different modalities, we encode V, and vj into embeddings: h, f(x") and h, g(x")) where f(-) and g(+) are two FMs hence h, and hy in different spaces. We aim to build the transformation that projects h, to the space of hj, as h, @(,, C ((rrr)) that considers the modality types of two samples and their relations. As result, the embedding h, is aligned with h; and can match relevant samples from the modality oj by embedding- baasid similarity search.
3.1 TRAINING & PREDICTION
Encoding & Transformation. In the training phase, BRRRIIGE, samples triple t.j that connects v, and V, from two modalities of and oj. respectively. The bridge module @ transforms the head node embedding h, to the space of modality ej to yield h, Specifically, the raw embedding h of sample v = {x, c} projected by the modality-specific projection head p as z = Po(h) E R to ensure all embeddings follow the same dimension. We also treat c ej Fij as categorical variables and generate their embeddings C₁ Cj. and Fij. respectively, which all are in dimension of The projected head node embedding is transformed by
h, = z, + (Zi, C22 Cj. Tiy);
(1)
where si 12 + Hd generates the relation-awar transformation embeddings additive to Zí.
Loss Function. We corrupt ti by replacing the tail node with others {vy' Jy cb, = c to build informative negative samples. Based on the encoded tail nodes and the transformed head node, as (h,, Zjz Bj..... ZjMah J. we perform contrastive learning with InfoNCE loss (Oord al., 2018) as
negative tails
exp(h; 2//T)
Lij = log exp(h; 2;/T) + Eg#, exp(h; 2//77)
(2)
where M is the number of the sampled negative tails; t is scalar temperature, and all embeddings are normalized by their fn-norm before passing to the loss function. This loss pushes h, close to the positive tail hy. As the base unimodal FMs are frozen, we only update the parameters of the transformation module vi and the modality-seecific projection head Pe-
Prediction. Though triples extracted from KGs are used in the training, BioBRIDGE does not refer to KG for its inference. For instance, for v; [x],e"] and the target modality C, we encode Vj, Vey E C by the base FM g(-) and project them into the normalized embeddings He [21, , ). Then, we encode v, with the base FM f(-) and transform it to the embed- ding space C,, yiclding the normalized h,, We can compare the similarity of h, with Hç efficiently through matrix inner product, as ÿ = HJ h4 E [-1,1]]c
3.2 IMPLEMENTATION
Dataset. We draw subset of PrimeKG (Chandak et al., 2023) to build the training knowledge graph. Specifically, we pick the six main node types from the graph: Protein, Molecule, Disease, Biologicul process (BP), Molecular function (MF), and Cellular component (CC) without the loss of generality. The statisties of the triples in the training KG are available Table The exact training set varies depending on the downstream evaluation datasets avoid data leakage in our experiments. We describe how we curate the training data based on PrimeKG in Appendix B.
Model. We categorize the six types of nodes into three modalities: protein sequence, SMILES strings, and natural language. Technically, we utilized ESM2-3B (Lin et al., 2023) to encode pro- teins, UniMol (Zhou et al., 2023a) encode drug molecules, and PubMedBERT (Gu et al., 2021) to encode diseases, biological processes, molecular functions, and cellular components. For a text node, concatenate its name and definition form the inputs for PubMedBERT.
4

---

Published as a conference paper at ICLR 2024
While there many potential options build the transformation, we deliberately choose vanilla six- six-layer transformer model for the bridge module v in Eq. to verify the plausibility of the method. In detail, stack 2,, Ci. Cj. and rij to build the input Z E I for the transformers. We draw the emhedding on the first position after the transformer as the output of dp to add to the input z. Please refer Appendix for the hyperparameter setups.
3.3 EXISTENCE OF BRIDGE MODULE AND LEARNABILITY
Assumptions 1 (Assumptions on the neural networks and Knowledge Graph). 1. Let K E Zt be the total number of modalities and let M1...., MR denote the K neural networks trained on the different modalities whose parameters are now frozen.
0, E kk....k), Sy S RR, Qt denotes the input space/ domain and the output space S& for every neural network is linear subspace of IRd. Specifically, we will assume the dimension of every subspace the same.
For every neural network Ms : S2g + Sy,
3. Let G (V,E) denote the knowledge graph where V is the set of nodes and & is the of edges. Each node v E V belongs to one of the modalities.
4. Every e E & of the knowledge graph which connects two nodes Vi, Vj E V has an associated relation type Fij- E RR
Theorem (Existence of Bridge Module). For any given pair of nodes vs, Us E of modality
types Kngs kuz E {1.... .. and with representations given by their appropriate neural networks Sv, E Sk., Se, E Ska,' which are connected by relation type Tig E R, there exists a bridge module B Rd (1,..., K} ,,,..., K} R IRd such that B (8u, Kups King Fij) + 8uz
Here, we present a theorem for the existence of unified bridge module that connects different modalities. Please refer to Appendix for the proof.
EXPERIMENT: CROSS-MODALII PREDICTION
In this section, we perform the experiments to test the prediction capabilities of BiOBRIDGE. Specifically, the prediction tasks can be categorized into:
In-domain entity and relation types. Both the types of the input entity and the input relation are present in the training knowledge graph, where we conducted two series of experiments: cross- modality retrieval tasks ($4.1) and semantic similarity inference ($4.2).
In-domain entity and out-of- relation types. We consider the case where the target re- lations are absent in the training graph, i.c., out domain. We conducted protein-protein interaction prediction for this case ($4.3).
Out-of- entity and in-domain relation types. We also conducted experiments for out- domain entities but in-domain relations: cross-species protein-phenotye matching ($4.4).
4.1 CROSS-MODALITY RETRIEVAL TASKS
Setup. BioBRIDGE is able perform cross-modality retrieval by matching the transformed em- bedding with the candidate samples in the target modality embedding space. To gauge the quality of the transformed embeddings, we compare our methods with suite of knowledge graph embedding (KGE) methods: TransE (Bordes et al., 2013), TransD (Ji et al., 2015), TransH (Wang et al., 2014), TransR (Lin et al., 2015), ComplEx (Trouillon al., 2016), DistMult (Yang et al., 2015), and RotatE (Sun et al., 2019), implemented with OpenKE (Han et al., 2018).
Metric. We refer to the KG link prediction literature use Hit@ K (K (1,3, 10)), and Mean reciprocal rank (MRR) to evaluate the prediction performance. MRR is the average reciprocal rank of all the positive test triples among the corrupted negative triples. Hit@ K measures the proportion of positive tail entities among the ranked top-K possible candidates. We caleulate these metrics the direction of tail entity prediction.
Dataset & Baseline. We split the PrimeKG triples to set up the cross- retrieval tasks. For each type of triple, we randomly sample 80%, 10%, and 10% for the train, validation, and test sets, respectively. Then, separate the test by triple types, with special focus the
5

---

Published as a conference paper at ICLR 2024
predictions for: (Protein, BP/MF/CC, Interacts with), (Drug. Disease, Indication). (Drug. Protein, Targer). (Protein, Disease, Associated with), {Drug, Disease, Contraindication) The statistics of the train/valid/test data used in this experiment available in Table 10.
Table Mean reciprocal rank (MRR) on the seven cross-modes prediction tasks. "Drug Disease" indicates the "contraindication" relation between drug and disease. The best are in bold.
Method
Protein BP
Proxein -+ MF
Protein +> CC
Drug +> Disease
Protsin ++ Drug
Disease
Prosein
Drug Disease
Transt
0.034
0.046
0,044
0.017
0,033
0.024
0.010
0.045
0.060
0.069
0,028
Trans
0.044
0.061
0.057
0.026
0.043
0.024
0.014
TransD
0.043
0.059
0.053
0,022
0,049
0.024
0.013
ComplEi
0.084
0.042
DistMalt
0.054
0.089
0.095
0.025
0.044
0.033
0.047
RotulE:
0.079
0.119
0.107
0.150
0,125
0.070
0.076
B1oBRIDGE
0.136
0.326
0.319
0.189
0.172
0.084
0.081
Result. We show MRR across the seven tasks in Table I and the breakdown performances in Tables 11, 12, 13, 14, 15, 16, and 17, respectively. We further report the overall average ranking of all methods across these tasks in Table 9. We found that BioBRIDGE is consistently ranked the best among the KGE methods. The specialized KGE algorithms learn the node and relation embeddings from scratch exclusively based on the KG, while our method builds on pre-trained FMs that already possess rich prior knowledge. As such, Bi BiORIIDE bridges modalities efficient way. Breaking down into the performance on different tasks, as shown in Table 1, we observed that BioBRIDGE gains higher margin over baselines on tasks with fewer triples from the KG. For instance, Bi OBRIDGE is around x better than the best baseline for "Protein - MF" while around 1. 1.6x better for "Protein - BP", which signals the benefit of BioBBs cLBBRIDGEE in bridging FMs with limited data for multimodal tasks over training multimodal model from scratch.
a much more data-
4.2 SEMANTIC SIMILARITY INFERENCE
Setup & Metric. The objective of this analysis is to evaluate the extent to which the encoded protein embeddings can capture biomolecular functional similarity, i.e., biological process (BP), molecular function (MF), and cellular component (CC). We follow the experimental protocol in (Unsal et al., 2022) that takes the gene ontology (GO) terms annotations of proteins as the target. For our method, we use the protein embeddings transformed to the BP, MF, and CC spaces as the input for evaluation. We compute the pairwise Manhattan Similarities of the encoded protein embeddings as the predictions. The final score is obtained by computing the Spearman' 's rank correlation between the predictions and the flattened groundtruth matrix, which is the larger, the better.
Dataset & Baseline. We leverage the test sets released by (Zhou et al., 2023b) where three 500 x 500 labeled matrices store the pair- wise Lin Similarities of the protein associated BP, MF, and CC, respectively. We aggregate these matrices to obtain 1,123 unique protein sequences and remove them from the training knowledge graph to avoid data leakage. We compare our method with the following base- lines: MSA Transformer (Rao et al., 2021), ESM-1B (Rives et al., 2021), ProtT5-XL (EI- naggar et al., 2021), ESM2-3B (Lin et al., 2023), OntoProtein (Zhang et al., 2022), and KeAP (Zhou et al.. 2023b).
The comparison on semantic similarity inference across methods. The best are in bold. "Avg" is short for the average of results.
Table 2:
Method
MF
BP
CC
Avg
MSA Transformer
0.38
0.31
0.30
0.33
ProtT5-XL
0.57
0.21
0.40
0.39
ProtBERT
0.41
0.35
0.36
0.37
ESM-1B
0.38
0.42
0.37
0.39
ESM2-3B
0.33
0.42
0.23
0.32
OntoProtein
0.41
0.36
0.36
0.38
KeAP
0.41
0.41
0.40
0.41
BiOBRIDGE
0.91
0.80
0.73
0.81
Result. We report the results in Table 2, where our method yields a substantial improvement, with around 2x better than the best baseline average. Across the baselines, observed the methods augmented by KG, including KeAP and OntoProtein, yield better results than the others, implying that KG connecting proteins and the biological attributes enhance protein representation learning. Nonetheless, BioBRIDGE learns to transform the protein embeddings to the biomolecular func- tional embedding space, thus aligning protein sequences better with the semantic meaning of func-

---

Published as a conference paper at ICLR 2024
tional terms. Also, the involvement of other modalities, like drugs from the KG in training, further enriches the supervision for the transformation model.
4.3
PROTEIN-PROTER INTERACTION
Table 3: The Fl scores of the selected methods on the protein-protein interaction task with three
datasets. "B+D" is short for the mean of BFS and DFS results. The best results are in bold. STRING
STSS27
S11ST45K
Method
Spin
BFS
B+D
B1D
BFS
B+D
KAAP
0.744
0.531
0.713
0.626
0.813
0.790
0.525
Aug (Std)
0.718 (0:040)
0.749 (QL139)
0.630
(0,004)
0,823
(0.009)
0.783
(0.013)
0,885 (0.101))
FSM2-38
0.687
0.735
0.637
0.839
0.801
0.768
0.725
0.641
0.827
0.781
0.884
0.690
0.761
0.672
0.819
0.778
0.881
Arg (Sid)
0,715 (0.1446)
0,750 (0.1350)
0.782
0.638 (0.106)
0,825 (0.010)
0.733
0,787 (0.013)
0,882 (0.0K2)
0.844
BIOERIDGR
0.704
0.764
0.645
0.540
0.791
0.890
0.765
0.722
0.638
0.834
0.787
0.391
0.701
0.778
0.644
0.529
0.769
0.588
Arg (Std)
0.723 (0.036)
0.755 (0.129)
0.739
0.643 (0.005)
0,834 (0.IN5)
0.739
(17722 (0.012)
0,890 (0.192)
0.836
Setup & Metric. We study the protein-proterie interaction (PPI) prediction task because it represents the second experiment setup: in-domain entity and out-of-domain relation. The PPI prediction task aims to classify 7 interaction types of pair of proteins: reaction, binding, post-translational modifications (pimod), activation, inhibition, catalysis, and expression. Although the ppi relation is present in PrimeKG, it only represents the physical interaction (similar to "Binding" in the seven types), while the other six types are out-of-domain.
Following the setup of (Zhang et al., 2022), we extract the protein embeddings with the baseline pre-trained protein models, which serve as the input for graph neural network model be trained on the PPI network. Our method uses the protein embeddings transformed to protein space with the relation ppi. We report the F1 score for this multi-class classification task.
Dataset & Baseline. Two baselines are selected for comparison: ESM2-3B (Lin et al., 2023) and KeAP (Zhou et al., 2023b). We test them on three PPI datasets: SHS27K (Chen al., 2019), SHS148K (Chen et al., 2019), and STRING (Lv et al., 2021). Following the setup in (Zhou et al., 2023b), we perform Breadth-Fing Search (BFS) and Depth-First Search (DFS) to generate two train/validation/fest splits, respectively.
Result. From the results in Table 3, we observe that though the results vary across splits, our method shows a consistent improvement over the basclines in most scenarios. It is illustrated that ESM2-3B performs better than the prior ofate-of-the-art KeAP, which can be attributed to its pre-training on an enormous protein database. BioBRIDGE further enhances the embeddings of ESM2 by injecting the relation "ppi", and then transforms back to the protein space. BİoBRIDGE exhibits greater benefit on the datasets with fewer samples like SHS27K as it enriches the protein embedding with the protein-protein interaction ontology information. When the number of training data increases, all methods tend to converge the same level while the baselines are still inferior to BiOBRIDGE.
4.4 CROSS-SPECIES POOTEIN-PHENOTPPE MATCHING
Setup & Metric. We propose this novel task test the capability of BioBRIDGE handle the cross mrdss-my transformation for oft-of-domain entity and in-domain relation. As PrimeKG only contains human proteins, we build a dataset of mouse proteins and the associated mouse phenotypes from the Mouse Genome Informatics (MGI) resource (Eppig et al., 2017), acting as out-of-domain entities. We elaborate on the curation process of this data in Appendix F. Since the modality "phe- notype" is absent in BioBRIDGE's 's training data, we transform the encoded mouse protein embed- dings to the "disease" space with the relation "associate with". We use a suite of ranking metrics to evaluate the matching performance, including Recall @ K, Precision @ K, and nDCG@K.
Dataset. We build two datasets for 1) matching mouse protein (MG) to mouse phenotype (MP) and 2) matching mouse protein (MG) to human phenotype (HP). The data statistics are available in Table 18. Basically, there are 28 mouse phenotypes to predict for Task and 353 human phenotypes for Task 2, respectively.
7

---

Published as a conference paper at ICLR 2024
Baseline. There is no previous work for this task. We create dual-encoder model with protein encoder: ESM2-13 IB (Lin et al., 2023) and a text encoder: PubMedBERT (Gu et al., 2021). To build the supervised baseline, we perform contrastive learning with the dual-encoder model on the paired mouse protein and phenotype.
Table 4: The comparison between the zero-shot B OBRIDGE ("O-shot") and the supervised baseline on the cross-species retrieval tasks. "Diff" indicates the relative improvement of BLOBRIDGE over the supervised baseline.
Task Mouse Protein Mouse Phenotype
Task 2: Mouse Protein to Human Phenotype
Metric
Supervised
B4 BRRTTSSR---t
Diff
Metric
Supervised B5oRRTDGR-00shr
Diff
Rec I
0.061
0.067
10%
Prec@1
0.001
0.043
3351%
Prec@1
0.360
0.338
-6%
Rec@1
0.001
0.022
3098%
Rec@3
0.145
0.185
28%
Prec@5
0.003
0.022
624%
@rec@3
0.289
0.316
9%
Rec@5
0.011
0.053
393%
Rec@5
0.194
0.278
43%
Rec@ 10
0.026
0.088
236%
Prec@5 S
0,233
0.292
25%
Rec@20
0.051
0.168
228%
nDCG@5
0.293
0.350
20%
nDCG@20
0.019
0.082
330%
Result. We report the results in Table 4. This is challenging task because neither mouse pro- teins nor mouse/human phenotypes were used in its training data. Despite this, the underlying protein FM underwent comprehensive pre-training on protein sequences across various species. As BioBRIDGE learns to bridge human protein and human diseases, it demonstrates the emergent ability to transform mouse protein from the protein space to mouse phenotype in the text space.
In Task 1, Bi BBRORRDGGGE showcases a large margin over the supervised baseline that was fine-tuned on the paired mouse protein and phenotype data. This observation underscores the feasibility of transferring BioBRIDGE to a novel domain without further training. In Task 2, the supervised baseline failed to extrapolate match mouse protein to human phenotype despite being learned on paired mouse proteins and mouse phenotypes. However, BioBRIDGE leverages the prior knowl- edge of the base FMs on protein and human disease. This inspiring result hints at the potential for novel bioinformatic analysis based on the bridged FMs enabled via cross-modal matching.
CASE STUDY: MULTIMODAL GENERATION
Table 5: Case study of multi-modal Q&A based on BioBRIDGE: Input a molecule/protein and question, retrieve the relevant diseases and proteins/gene ontology terms, and prompt LLM to answer. The references are drawn for the input molecule/protein-bee based drug pages from DrugBank.
Bega Mallnkk
Quediis What
Retiered Disomm
Butiaved Postiss
Ansrer
Relereer
Coupe lenccce supesacn
yolk
TUa0
[-] Ue Rido ausiiaa ondiie du and Viuritae aila slakid
tull
wochooo qeencccce biijeny
drig'
Dutto
and
liunna LC cnd immur ahaay
TU3DI0
Eng codebe to dkll as Bia anreent acsiit
decant
seau of Bechaalpeeeeee rach TUEOS,
servine
TURACC tachang doo pracis.
kitiir ol Vermmnne
Saar Scrtt Loydl off orery
whcch in hatoralle anarar fan appec d: ocl duth raku calorr casopii bot
Unglanc
ANTEN DOTENAATINELTE TRETY SEN.EE----- CTLLEE
lopsoctila adirima.
c11 popp pobloo
ABLEICTTVELAATTTT ORTSITLY
rigannas
The plar demp crledd
14 lovm
podre uo
precoce powd drt how
Miakin repilatioo
stlly i of me
dng Sun agar
In this section, we show how BioBRIDGE supports a multimodal generation system through cross- modal retrieval. We leverage Galactica-305 (Taylor et al., 2022) that was fine-tuned on instruction datasets as the base generator". We prompt Galactica to answer the input question or generate target molecules following the input instruction. The used prompts are in Appendix G.
Huggingface: Setpgi//teelleeeeeemnnniiiaaaeee
8

---

Published as a conference paper at ICLR 2024
Multiodal Question & Answer. BioBRIDGE accommodates multimodal input that consists of a molecule SMILES string/protein sequence with an associated question in natural language. In this task, BioBRIDGE serves cross-modal retriever to enhance the contexts for Galactica's response to the input question. Particularly, it retrieves the following: (1) identify the potential protein targets of the input molecule, (2) identify the disease indications for the input molecule, (3) identify associated diseases for the input protein, and (4) identify gene ontology terms related to the protein. We choose the drugs from DrugBank that are not in the BioBRIDGE' 's training KG. We also involve investigational drugs to test if Bi BLLBBRIGGGG can aid in proposing
ceshhanisms-of-acctin
Results are shown in Table 5, with more results in Table 19 in the appendix. demonstrates that BioBRIDGE provides key evidence that prompts Galactica to reach the right answer. For instance, BiOBRIDGE can pinpoint group of tubulin proteins and oncology-related conditions for Vin- cristine. This process enables Galactica to provide an accurate response, indicating that this drug in- hibits the mitosis of cancer cells. In addition, in Table 19, the investigational drug Rimacalib, which has been used in trials studying the treatment of rheumatoid arthritis, is identified by our method to possess immunomodulatory, anti-inflammatory, and anti-arthritic effects. It hence prompts Galactica to reach the answer that this drug may treat diseases such as rheumatoid arthritis.
Table 6: Case study of multi-modal generation based on BicBRIDGE: Input the target condition and the intended mechanism, retrieve the relevant proteins, and prompt LLM to generate the target small molecule drug.
Target Condition maligtont tymphoms
Targst Effhet
Ratrieved Protein
Generated Malscule
Mos Similar Drug
inhibrson mitosss meluphas of cancer cells via p-lychrmithergg.
MTTI
BCL22
TYMS
CAT
CASPS
CSP3
COKNIA
CREBBP
Prosze anincsppaaci agant cated the treamment itage itage Hidglin's contiinusion other chamethengaaiic agerrs.
S002
Multi-Modal Generation. This task aims to achieve the text-guided generation of mooel-mo drugs. We use BiOBRIDGE to enrich the context for Galactica by retrieving the target proteins that are possibly associated with the target conditions. To validate the structural integrity of the generated molecules, we utilize RDKit to calculate the Tanimoto distance between the generated SMILES string and all candidate small molecule drugs listed in DrugBank. We then identify the most similar drugs. Results are shown in Table with more in Table 20. We also made a base- line that prompts Galactica to generate the molecule directly, shown in Table 21. We found that (1) BİOBRIDGE prompts Galactica to generate valid drug molecules; (2) the generated molecule usually shares similarities with the real drugs that were considered effective for the target condition; (3) Prompting Galactica without RAG leads to poor generation results. For instance, in Table 6, the model-generatee drug is most similar to Procarbazine, which is used to treat stage III/IV Hodgkin's 's disease in chemotherapy by impeding the division of cancer cells. This implics the generated drug probably fits the proposed target effect and treats lymphoma.
6 CONCLUSION
This paper investigated bridging unimodal biomedical foundation models (FM) for multimodal tasks. We identified that BioBRIDGE can effectively transform the embeddings to the target modal- ity, considering the types of source modality, target modality, and their relations. It is with great parameter efficiency: only the bridge module needs training while all the base FMs are kept fixed, supervised by the relational information from biomedical knowledge graphs. We also identified that BioBRIDGE can handle a diverse set of cross-modal prediction tasks by extrapolating to in- domain/out-of-domail entities and relations. And the yielded performances are par with the supervised specialist models in each task. In addition. we demonstrated how the bridged FMs can support generation tasks with multimodal inputs. In the future, envision that Bi BiOBRIDGE can be extended to connect pre- pre-trained FMs from other domains as long as entities across different modalities can be represented in KG.

---

Published as a conference paper at ICLR 2024
REFERENCES
Viraj Bagal, Rishal Aggarwal, PK Vinod, and Deva Priyakumar. MolGPT: molecular generation using a transformer-ecooder model. Journal of Chemical Information and Modeling. 62(9))2064- 2076, 2021.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv: (tt::2/08.0000 20t.:2008..7.588.. 2021.
Antoine Bordes, Nicolas Usunier, Alberto Garcia- Baran,, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data, Advances in Neural Information Pro- cessing Systems, 26, 2013.
Stephen P Boyd and Lieven Vandenberghe. Convex optinization. Cambridge university press, 2004.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
Payal Chandak, Kexin Huang, and Marinka Zitnik. Building a knowledge graph to enable pre- cision medicine. Scientific Data, 10(1):67, 2023. URL
341597-023-019660
Muhao Chen, Chelsea J-T Ju, Guangyu Zhou, Xuelu Chen, Tianran Zhang, Kai-Wei Chang, Carlo Zaniolo, and Wei Wang. Multifaceted protein-protein interaction prediction based on siamese residual renn. Bioinformatics, 35(14):305-i3144 2019.
John Conway. course in functional analysis, volume 96. Springer, 2019.
George Cybenko. Approximation by superpositions sigmoidal function. Mathematics control, signals and systems, ((4):303-314 1989.
Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. Translation be- tween molecules and natural language. In Conference on Empirical Methods in Natural Language Processing, 2022.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang. Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: Toward un- derstanding the language of life through supervupe learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, (4(11):7112-77) 2021.
Janan T Eppig, Cynthia I. Smith, Judith A Blake, Martin Ringwald, James A Kadin, Joel E. Richard- son, and Carol Bult. Mouse genome informatics (mgi): resources for mining mouse genetic, genomic, and biological data in support of primary and translational research. Systems Genetics; Methods and Protocols, pp. 2017.
Benedek Fabian, Thomas Edlich, Héléna Gaspar, Marwin Segler, Joshua Meyers, Marco Fiscato, and Mohamed Ahmed. Molecular representation learning with language models and domain- relevant auxiliary tasks. arXiv preprint arXiv:201 1. /3230, 2020.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190, 2023.
Vladimir Gligorijević, P Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Beren- berg, Tommi Vatanen, Chris Chandler, Bryn Taylor, Ian M Fisk, Hera Vlamakis, al. Structure- based protein function prediction using graph convolutional networks. Nature communications, 12(1):3168, 2021.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing, ACM Transactions on Computing for Healthcare (HEALTH), 3(1): 1-23, 2021.
10

---

Published as a conference paper at ICLR 2024
Logan Hallee and Jason P Gleghorn. Protein- proteen interaction prediction is achievable with large language models. bioRxiv, pp. 2023-002 06, 2023.
Paul R Halmos. Finite-dimensional vector spaces. Courier Dover Publications, 2017.
Xu Han, Shulin Cao, Lv Xin, Yankai Lin, Zhiyuan Liu, Maosong Sun, and Juanzi Li. OpenKE: An open toolkit for knowledge embedding. In Proceedings of EMNLP, 2018.
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, 4 (2):251-257, 1991.
Kexin Huang, Cao Xiao, Lucas M Glass, and Jimeng Sun. MolTrans: molecular interaction trans- former for drug-target interaction prediction. Bioinformatics, (7(6))830-82) 2021.
Vassilis loannidis, Xiang Song, Saurav Manchanda, Mufei Li, Xiaoqin Pan, Da Zheng, Xia Ning, Xiangxiang Zeng, and George Karypis. Drkg drug repurposing knowledge graph for 19. https: / com/ http//gnnd//////// 2020.
Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. Knowledge graph embedding via dynamic mapping matrix. In Proceedings of ACL, pp. 687-696, 2015.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool. Russ Bates, Augustin Zídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-58,, 2021.
Yogesh Kalakoti, Shashank Yadav, and Durai Sundar. Transdti: transformer-bast language models for estimating dtis and building a drug recommendation workflow. ACS omega, 73))2706-2777,
2022.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. arXiv preprint Wiii:23344.024333 2023.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation em- beddings for knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, (1166667):1123-1111 2023.
Shengehao Liu, Yutao Zhu, Jiarui Lu, Zhao Xu, Weili Nie, Anthony Gitter, Chaowei Xiao, Jian Tang, Hongyu Guo, and Anima Anandkumar. A text-guided protein design framework. arXiv preprint 200.v:2302.04611 2023.
Shuwen Liu, Bernardo Grau, Ian Horrocks, and Egor Kostylev. Indigo: ban-based inductive knowl- edge graph completion using pair- wise encoding. Advances Neural Information Processing Systems, 34:2034-204 2044, 2021.
Xinyu Lu, Lifang Wang, Zejun Jiang, Shichang He, and Shizhong Liu. MMKRL: A robust embed- ding approach for multi-modal knowledge graph representation learning. Applied Intelligence, pp. 1-18, 2022.
Guofeng Lv, Zhiqiang Hu, Yanguang Bi, and Shaoting Zhang. Learning unknown from cor- relations: graph neural network for inter-novel-protiin interaction prediction. arXiv preprint arXiv:2105.06700 2021.
Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian. Benjamin Mohr. James M Holton, Jose Luis Olmos Jr, Caiming Xiong, Zachary Z. Sun, Richard Socher, et al. Large language models generate functional protein sequences across diverse families. Nature Biotechnology, pp- 1-8. 2023.
Hatem Housselly-Sergiee Teresa Botschen, Iryna Gurevych, and Stefan Roth. A multimodal translation-base approach for knowledge graph representation learning. In Proceedings of the Seventh Joint Conference Lexical and Computational Semantics, pp. 225-234, 2018.

---

Published as a conference paper at ICLR 2024
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. In NeurlPS, 2018.
Jianing Qiu, Lin Li, Jiankai Sun, Jiachuan Peng, Peilun Shi, Ruiyang Zhang, Yinzhao Dong, Kyle Lam, Frank P-W Lo, Bo Xiao, et al. Large ai models in health informatics: Applications, chal- lenges, and the future. arXiv preprint arXiv:2303. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Picter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. In International Conference on Machine Learning, pp- 8856. PMLR, 2021.
Nikhil Rasiwasia, Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert RG Lanckriet, Roger Levy, and Nuno Vasconcelos. A new approach to cross-modal multimedia retrieval. In Proceed- ings of the 18th ACM international conference on Multimedia, pp. 2-0- 2010.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15):e20162311 2021.
Samuel Sledzieski, Rohit Singh, Lenore Cowen, and Bonnie Berger. Adapting protein language models for rapid dti prediction. bioRxiv, pp. 2022-11, 2022.
Teague Sterling and John J Irwin. Zinc 15-lia discovery for everyone. Journal of chemical information and modeling. 55(1 25(11))332--3337 2015.
Gilbert Strang. Linear algebra and its applications. 2012.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding 10197, 2019.
by relational rotation in complex space. arXiv preprint
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: large language model for science. arXiv preprint arXiv:2211.000S: 2022.
Komal Teru, Etienne Denis, and Will Hamilton. Inductive relation prediction by subgraph reasoning. In International Conference on Machine Learning, pp. 9448-9457 PMLR, 2020.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Com- plex embeddings for simple link prediction. International Conference on Machine Learning, pp.
PMLR, 2016.
Serbulent Unsal, Heval Atas, Muammer Albayrak, Kemal Turhan, Aybar Acar, and Tunca Doğan. Learning functional properties of proteins with language models. Nature Machine Intelligence, 15:227-245, 2022.
Sheng Wang, Yuzhi Guo, Yuhong Wang, Hongmao Sun, and Junzhou Huang. SMILES-BERT: large scale unsupervised pre-training for molecular property prediction. In Proceedings of the 10th ACM international conference bioinformatics, computational biology and health informatics, pp. 436, 2019a
Yanbin Wang, Zhu-Hong You, Shan Yang, Xiao Li, Tong- Hai Jiang, and Xi Zhou. A high efficient biological language model for predicting protein-protein interactions. Cells, 8(2):122, 2019b.
Zhen Wang, Jianwen Zhang, Jianlin Feng and Zheng Chen. Knowledge graph embedding by trans- lating on hyperplanes. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 28, 2014.
12

---

Published as a conference paper at ICLR 2024
Zichen Wang, Steven A Combs, Ryan Brand, Miguel Romero Calvo, Panpan Xu, George Price, Nataliya Golovach, Emmanuel Salawu, Colby J Wise, Sni Priya Ponnapalli, et al. Lm-gyp: an extensible sequence and structure informed deep learning framework for protein property predic- tion. Scientific reports, 12(1 ):6832, 2022a.
Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medelip: Contrastive learning from unpaired medical images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3876-3887 2022b.
Ruobing Xie, Stefan Heinrich, Zhiyuan Liu, Cornelius Weber, Yuan Yao, Stefan Wermter, and Maosong Sun. Integrating image-based and knowledge-based representation learning. JEEE Transactions Cognitive and Developmental Systems, 12(2):169-17) 2019.
Hanwen Xu, Addie Woicik, Hoifung Poon, Russ B Altman, and Sheng Wang. Multilingual trans-
lation for zero-shot biomedical classification using biotranslator. Nature Communications, 14(1): 738, 2023a.
Minghao Xu, Xinyu Yuan, Santiago Miret, and Jian Tang. Protst: Multi- mudti-modality learning of protein sequences and biomedical texts. In ICML, 2023b.
Bishan Yang, Scott Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In Proceedings of the International Conference Learning Representations (ICLR) 2015, 2015.
Ningyu Zhang, Zhen Bi, Xiaozhuan Liang, Siyuan Cheng, Haosen Hong, Shumin Deng, Qiang Zhang, Jiazhang Lian, and Huajun Chen. Ontoprotein: Protein pretraining with gene ontology embedding. In International Conference on Learning Representations, 2022.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-Mol: A universal 3d molecular representation learning framework. In The Eleventh International Conference on Learning Representations, 2023a.
Hong- Yu Zhou, Yunxiang Fu, Zhicheng Zhang, Bian Cheng, and Yizhou Yu. Protein representation
learning via knowledge enhanced primary structure reasoning. In The Eleventh International Conference on Learning Representations, 2023b.
13

---

Published as a conference paper at ICLR 2024
A PROVING EXISTENCE OF BRIDGE MODULE AND LEARNABILITY
This section aims to show the existence of a bridge module, followed by its learnability. Given our setup of unimodal FMs, we will show the existence of transformation modules that can take us from any modality's 's output space to another under certain assumptions. We will proceed to first state our assumptions and then the statement, followed by the proof for the statement.
Assumptions (Assumptions on the neural networks and Knowledge Graph), 1. Let K Z be the total number of modalities and let M1..... MR denote the K neural networks trained on the different modalities whose parameters are now frozen.
2. For every neural network Ms S2 + Sk. d >> 0, E b), S IRd, QN denotes the input spacel domain and the output space Sy for every neural network a linear subspace of TRd. Specifically, we will assume the dimension of every subspace is the same.
3. Let 9 = (V,8) denote the knowledge graph where V is the set of nodes and & is the set of edges. Each node
V belongs one of the K modalities.
4. Every e E E of the knowledge graph which connects two nodes Us, D2 E V has an associated relation type rij E R.
Theorem 1 (Existence of Bridge Module). For any given pair of nodes U, E V of modality types ke,s Re, E {l....K), and with representations given by their appropriate neural networks So, Skup S0y E &Skay' which are connected by relation type Fij E RR there exists a bridge module B Rd X fl..... K} x (1....,,K) x R + such that B : (Sv,, Kuc Toyy + Suz
Proof. At a very high level, direct consequence of the Hahn- Theorem (Conway, 2019) can be used to see that given a Euclidean space and given two compact, convex subsets of that Euclidean space, there exists continuous function mapping between them (given that there is linear func- tional that separates them). Given our assumption that our subspaces are linear and therefore convex, as well as the assumption of continuity of neural networks, we can quickly assert the existence of transformations between the spaces.
Here, however, we will go into the details and show a simple proof using linear algebra. We will use a multi-step approach to prove the existence of the bridge module. Firstly, we will look at expressibility in terms of linear subspaces and then introduce conditional transformations based on relation types. Subsequently, we will look at the compositionality and aggregation of paths (as we are dealing with knowledge graphs) for the bridge module. We will specifically look at linear subspaces here, but similar arguments can also be established for convex hull-like spaces or other spaces that have an established geometry.
First, we look at the linear mapping between linear subspaces. Given any two linear subspaces Sn,, Sy. a linear transformation between the two is one which maps points in Sn to S, i.e. Tay S. + S,y and this can always be represented using matrix Any as (p) = Azy p,p E Sn. The existence of such a transformation is guaranteed by the rank nullity theorem (Strang, 2012) without any loss of information (as the dimensions of the subspaces are taken to be the same by construction).
Next, any relation in raj E R can be represented as a vector, i.e., relation embedding r(ry) R..p p > 0 and the transformation T between subspaces can be a function over the matrix Any as well as r(ray) i.e. Agy f [Aay, (Amy,r(rig)) and the conditional transformation (relation dependent) of any point p E S2 via relation riy is now given as Thy (p) Ary p. Such a formulation also allows us to directly model compositionality of transformations (due to the properties of matrices) as given two transformations Tay: Tys between S₂ + S,, + S., a composition Tyy o Tay (given by ATTh Ary in matrix notation) captures a transformation from S, to S., Given that we are dealing with linear subspaces and transformations, these compositions are always defined (Strang, 2012).
Next, we look at the aggregation of multiple paths - since we are dealing with KGs, there can be mul- tiple paths between any two nodes. In such cases, the transformation between any two subspaces can then be obtained as a weighted combination of the paths, i.e., Aagg = ΣN A;A; such that ΣN W/ = 1 where w; is the weight associated with a path and A gives the transformation matrix between the spaces. When all w/ O, this forms convex combination and ensures that the aggre- gated transformation is still within the space by the original transformations (Boyd & Vandenberghe, 2004) and therefore confirms the existence of unified bridge module.
14

---

Published as a conference paper at ICLR 2024
Now, given the existence of the above transformations defined by the KG, the convergence of sequences of linear transformations (Halmos, 2017), the universality of neural networks (Homik, 1991: Cybenko, 1989) and the continuity of the unimodal FM's ensure that such conditional trans- formations (given by relation types) can be approximated when appropriate loss functions are em- ployed via SGD.
B ESTABLISHING TRAINING DATA BASED ON PRIMEKG
Table 7: The statistics of nodes and edges of the raw PrimeKG data and the processed KG for training Bi BiOBRIDG.. "Original": the original PrimeKG; "Processed": the processed KG; "Dropped": how many samples are dropped in the processing.
Nodes
Modality
Original
Processed
Dropped
Percent dropped
biological process
28,642
27,478
1,164
4.06%
protein
27,671
19,162
8,509
30.75%
disease
17,080
17,080
0
0.00%
molecular function
11.169
10,966
203
1.82%
drug
7,957
6,948
1,009
12.68%
cellular component
4.176
4,013
3.90%
Summation
96,695
85,647
11,048
11.43%
Edges
Relation type drug.drug
Original
Processed
Dropped
Percent Dropped
2,672,628
2,241,466
431,162
16.13%
prottin
642,150
629,208
12,942
2.02%
bioprocess protein
289,610
272.642
16,968
5.86%
celcommp-protein
166,804
149,504
17,300
10.37%
disease protein
160,822
155,924
4,898
3.05%
molfune protein
139,060
133,522
5,538
3,98%
bioprocess hoooooessshioproeess bioprocess
105,772
99,630
6,142
5.81% %
disease disease
64.388
64,388
0.00%
contraindication
61,350
60,130
1,220
1.99%
drug-po trug-protein
51,306
47,614
3,692
7.20%
molfunc.m molfune
27.148
26,436
712
2.62%
indication
18.776
17,578
1,198
6.38%
cellcomp_edd ceccooompeeelompppppcm
9,690
9,200
490
5.06%
off-label use Summation
5,136
4,998
2.69%
4,414.640
3,912,240
502,400
11.38%
This paper aims to make a proof of concept in bridging uni-modal FMs, so simplifies the KG in the experiment by only keeping six modalities. Nonetheless, protein, disease, drug, and gene ontology terms are the main biomedical entities covering the majority of real-world biomedical tasks (c.g., drug discovery, repurposing, PPI, protein function prediction. drug target identification, drug-target interaction prediction).
The raw PrimeKG provides a bind of biomedical entities from widespread of sources. However, it does not provide the associated properties for all the entities, e.g., the sequence structure of the proteins. To train BiOBRIDGE, we try to link the entities to the external knowledge bases and filter out those without the required property. The statistics of raw KG and the processed are available in Table We describe the data curation process for each type of entities as the following.
Protein Data. There are 27,671 proteins in total in the original PrimeKG. We try to match the provided NCBI gene ID of the proteins to the UniProtKB/Swiss- uniprot id-mapping tool https / www.. uniprot. org/ dd-aappin.......... We were able trieve 27,478 protein sequences that are matched with the gene ID.
sequence database, via the
We delve deeper into the unmapped proteins and find most non parporii-nooding genes (pseudo- genes, rRNA, ncRNA genes), which are genes unable to produce proteins. It is hence reasonable to ignore them for protein-centric tasks.
15

---

Published as a conference paper at ICLR 2024
Drug Data. There are 7,957 drugs in the original PrimeKG, We try match the offered DrugBank ID to the database ht https: /go /tgoodduuuee drugbank, com/ drugs. We dropped drugs without SMILES strings to obtain total of 6,948 drugs.
Gene Ontology Terms. Biological process, molecular function, and cellular component are rep- resented by all gene ontology (GO) terms. We leveraged AmiGO for searching the detailed de- seriptions of the GO terms through their IDs DDs 1Dshtpp: //ts::///mmm /amigo. geneont o ogy org/ ami / search/ ontol ogy. We were able retrieve the features of the most GO terms then kept 27,478 BP, 10,966 MF, and 4,013 MF, in the training data.
Disease. The descriptions of those diseases provided by the raw PrimeKG data able keep all 17,080 diseases for training.
C DATA STATISTICS OF THE KNOWLEDGE GRAPH
Table 8: The data statistics of the training knowledge graph drawn from PrimeKG.
Heaad Node
Tail Node
Relation
Number
protein
protein
ppi interacts with
642,150
biological process
144,805
cellular component
interacts with
83,402
molccular function
interacts with
69,530
discase
associated with
80,411
drug
target
16,380
enzymc
5,317
transporter
3,092
carrier
864
drug
drug
synergistic interaction
2,672,628
disease
contraindication
30,675
indication
9,388
off-label use
2,568
protein
target
16,380
enzyme
5,317
transporter
3,092
carrier
864
disease
disease
parent-child
64,388
gene/protein
associated with
80,411
drug
contraindication
30,675
indication
9,388
off-label use
2,568
D CROSS-MODALITY RETRIEVAL TASKS
Table 9; The mean and standard deviation (in parenthesis) of the ranks of all compared methods across the seven cross-modality retrieval tasks. Lower rank better. Best are bold.
Method
Rank MRR ())
Rank-Hits@ (4)
Rank-Hits@3-00 ())
Rank-Hits@1 (L)
TransE
7.71 (0.00)
7.71 (0.00)
7.43 (0.52)
5.00 (0.00)
TransR
5,00 (1.41)
4.57 (1.63)
5.14 (1.47)
5.00 (0.00)
TransH
5.86 (0.82)
5.71 (0.55)
6:00 (0.41)
5,00 (0.00)
TransD
6.71 (0.84)
6.57 (0.82)
6.57 (0.82)
5.00
ComplEx
3,00 (0.63)
3.43 (0.55)
3.00 (0.63)
2,86 (0.63)
DistMult
4.57 (1.03)
5.00 (1.67)
4,00 (1.79)
3.14 (0.89)
Rotath:
2.14 (0.41)
2.00 (0.00)
2.43 (0.84)
2.86 (0.98)
BioBRIDGE
1.00 (0.00)
1.00 (0.00)
1.00 (0.00)
1.00 (0.00)
16

---

Published as a conference paper at ICLR 2024
Table 10: The data statisties of the train/valid/test splits used in cross-modality retrieval tasks.
Triple Type
Head Node
Tail Node
Train Triple
Valid Triple
Test Triple
Protein Interacts with - BP
16,808
12,047
111,649
11,620
13,052
Protein Interacts with
17,289
4,284
55,252
5,438
6,071
Protein Interacts with CC
17,211
1,684
61,471
6,250
7,031
Protein Associated with Disease
8,548
5,566
63,869
6,659
7,434
Drug Indication Disease
1,279
7.290
Drug Contraindiation Disease
1,217
1,181
24,423
2,676
2,966
Drug Target Protein
5,074
2,620
12,952
1,055
Table 11: Cross-modality prediction performance for triple type: "Protein Interacts with BP".
Method
MRR
Hit@ 10
Hit@3
Hit@1
MRR Rank
Hit@ 10 Rank
Hit@3 Rank
Hit@ Rank
TransE
0.034
0.097
0.020
0.000
5
TransR
0.045
0.136
0.023
0.000
5
5
TransH
0.129
0.023
5
TransD
0.125
0.020
0.000
5
ComplEx
0,218
0.067
0.020
DistMult
0.054
0.045
0.013
3
RotatE
0.079
0.239
0.057
0.012
BIOBRIDGE
0.136
0.282
0.146
0.066
1
Table 12: Cross-modality prediction performance for triple type: "Protein Interacts with MF".
Method
MRR
Hit@ 10
Hit@3
Hit@1
MRR Runk
Hit@ 10 Rank
Hit@3 Runk
Hit@ Rank
TransE
0.046
0.143
0.030
0.000
TransR
0.060
0.196
0.041
0.000
5
TransH
0.199
0.040
0.000
TransD
0.185
0.039
5
ComplEx
0,289
0.020
DistMult
0.080
0.033
2
RotatF
0.119
0.360
0.103
0.021
3
BiOBRIDGE
0.326
0.653
0.382
0.177
Table 13: Cross-modality prediction performance for triple type: "Protein interacts with CC
Method
MRR
Hit@ 10
Hit@3
Hit@1
MRR Rank
Hit@ 10 Rank
Hit@3 Rank
@it@ Rank
0.044
0.133
0.026
0.000
TransR
0.048
0.142
0.023
0.000
5
TransH
0.057
0.176
0.000
TransD
0.162
0.026
0.000
ComplEx
0.099
0.267
0.023
3
DistMult
0.092
0.034
RotatE 0.107
0.330
0.079
0.018
4
BioBRIDGE
0.319
0.676
0.373
0.160
1
Table 14: Cross-modality prediction performance for triple type: "Drug Indication Disease".
Method
MRR
Hit@ 10
Hit@3
Hit@1
MRR Rank
Hit@ 10 Rank
Hit@3 3 Rank
Hit@ Rank
0.017
0.057
0.006
0.000
TransR
0.152
0.019
0.000
5
TransH
0.074
0.015
0.000
TransD
0.064
0.018
0.000
ComplEx
0.042
0.084
0.028
0.010
3
DistMult
0.025
0.063
0.010
0.004
4
RotatE
0.150
0.357
0.150
0.054
BioBRIDGE
0.189
0.380
0.176
0.071
1
17

---

Published as a conference paper at ICLR 2024
Table 15: Cross-modality prediction performance for triple type: "Protein Target Drug".
Method
MRR
Hit@
Hit@3
Hit@1
MRR Rank
Hit@ Rank
Hit@3 Rank
Hit@ Rank
TransE
0.033
0.086
0.016
0.000
8
8
5
TransR
0.069
0.204
0.058
0.000
4
4
5
TransH
0.043
0.127
0.018
0.000
7
5
TransD
0.049
0.150
0.018
0.000
5
5
ComplEx
0.079
0.184
0.075
0.021
3
3
DistMult
0,044
0.104
0.035
0.008
5
RotatE
0,125
0,258
0.131
0.051
2
BicBRIDGE
0.299
0.107
Table 16: Cross-modality prediction performance for triple type: "Disease
Associated with Pro-
tein".
Method
MRR
Hit@ 10
Hit@3
Hit@1
MRR Rank
Hit@ 10 Rank
Hit@3 Rank
Hit@l Rank
TransE
0.015
0.002
0.000
TransR
0.066
0.007
0.000
TransH
0.014
0.020
0.003
0.000
5
TransD
0.018
0.002
0.000
ComplEx
0.048
0.106
0.031
0.010
3
3
DistMult
0.047
0.113
0.035
0.010
3
RotatE
0,076
0.172
0.059
0.021
2
2
2
BLOBRIDGE
0.081
0.184
0.071
0.025
Table 17: Cross-modality prediction performance for triple type:
"Drug
Contraindication Dis-
ease".
Method
MRR
Hit@ 10
Hit@3
Hit@1
MRR Rank
Hit@ 10 Rank
Hit@3 Rank
Hit@ Rank
TransE
0.010
0.015
0.002
0.000
5
TransR
0.029
0.066
0.007
0.000
5
TransH
0.014
0.020
0.003
0.000
5
TransD
0.013
0.018
0.002
0.000
7
7
5
ComplEx
0.048
0.106
0.031
0.010
3
4
4
3
DistMult
0.047
0.113
0.035
0.010
4
3
3
RotatE
0.076
0.172
0.059
0.021
2
2
2
2
BiOBRIDGE
0.081
0.184
0.071
0.025
1
1
Table 18: The statistics the built cross- cross-specifies retrieval dataset.
Mouse Protein Mouse Phenotype
Mouse Protein Human Phenotype
Item
Train
Test
Item
Test
MGI
6,069
6,068
MGI
1,615
MPI
28
28
HPO
353
Pairs
38,705
37,937
Pairs
3,515
Mean MPI
6.38
6.26
Mean HPO
2.18
E MOUSE PROTEIN AND PHENOTYPE DATASET
We retrieved the paired mouse protein and phenotype data from httpt:://t////tt http:://www....oomrie.c.o jax. org/ down oads reportss////////e// /MMD HiMD__HaaaPPhheeee ype. rpt, yielding total of 29 99,686 paired mouse protein ID (MGI), human protein ID, and mouse phenotype ID (MPI), namely datal. We then retrieved the detailed descriptions of mouse phenotypes from https:/ www. iinformate ics jax org/ wwt/dwwwl.oooooooooooooooo... dwwddoowoddd wttpd///rrpttt//eeeooooo.o /MtphMoooooooeee oltoy yield- ing 14,201 MPI and their definition, namely data2.
To obtain the exact sequence structure of mouse proteins, we called UniProt https:://// //ww.. uniprot. orq httpidddmmpppnn mapping system, choosing "MGI" under "Organism-specific databases" as the source database, "UniproKB/Swiss-PPe as the target database. This process maps MGI the corresponding protein sequences, namely d.ta3.
18

---

Published as a conference paper at ICLR 2024
We only keep MGI that is with protein sequence referring to data3 from datal. Also, we only keep MGI in datal that is linked to more than one MPI. Finally, we conduct a filter from datal to keep human proteins that are in the PrimeKG data and link to more than one human protein. The obtained dataset then has 14,024 MGI, 14,200 MPI, 1,582 Human Proteins, and 3,279 Human Phenotypes in total.
F CASE STUDY: MULTIMODAL Q&A
Table 19: Additional case studies of multi-modal Q&A based on Bi OBRRIDE:: Input a molecule and question, retrieve the relevant diseases and proteins, and prompt LLM to answer. The references are drawn for the input molecule-based drug pages from DrugBank.
Midecade
Oreatkos
Berrieed Diowee
Retrieeed
Relivreee
XEXPRII
dincuce prizee the moceade this cual receedd drng vay live iewn world lineyy wii valorrnnee.. antartwii
Eanactt bee axe maa uaaiag the vorera (Drggtesic DB12071
pvorerapien spanall ansepanois
ITRC
CLHDCHOCHCI
dmesio of do
alecgi Dreakt Cornatto
MEATKS
Tax disa Eai labil de adtiigg of MARK34 souds play rokia ia the pro-
peo
EIF33
semiis llggg oon dereallls aloepe nalvna
The roneeated
alaggic aubaaa
CREMIP
abiiii by CDCC, whic incered is slaan the cyds praliteztine lenaaa cite. This antiiise laka eytlcralionn oloocse
Supec arydaramee
sungeotee hoatt larkere
het eick 24 I. which ain invtived is rega adio of iennaso siqueias anl infuernakes
Sny ucimac to0000 and Cheanst winch tuaverptiou. co-ictitoon ful pronuee the on eeposciion gorar lineleed ced-
Ovenil 8x sain platracodmunte of dio stadl cudlieds tres sax miseee option the woleen dilin wilk as porkioss sllega
SEORRURDNT TIREECICON SOCHHID
DON4
eodoctk sugets wokertee dry type orter meccamic abealyy dibbiic kunacidect and hipelar disectore Appenbyl pepsiiue heee (1NPP4) the buuae piaa ITAP, which run ranage tye dub and cheatyy The dog the mik nnvenry sriers docae odher coepd eneeecadd duseton Untiiiiio of BAP wiich may heee is stiiayy in managin sime
Saupliii dispees peeduued mir-- ofope dabens [DrgBlokk Baasiss)
Syp
F2
rais pasi
(DSOUCTICUCC
Amazio of du
Mpplar dhonke
19
PLAT
doeder
ACT
D0M
Sevie
WEAT cas The robecy
5052
ford poorllc
Karzlee Iastanne HD" smsieein oud sea day skons,
N051
assccaaed cood dii dl dong
COT942
2. Phdnneacy andian (hoea))
maadans gatti dacns (Deeghaak (i0066))
ske Aseas
Cheree daskes lies
(e).
Pepir
Ectaniia Istiokin dionss
CITT44
The man durmacody dee cnad exdectaa dne wald capeed
kay sogenaircy trut diseee
ex) 1. ree here
aus of ferenskry cleat lepat
and peotigheddvsss radqunodd systau 3. may lis artoedir and
ITsis antly mon betereeesst rach recleddi kmo uaikle.
www.eeaacyycccccccccc clichs Addiseals liv
DP2MI
Toaae rstaut
HOOGACIC (ORD C-NEee1100
ac Haraac--
slegic dkabde
00601
scfins and alapa rockeee The rads place andderaea of stul.
maic atzeceyinn and as-
Aog'
old
clus the valuss reaverineatio aonteee inme in piis read iepertiis
inservelld del is ootirs de
skgt: on
and innnan: faactee cae lune and-aabarratt and allepi abhouhh
cun with lacovar compting.
DR05
undenttod as aloepa
pepakey cogenctviis
digg rus le effiatee touting and wihoo linglo
D020
regondeens
Wha ans- staal oxciiin
akoball Sepeadeace
S2.944
The seall au releeda dreg desothed cuatiple porazs besttod the
Dounsscare landdd has heen mnd in unde sudsig the Patans, Depordice Bra
IC JOMOHOOL Ciciren)
HIRIA
ripi maan vakoe bureiia tim feci recpees pa hecpo The
dyariii of thüs dong
akobal sheu
22.5442
dreg and slated deperdinca cabor soe alss and depandrns uce and riled and
crmabo dqpaknnoe
SLD844
aemamnaaann dar tnchi the decrabdanmnn de sa- intsuventer syvere vanecess depassc acty Khiiii ec waiking
Reaa ixparees, saaeg
cocall shave
anc Tragllade nailan))
cogi fuaition queiii
the kanna Inaatee of Bioc perides, das tins cold poissall sqatis and with the oon and foomss and cosd.
allvvaae reqe toei.
Whater of chand oveiti
TF
Starliiee alds lool
Buldyy loy disc
5002
raly hene and howe wht offect in targin seneaal pei- supecoddd dier soreaie 2. mradteriing bng growth facae bate and vaciilar at- growt
in of doxtedd items refee. creaiig prencing mecha--
Apuria of thu Jog?
11.5
heware
shes Aaciie
NINE2
ierea aude towes treg maey oflectee tomingg di- clsts that weuhe ennuatioo and voddaate siese cach cliffii the diuase, dustiaal
TIRC
ton and danuging rab-
darthed
www.ggggggddd rollas disnee
VEGRA
have poicesa. in basing gutenci plauuct rellos dictsc yalltbdarr Fieerer it was that dene josr specalitio aowe lnt ous. peovided
mnliic ads mennse low-
and slan cares laom in
104-00000A
sacna (kangg of pa-
exparty
19

---

Published as a conference paper at ICLR 2024
Table 20: Case study of multi-modal generation based on BioBRIDGE: Input the target condition and the intended mechanism, retrieve the relevant proteins, and prompt LLM to generate the target small molecule drug.
Target Condition
Targel Effect
Retrieved Protein
Generated Molecule
Mosl Similtar Drug
Depressive diserler
drug trests depressive discrler by being selective dopumine receptor unisgonist, hence inhibiing dopumin- ergie hyperactivity.
IITRIA
OPRMI
HIKZA
HTRZC
BONF
CNRI
PENK
TPH11
Levorpbunal an opioid analgesic
used to treat moderate severe pain.
Depressive diserder
The depressive discrder being depumine receplur antagonist, heace inhibiting dopamin- ergae hyperactivity.
DRD2
HIRIA
HTR2C
ILI8
RONF
CNRI
PENK
TPHI
Ty2456102 his been used trials stadying the health services research and busic science of Amssty Disor- ders and Alcohol Dependence.
Breast
We drug that target following proteins: [protcins..] to teat breast cancer.
BRCA2
NOTCH2
TYMS
RGFR
NOTCIE
TP53
CDKNZA
ESR2
NCOAJ
Capecitabine
mucleoside
metahollc inbihitor indicuted
traat differens gastroimestiral, in- cluding pomcrestic concer, and breast cancer.
Breust Cancer
We design drug that target proteins [protcins..] treat breast cancor.
BRCA2
NOTCHE
TYMS
BGFR
NOTCHB
TP53
CDKN2A
ESR2
Mitorantroee a chemotherapeu tie agent used for the treamment of
ERBB2
secondacy progessiive, relapsing. worsening relapsing- romitingg mulliple
20

---

Published as a conference paper at ICLR 2024
Table 21: The baseline study of multi-modal generation based Galactica generator without the retrieval augmentation by BioBRIDGE; Input the target condition and the intended mechanism and prompt LLM to generate the target small molecule drug directly.
Target Condition
larget Efflct
Retrieved Protein
Generuted Modecule
Most Similar Drug
maliguant
ralilition
metaphase
polychemuberapy.
Hosa compooweaenee un cs Meperidine an opiond agonist with analgesic and sedative properties used
to mantggo severe pain.
malignant
metophase
OH
polychemoterrpy..
EAMLES ThIns
Phenol antiseptic and disinfec- setting..
Depressive disorder
drug treats depressive disorder
being selective dopamine recepoo untagonist, hence inhibiting dopamin ergic hypenctivity.
Thcrolinuu cakineerin inhibitar prevent organ transplant joction to severe atopic dermatiiis.
Teprrssive discedr
The dnig mests depressive disordrryy by NA
being selective dopamine receptor amagonist. hence inhihiiin dopamin- argic hyperactivity.
A waler-sohife soluble, culurless crystal chemicsl modikine, manufacture laqques, perfume esters.
Breast cancer
We wast to design a drug to treat NVA beeast cuncer.
5000000000000000
10000
FODESRIO Poldecsd OCCOOOOOOO
Polidocanal is selerosing agent used the trealment of uncumpiictted spider incomplictted retio- etcr, extremity.
than
Breast Cancer
We wast to desiga a drug to teat NAA
berast cancer.
owp.geo--ooooocee..... C-020-01
12000 129 Succini Add OC1-0p0000)
A water-schible, coloriess crysta acid chemical modikine. manufacture loqquers perfume
21

---

Published as a conference paper at ICLR 2024
G CASE STUDY: PROMPT
Listing 1:
The prompt for prompting language models questions takes the drug molecule structure as the input.
prompt """Drug molecule structure: [START_T TTARRTT_SMMA SMILES] (ami (es) HIND__SMMIBR
Target proteins: (protein_names
Associated diseases:
(aisease_names)
Consider the associated di seasss and the proteins this molecule targetss, [input_question)
Listing 2: The prompt for prompting language models to generate drug molecule guided by user inputs natural language.
prompt """The drug may targeting the proteins:
(protein_names)
(text_guidance)
Generate the most possible SMILES structure this drug.
H ADDITIONAL RELATED WORKS
Knowledge Graph Learning. Knowledge graph embedding learning algorithms were proposed for KG completion tasks (Han et al., 2018), i.e., given the head node and relation, predict the most prob- able tail node. These approaches assume a fixed set of nodes and not handle inductive prediction, i... prediction for nodes not seen in the training. More recent efforts utilize graph neural networks to handle inductive setup (Teru et al., 2020; Liu et al., 2021), which generate embeddings for new nodes by aggregating its neighbors' embeddings. Nevertheless, they still do not handle isolated new nodes with no link to known nodes. Also, the inherent node properties, e.g., the sequence structure of protein nodes, were usually neglected in the encoding. On the contrary, the proposed BicBRIDGE learns from KG. representing new nodes with base FMs and transforming the embeddings via bridge modules without needing neighbor graph inputs.
Multimodal Learning. Early works in multi-modal learning focus on cross-modal retrieval, such as based on Canonical Correlation Analysis (CCA) to learn a common representation space for differ- ent modalities (Rasiwasia et al., 2010). In the era of deep learning, researchers turned to fuel deep models with large cross-modal data, such as text and image, get better performances (Radford et al., 2021). Multimodal learning was also introduced to knowledge graph embeddings, such as in (Xie et al., 2019; Housselly-Sergiee et al., 2018; Lu et al., 2022).z However, these methods primar- ily focus on visual and textual modalities, which may not directly suit our specific application needs. critical limitation these methods their confinement to predicting nodes within the training KG during the testing phase because these methods all assume the KG is fixed. In contrast, BioBridge stands out by leveraging base unimodal Feature Models (FMs) to process inputs from any modality present in the KG. It can then convert these inputs into any desired target modality.
22

---

Published as a conference paper at ICLR 2024
I TRAINING SETUPS
In this paper, we aim to prove that bridging uni-modal FMs is feasible. As such, we deliberately choose simple architectures, like transformers, to transform input embeddings and original InfoNCE loss to learn the transformation modules. All are kept to a minimum but necessary to make Bio- Bridge work.
We did experiments to check the sensitivity of BioBridge w.r.t. hyperparameters; we chose the batch size be (512, 1024, 4096), and training epochs to be {10, 50, 100). We found the performance was not significantly different for the tried batch size. The method turned out to be converging within 50 epochs, and training with more epochs does not lead further improvement. As such, we keep the same set of hyperparameters for BioBridge across all experiments: batch size 4096, training epochs 50, and learning rate le-4.
In our experiments, we also did the ablations for the bridge module: Variant I: in Eq. (l), removing the residual connection, i.e., using
1b(2, (5, Variant 2: Eq. (1), using RotatE transfor- mation, i.c., using h (22, C22 cy. r). In our experiments, Variant failed to converge; Variant 2 obtained worse performance than the additive transformation in Eq. (1).
23


Reviews:
[{"review_id": "iraaGuFLMG", "reviewer": "Reviewer_cgGU", "summary": "This submission tackles the problem of learning large multi modal ML models without requiring the pairwise cross modal dataset (as it is infeasible in where no of models >2). unlike recent work that aligns all modalities to a single modality, this submission takes a different approach of learning cross modal alignment transformation in embedding space while keeping the underlying unimodal fixed/frozen. Given Given an input embedding that was encoded by a unimodal model, the proposed submission transforms it to the embedding space of the target modality accounting for their relations. the cross modal alignment transformation is parametrized by a vanilla transformer module and learned with a contrastive loss. The proposed method is evaluated on several benchmarks/tasks such as protein protein interaction, protein-phenotype matching, cross modal retrieval where it outperforms several baselines.", "questions": "Could the submission place the main assumptions and theorem from appendix into the main text? The appendix could include the proof.  I imagine this will make main paper more self contained and detailed. In the current form, methodology section is quite short.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the problem of aligning large unimodal models efficiently is very relevant in general and even more so when the modalities are proteins, drugs and diseases (focus of this submission) as it opens up plethora of clinical applications.\n\n- the problem is well motivated in introduction and contextualised. The paper is clearly written and easy to follow except one section (see below). \n\n- the idea of aligning the different embedding space of unimodal pretrained models with a cross modal transformation is sensible and simple. \n\n- experimental validation: evaluation of proposed s convincing on several benchmarks where the proposed method outperforms several baselines and in some cases is the only applicable solution. evaluation and applicability of the proposed method on cross modal retrival and gene-phenotype matching are quite interesting.", "weaknesses": "- Presentation of Related work : Currently the submission only has one paragraph on knowledge graph learning and barely describes the embedding alignment literature e.g. in the context of cross modal retrieval (one of the application of proposed method), one can also mention deep CCA literature as Canonical correlation analysis (CCA) is the core of many cross modal retrieval methods.  \n\n- Presentation of Methodology:  the submission should motivate the solution somewhat intuitively. Section 3.2 on encoding and transformation is very to the point and concise. the proposed methodology could be motivated better e..g by contextualizing wrt some prior work on KGE literature. Although there is a paragraph in the related work on KGE completion, the proposed method is not contexualized. Similarly, It is not very clear to me what parameters are optimized with contrastive loss since the submission keeps the pretrained model frozen. \n\n- Parameterization of alignment transformation as a transformer : the submission should also somehow motivate this choice wrt other options starting with simplest one such as a MLP or a vanilla autoencoder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z0oRHoQJ6f", "reviewer": "Reviewer_VYFb", "summary": "This paper introduces BioBridge, a novel framework for training across modalities (entities) that bridges independently trained uni-task models to establish cross-task abilities in biomedical domains. BioBridge employs contrastive learning to align entity representations and to facilitate learning of transformations between them. The resulting model demonstrates strong performance relative to several baseline knowledge graph (KG) embedding methods in retrieval tasks and highlights potential applications in the guided discovery of new drugs. Overall, BioBridge represents a promising approach for the integration of biomedical data resources and the enhancement of performance in various downstream tasks.", "questions": "- Can the proposed method be applied to other types of downstream tasks, such as image captioning and visual question answering? If applicable, could the authors provide empirical results and case studies?\n- Could the author offer a more detailed explanation of the compared baselines? Specifically, how are they trained, and can they also learn from an external biomedical knowledge graph to ensure a fair comparison?\n- Would employing different contrastive learning objectives, such as SimCLR or MoCo, in place of InfoNCE, impact the performance?\n- Also, please refer to weaknesses for other concerns.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper proposes a novel concept for learning across modalities via the bridging of knowledge graphs.\n- The authors have conducted extensive experiments on various types of entity mapping and numerous approaches to tail entity prediction.\n- With only the bridge module requiring updates during training, and all base feature models (FMs) remaining fixed, the proposed method is computationally efficient.\n- Overall, the paper is well-written, with clear explanations of the methodology and empirical results.", "weaknesses": "- The term \"multimodal\" as mentioned in this paper is confined to different types of biomedical entities. While the authors compare their work with \"ImageBind,\" the experimental section lacks tasks that bridge text and image modalities, which are more complex and crucial for multimodal foundation models.\n- The learning process is guided by knowledge graphs, limiting the scope of \"modalities\" to those represented within biomedical knowledge graphs. Therefore, instead of the broad term \"biomedical foundation model,\" it would be more accurate to describe it as a \"biomedical knowledge graph foundation model.\"\n- The paper does not present ablation studies, such as evaluations of the contrastive learning objectives or hyper-parameter tuning.\n- The case study focusing on molecule generation is intriguing. Quantitative assessments of generation performance would be beneficial, for instance, by making direct comparisons with general-domain foundation models, or by offering more qualitative examples to demonstrate the efficacy of the proposed framework.\n- The baseline comparisons are predominantly with knowledge graph link prediction methods. It is unclear whether the observed advantages stem from the effective transformation learning of the proposed method or from the knowledge supervision of the biomedical knowledge graph.\n- This paper omits a discussion on limitations and potential failure modes. The authors are strongly encouraged to offer deeper insights into the generalizability of BioBridge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jbw3aUEKib", "reviewer": "Reviewer_xFRj", "summary": "This paper proposes BioBRIDGE to bridge unimodal FMs, keeping each FM fixed. BioBRIDGE utilizes a multi-modal knowledge graph to learn cross-modal relationships of entities. Compared to multi-modal FMs and ImageBind, BioBRIDGE is parameter efficient. The experiments on cross-modal retrieval tasks demonstrate the effectivness of BioBRIDGE. The paper also shows that BioBRIDGE has good generalization ability.", "questions": "1. What are the contributions of the contrastive learning?\n2. How about the influence of the knowledge graph on this method? From your perspective, what could be the challenges if BioBRIDGE is adapted to other domains?\n3. For the cross-modality retrieval tasks, are the baselines trained on single modality or multiple modalities? Why not include multi-modal embedding methods published recently?\n\n    a. Lu X, Wang L, Jiang Z, et al. MMKRL: A robust embedding approach for multi-modal knowledge graph representation learning[J]. Applied Intelligence, 2022: 1-18.\n\n    b. Cao X, Shi Y, Wang J, et al. Cross-modal knowledge graph contrastive learning for machine learning method recommendation[C]//Proceedings of the 30th ACM International Conference on Multimedia. 2022: 3694-3702.\n\n    c. Zhu J, Huang C, De Meo P. DFMKE: A dual fusion multi-modal knowledge graph embedding framework for entity alignment[J]. Information Fusion, 2023, 90: 111-119.\n4. Though it is easier to collect unimodal data than to collect paired data from two modalities, could the model trained on the paired data perform better or competitively to model trained on the large unimodal data?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of bridging several unimodal FMs is novel. \n2. The paper is well written and easy to follow.\n3. Compared to existing studies, BioBRIDGE keeps all unimodal FMs fixed and thus is parameter efficient.", "weaknesses": "1. The reasons of using contrastive learning is not clear. It would be better to provide further explanation and experimental supports.\n2. The baselines in Section 4.1 are several years ago. No recent studies are included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qIUJCbMdPN", "reviewer": "Reviewer_L5qe", "summary": "The paper proposes to bridge multiple biological modalities through KG. The motivation is that most of the data are uni-modal, paired data is scarce and due to the combinatorial explosion, infeasible to collect and train a multi-modal model. The author learns a transformation layer among the uni-modal representations to achieve multi-modal learning.", "questions": "- Looking at equation 1, the transformation seems dependent on the (1) target modality node type and (2) relation type and (3) individual node in the target modality. For example, given the same node drug, it could have one transformed representation for protein A, a different one for protein B, and a different one for disease, etc. Is that right? Could the authors describe the design choice of this? why not map all of them into a single unified space? This brings a separate question: how do you conduct a similarity search if the embedding is dependent on individual node?\n- The transformation loss function (eq.2) makes intuitive sense, but there seem to be numerous other options to achieve similar goals such as self-supervised link prediction. Have the authors experimented with other transformation techniques? if so, could the authors provide any additional information? if not, could the authors describe the motivation on why choosing this particular approach? \n- The author uses a transformer model to achieve the transformation. It seems unnecessary since it is just encoding 4 embeddings? Have the authors experimented with other simpler approaches? \n- How are the negative samples created? \n- For individual application, is the model trained on every possible bridge transformation or is the model different for each application and individual task?\n- Cross-modality retrieval seems to be exactly link prediction. In that case, there are numerous approaches for GNN-based link prediction that is missing as a baseline and has shown much better performance compared to the KG embedding methods. Have the authors compared to any of the latest link prediction method? \n- For semantic similarity task, I worry that there is data leakage. Since the protein node is connected to these GO terms in the PrimeKG, the embedding already implicitly knows the labels during training. Have the authors addressed this concern by conducting some holdout protein-go links?\n- For PPI, since it is the same modality between the head and tail nodes, why do we need bridge module? \n- For cross-species task, why is phenotype not available? it seems to be available in PrimeKG? \n- For the multi-modal generation, it is an interesting application, but have the authors checked if the retrieved list is novel predictions or existing links? It will be great to compare with a baseline that is just retrieving the top K entities in the KG and show the difference on answers. \n\n\n\nI am happy to raise score if the authors address my questions.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Interesting problem setting on bridging multi-modal FMs in biology. Makes intuitive sense.\n- Interesting approach on the bridge module\n- Lots of interesting applications and analysis", "weaknesses": "- The method proposes several new modules that lack motivation and seem ad hoc.\n- The clarity of the technical methods of the paper can be improved", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)