PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Authors: Alejandro Velez-Arce
Marinka Zitnik
Venue: ICML 2025 poster (2025)
Keywords: biological chemistry
biomedical AI
computational biology
biomedical informatics
machine learning platform
machine learning framework
AI for drug discovery
Primary area: applications->health_medicine
TL;DR: PyTDC is a machine learning platform for training, evaluation, and inference on multimodal biomedical AI models, integrating single-cell data, benchmarks, and a model server to enable context-aware model development for therapeutic discovery.

Abstract: Existing biomedical benchmarks do not provide end-to-end infrastructure for training, evaluation, and inference of models that integrate multimodal biological data and a broad range of machine learning tasks in therapeutics. We present PyTDC, an open-source machine-learning platform providing streamlined training, evaluation, and inference software for multimodal biological AI models. PyTDC unifies distributed, heterogeneous, continuously updated data sources and model weights and standardizes benchmarking and inference endpoints. This paper discusses the components of PyTDC's architecture and, to our knowledge, the first-of-its-kind case study on the introduced single-cell drug-target nomination ML task. We find state-of-the-art methods in graph representation learning and domain-specific methods from graph theory perform poorly on this task. Though we find a context-aware geometric deep learning method that outperforms the evaluated SoTA and domain-specific baseline methods, the model is unable to generalize to unseen cell types or incorporate additional modalities, highlighting PyTDC's capacity to facilitate an exciting avenue of research developing multimodal, context-aware, foundation models for open problems in biomedical AI.

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Alejandro Velez-Arce
Marinka Zitnik 3456
Abstract
1. Introduction
The application of transfer learning in biology has gained momentum, marking a pivotal shift toward harnessing the wealth of biological measurements available today. Founda- tion models pre-trained on structural, genomic, and network biology data have demonstrated remarkable potential in improving predictions in data-limited scenarios and accel- erating biological discoveries (Theodoris, 2024). These models capture fundamental patterns in biological systems, creating a foundation for tasks ranging from disease mod- eling to therapeutic development. As this field advances, the need for robust strategies to benchmark these models becomes increasingly important. (Liu et al., 2023) illustrate the importance of diverse downstream task evaluations in assessing the utility of network biology foundation models. However, to be useful in therapeutics, benchmarking must also align with biomedical goals. Focusing benchmarks solely canonical biology tasks such cell type clus- tering and annotation neglects deeper biological insights, such as understanding disease mechanisms. Therefore, de- veloping benchmarks prioritizing therapeutic relevance is essential to guide the evolution of models and ensure they address the challenges the heart of biomedical research.
Existing biomedical benchmarks do not provide end-to-end infrastructure for training, evaluation. and inference of models that integrate multimodal biological data and a broad range of machine learning tasks in therapeutics. We present PyTSC an open-source machine-learning platform pro- viding streamlined training, evaluation, and in- ference software for multimodal biological Al models. PyTDC unifies distributed, heteroge- neous, continuously updated data sources and model weights and standardizes benchmarking and inference endpoints. This paper discusses the components of PyTDC's architecture and, to our knowledge, the first-of-its-kind case study the introduced single-cell drug-target nomination ML task. We find state-of-the-ate methods in graph representation learning and domain-specilf meth- ods from graph theory perform poorly on this task. Though we find a context-aware geometric deep learning method that outperforms the evaluated SoTA and domain-specific baseline methods, the model is unable to generalize to unseen cell types or incorporate additional modalities, highlight- ing PyTDC's capacity to facilitate an exciting av- enue of research developing multimodal, context- aware, foundation models for open problems in biomedical AI.
Single-cell genomics has enabled the study of cellular pro- cesses with remarkable resolution, offering insights into cellular heterogeneity and dynamics (Luecken et al., 2023). Progress in data generation and computational methods de- signed for single-cell analysis (CZI Single-Cell Biology, et al., 2023) has facilitated context-aware machine learning models that incorporate cell cell-type ious therapeutic areas (Chen et al., 2024). Despite these advances, there remains need for datasets, benchmarks, and tools for reproducible development and benchmarking of out-of- generalization (OOD) (Ektefaic et al., 2024) and domain-specific metric (Velez-Arce et al., 2024) performance of methods that integrate single-cell analysis with diverse therapeutic approaches. Given key problems in genomics intrinsically involve multiple modalities and adapting general-purpose sequence models for these tasks remains unclear (Garau-Luis et al., 2024), any system facil- itating method development in single-cell therapeutics must incorporate numerous biomedical modalities.
data across var-
AArellAI Inc. Ccalculus.house, The Residency Inc. P Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA *Broad Institute of MIT and Harvard, Cam- bridge, MA, USA 'Harvard Data Science Initiative, Harvard University, Cambridge, MA, USA "Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard Univer- sity, Cambridge, MA, USA. Correspondence to: Alejandro Velez-Arce <alejandro. a @ @mmma...omm,, Marinka Zit- nik <marinka@ hms edd>..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Kahn et al. (2022) shows us that when domain-specifie de- mands machine learning solutions increase, framework innovation is often the catalyst for accelerated scientific research by prioritizing open, modular, customizable inter- nals and state-of-the-arr research-ready models and training setups across a variety of domains. As such, we've devel- oped the first (table 1) platform facilitating the curation of multimodal single-cell data, retrieval and inference of context-aware biomedical representation learning models, and therapeutic-task-speeeee marking. We present PyTDC (figure 1), machine-learingg platform providing streamlined training, evaluation, and in- ference software for single- cell biological foundation mod- els. PyTDC introduces an API-first architecture (Beaulieu et al., 2022) that unifies heterogeneous, continuously up- dated data sources. The platform introduces a model server (figure 2). which provides unified access continuously up- dated model weights across distributed repositories and stan- dardized inference endpoints. Building upon Therapeutic Data Commons (Huang et al., 2021; 2022; Velez-Arce et al., 2024), we present single-cell therapeutics tasks, datasets, and benchmarks for model development and evaluation on OOD predictions and domain-specific metries.
troscopy (Brown et al., 2023; Ye et al., 2022), 3D struc- tural protein data (Meslamani et al., 2011; Mysinger et al., 2012; Liu et al., 2015), cell-type-specific protein- protein interaction networks at single-cell resolution (Li et al., 2024), and biomedical knowledge graphs (Chan- dak et al., 2023). The novel API-first (Beaulieu et al., 2022) (section 3.1) architecture provides abstractions for complex data processing pipelines (Huang et al., 2021), data lineage (Thiago et al., 2020), and versioning (van der Weide et al., 2017). It augments the stability of emerging biomedical AI workflows, based on advancements from (Schick et al., 2023; Patil et al., 2023; M. Bran et al., 2024), with continuous data updates (Liu et al., 2022). More details in section 3.1 and B.
method fine-tuning and bench-
3. We have released open source model retrieval and deployment software that streamlines AI inferenc- ing, downstream fine-tuning, and domain-specific evaluation for representation learning models across biomedical modalities. The PyTDC model server (sec- tions 3.2 and C and figures and 1) facilitates the use and evaluation of biomedical foundation models for downstream therapeutic tasks and enforces alignment between new method development and therapeutic ob- jectives. We have released support for inference on sev- cral state-of-the-an (SoTA) models (Lopez et al., 2018; Theodoris et al., 2023; Cui et al., 2024; Li et al., 2024; ESM Team, 2024) (section C) as well as benchmark- ing software for domain-specific evaluation (Velez-Arce et al., 2024).
Overall, the contributions can be summarized as (high- lighted in Table 1):
1. We integrate single-cell analysis with multimodal ma- chine learning in therapeutics via the introduction of contextualized tasks. PyTDC presents datasets, models, and benchmarks for: single-cell drug-target nomination (Li et al., 2024; Velez-Arce et al., 2024) (section 4, 4.3, and A.1), single-cell chemical and genetic perturba- tion response predictions (Hetzel et al., 2022; Roohani et al., 2023) (section A.2), and cell-type-specific protein- peptide interaction prediction (Grazioli et al., 2022a; Brown et al., 2023) (section A.3). In our benchmarks we measure: the ability of models perform at most predic- tive biological contexts (Kwon et al., 2024), perturbation response prediction performance at out-of-distribution samples (Piran et al., 2023), and model robustness to negative sampling heuristics for distriibbtii pre- dictions (Kim et al., 2023).
In this work (section 4), we present a case study on single- cell Arugetarget nomination (Li et al., 2024; Velez-Arce et al., 2024). PyTDC facilitates our analysis showing inade- quate performance of domain-specific methods and state-of- the-art (SoTA) ceneral-purposse graph representation learn- ing methods. Using a novel and adaptable framework for contextualized metrics (sections 3.3 and 4.2), we measure therapeutic relevance of these methods by their predictions at most predictive cell types (Zhang et al., 2022; Kwon et al., 2024) and find superior performance by (Li et al., 2024) on these domain-specifee metries. Though promising result, the inability of (Li et al., 2024) to generalize to unseen cell types and incorporate additional modalities leaves this as an exciting machine learning research task in therapeutics for the PyTDC-enabled development of methods leveraging foundation models and transfer learning.
2. We introduce a collection of continually updated het- erogeneous biomedical data sources enabled via a novel architecture. PyTDC's modalities span but are not limited to: single-cell gene expression atlases (CZI
Single-Cell Biology, al., 2023; Jones et al., 2022),
2. Related Work
single-cell chemical (Aissa et al., 2021; Klein, 2023)
and genetic (Norman et al., 2019; Replogle et al., 2022)
Datasets, benchmarks, and platforms. Numerous datasets, benchmarks, and platforms have been released supporting training and evaluation of predictive models on therapeu- tic tasks (Wu et al., 2018; Huang et al., 2021; Xu et al., 2022; Huang et al., 2022; Zhu et al., 2022; Li et al., 2022).
perturbations (Peidli et al., 2024), clinical trial outcome data (Fu et al., 2022), peptide sequence data (Grazi- oli et al., 2022b; Gao et al., 2023), peptidomimet- ics protein-peptide interaction data from AS-MS spec-
2

---

PYTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Table Assessment SoTA datasets, benchmarks, and ML platforms Focusing on single-cell therapeutics, data modalities, and model inferencing support, we compare PyTDC's capabilities, including features developed by the (Velez-Arce et al., 2024) team, with SoTA platforms (Wu et al., 2018; Huang et al., 2021: Zhu et al., 2022: CZ1 Single- Cell Biology. al., 2023; Luecken et al., 2023; Peidli et al., 2024). We find PyTDC outperforms in its coverage of single-cell therapeutic tasks (sections 4 and A) and data modalities (sections 3.1 and B) and support novel inferencing capabilities for representation learning models (sections 3.2 and C) and domain-spectt speciifc (section 3.3) evaluation of transfer learning methods.
Feature
TDC
TorchDrug
MoleculeNett
OpenProbleens
scPerturb
CELLXGENE
PYΤDC
Single-cat cell Target identification Task
X
K
K
CRISPR-Ce Perturbation Response Prediction
Cell Perturbation Response Prediction
K
K
Single-cell Protcin- Proicin-Peptide Binding Interaction Prediction
X
X
X
X
Structure-Bal Based Drug Design
Clinical Trial Outcome Prediction
K
K
Knowledge Graphs
X
V
X
X
Python API
Single-cell Gene Expression Atlases
Y
External API data retricval
X
X
X
Data Views
X
X
Pre-computed representation learning embedding retrieval
Al Inferencing for truined represenlation learning models Al Inferencing for pre-trained transformer models
X
There have also been more recent releases of benchmarks for canonical single-cell biology ML tasks (CZI Single-Cell Biology, et al., 2023; Luecken et al., 2023), including rep- resentation learning models in single-cell biology (Initiative, 2023). None of these systems are tailored meaningful evaluation of representation learning methods (Lopez al., 2018; Cui et al., 2024; Theodoris et al., 2023; Chen et al., 2024; Li et al., 2024) at therapeutic tasks (Velez-Arce et al., 2024). PyTDC has been developed with the datasets, ar- chitecture, and model training, evaluation, and inference capabilities detailed table 1, figures 2 and I, and section 3, to address the need for multimodal machine learning platform for the development of biomedical methods lever- aging transfer learning (Theodoris, 2024).
3. Streamlining training, evaluation, and inference
3.1. Multimodal Single-cell Data Retrieval
A collection of multimodal continually updated heteroge- neous data sources is unified under the introduced "API- first-dataset" architecture. Inspired by API-first design (Beaulicu al., 2022), this microservice architecture im- plemented using the model-view-controlle (Bucanek, 2009; Malik et al., 2021) to enable multimodal data views (Churi et al., 2016; Emmerik et al., 1993; Ram- merstorfer Mössenböck, 2003) under a domain-specific- language (Membarth et al., 2016). We discuss further details and data sources for the modalities section 1 and tasks in table 1.
design pattern
Therapeutic and single-cell foundation models. Foun- dation models trained on Therapeutics Data Commons (Huang et al., 2021; 2022; Velez-Arce et al., 2024) have been shown to generalize across several therapeutic tasks (Zam- brano Chaves et al., 2024). Additionally, parallel efforts in training foundation models large single-cell atlases have shown potential advance cell type annotation and match- ing of healthy-diseass cells to study cellular signatures of disease (Yang et al., 2022; Hao et al., 2024; Cui et al., 2024). PyTDC bridges these efforts by providing formal definitions of therapeutic tasks, datasets, and benchmarks at single-cell resolution in order to provide precise therapeutic predictions incorporating cell type contexts (Kwon et al., 2024: al., 2024) and developing machine learning infrastructure for streamlining transfer learning method (Theodoris, 2024) development, evaluation, and inference.
3.2. PYTDC Model Server
PYTDC presents open source model retrieval and deploy- ment software that streamlines AI infereneing and exposes state-of-the-art research-ready models and training setups for biomedical representation learning models across modal- ities. In section C and figure 2, we describe the architectural components of the model server and its integration into the model development lifecycle with the TDC platform. In figure 1, we show the components of PyTDC enabling the full development lifecycle for transfer learning ML methods in therapeutics. Figure 3 also shares how workflow span- ning sottredd-tt--huusadde of lines of code is simplified to below 30 with PyTDC's model server and data retrieval capabilities.
3.3. Context-specific metrics
In machine learning applications, data subsets can corre- spond to critical outcomes. In therapeutics, there is evidence
3

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Retrieve contextualized ML models
Curate multimodal single-cell data
Fine-tune and evaluate
Repository
THERAPEUTIES
TOC2 Model Server
DATA SOMMONE
TDC Made!
Loode
Endpoirts
Fonction:
From www.:liieeellsseleee.. Inport CellKGene
datalonder (ellXGeneenmee"" TTabll Sapiens Al1 cells")
York
from túc import tdc_hf_inn interface
datalonder-get, data(
tdc_hf
value_ welue_filters's filter='tissue 'brain' and sex
[if__intrracee"eee yarttnns-tteentivee""))
from tdc. www.bbennn__eeeeeee import scdt:L_rroup
'sale"
deeppurpose rodel from this repo
group proti__proppsSe SCDTIIGrpu))
4p_nodel
train_val group, iep.gge.. get__tnain_valid_spl..... cst group.get_ "et_tte e train model and test the
# (gst()e))
rde_hf. load (appurrpose"-ddaata
tdc_hf. wwc..h..rredict.ee pro.ccccteeepppppeseeeeee (ewppppooseeppppooooooo (YYOUR
Fros edr..ultii prt.buuuttturrbotcccooo Imsport
SMILES STRINE"])
group.en (ippevaalutte(reds)
Parturtoutcase
fine-tune for some defined splits
data loador Parturboutconee
dp. pp_nodel..triing tradel.trainttriinn val, test)
ranc-"sccortuse sappertub__duu_..ee Alsssrrawweesszyys222111
data data loador. get, data()
import fdcttffainterfcee
from transformers irport BertModel
geneformer toc_hh__interface" (eeeeffrrro))")"" sodel geneformen, lood() assert isinstance Bertrodel), type(redel)
Figure 1. present PyTDC, a leaanin-learning platform providing streamlined training, evaluation, and inference software for single- cell biological foundation models to accelerate research in transfer learning method development in therapeutics (Theodoris, 2024), PyTDC introduces an API-first (Beaulicu et al., 2022) architecture (sections 3.1 and B) that unifies heterogencous, continuously updated data sources. The platform introduces model server, which provides unified access to model weights across distributed repositories and standardized inference endpoints (sections 3.2, B.3.3, and C). The model server accelerates research workflows (Kahn et al., 2022) by exposing oftte-of-th-eeee art, rescarch-ready models and training setups for biomedical representation learning models across modalities. Building upon Therapeutic Data Commons (Huang et al., 2021; 2022; Velez-Are et al., 2024), we present single-cell therapeutics tasks, datasets, and benchmarks for model development and evaluation (sections and A).
that the effects of drugs can vary depending on the type of cell they are targeting and where specific proteins are acting (Zhang et al., 2022). Inspired by the success of candidate Graph Foundation Model architectures (Huang et al., 2023) at subgraph- views for classification tasks (Mao et al., 2024) and building upon the concept of 'slices' (Chen et al., 2020a), we assess model performance on critical biologi- cal subsets by introducing the framework and definitions for contextualized metrics measure model performance at critical biological slices. Our benchmark for single-cell drug-target nomination measures model performance at cell- type-specific subgraphs by computing classification and retrieval metrics aggregated over the top-performing cell types. See Section 4.2 for definitions.
4. Evaluation of graph representation learning models at context-aware drug-target nomination
Identifying disease pathways, which are groups of proteins linked to specific diseases, is crucial area of research that can lead valuable insights for diagnosing, prognosis, and treating diseases. Computational methods facilitate this dis- covery process by utilizing mrotein-proted interaction (PPI) networks. These methods typically begin with a few known disease assooiats proteins and seek to identify additional proteins the pathway by exploring the PPI network sur- rounding these known proteins. However, these approaches have often had limited success, and the reasons for their fail- ures are poorly understood (Agrawal al., 2018). Single- cell analysis provides promising pathway to address this
4

---

PYTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
bAC
GENE SOOO
SC Drug-Target Nomination
Model Store
Task- Fine
Tokenizer Functions
Foundation Models
tuning
Perturbation
Lugenbbe Repreancations
Response
Model
Data Processing Functions
Graph and Generative Models
Tase-fp Evaluation
Dormutionn
SC Protein-Peptide (TCR-Epitope)
Embedding Extraction
Model Server Training & Inference
TDC Benchmarking Modules
TDC DataLoader & Tasks
boc Ieppereticcs
C2 CZ CENE
Pre-computed
Embeddinga
Figure 2. AT inferencing and model evaluation components. The PyTDC, model server (sections 3.2 and C) streamlines retrieval, inferencing, and training setup for an array of context-aware biological foundation models and models spanning multiple modalities. model store retrieval API provides unified Chan-Zuckerberg CELLxGENE Census fine-tuned models, and TDC (Huang et al., 2021; 2022; Velez-Arce et al., 2024) storage. The model also provides access model classes, tokenizer functions, and inference endpoints supporting PyTorch (Paszke al., 2019) and Hugging Face Transformers (Wolf al., 2020). Extracted embeddings, from either model server inference pre-computed embedding storage, are ready for downstream use by task specifcc benchmarking modules.
model weights stored in the Hugging Face Model Hub (https:/huggingface https//uuuggngfaccccoooollooo
by enabling the study gene expression and function at the level of individual cells across healthy and disease states (Jones et al., 2022; CZI Single- Biology, et al., 2023: Kwon et al., 2024). To facilitate biological discoveries using single-cell data, machine-learning models have been devel- oped to capture the complex, cell-type-specific behavior of genes (Lopez et al., 2018; Yang et al., 2022; Theodoris et al., 2023; Cui et al., 2024; Li et al., 2024). PyTDC intro- duces task, dataset, and benchmark for development of contextual models to nominate therapeutic targets cell type-specific manner (Li al., 2024).
tinct roles the disease pathogenesis of conditions like rheumatoid arthritis (RA) and inflammatory bowel diseases (IBD) (Li al., 2024). To our knowledge, work system- atically evaluates state-of-the-arr methods' ability to identify most predictive cell types in drug target nomination.
4.1. Single-cell drug-target nomination formulation
Single-cell drug-target nomination (tdo_ml scDTN) is predictive, non-generative task formalized as learning an estimator for disease-specific function of target protein and cell type outputting whether the candidate protein t is a therapeutic target in that cell type c:
Machine Learning Challenges. Fragmentation of disease and drug pathways when overlaid networks has been stud- ied quite extensively. In fact, an overwhelming majority of pathways don't correspond to well-connected components in PPI networks, leading to poor performance of state-of. the-art disease pathway discovery methods (Agrawal al., 2018). One promising technique in machine learning for omics is identifying the most predictive cell-type contexts for a biomarker (Velez-Arce et al., 2024). This approach can help determine the cell types that play crucial and dis-
= (1.
Target candidate set. proteins, nucleic acids, or other molecules drugs can interact with, producing a therapeutic effect or causing a biological response. The target candidate set is constrained to proteins relevant to the disease being treated. is denoted by:
The target candidate set includes
{l1,.... ..... IN.).
5

---

PYTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Figure 3. The helow code illustrates the integration of PyTDC's single-cell datasets with the model (sections 3.2, C, and B.3.3). Here retrieve the (Peidli et al., 2024; Aissa et al., 2021) chemical perturbation response dataset (section A.2.4, A.2, and A.2.2) and tanuuudd-st
extract gene expression vector embeddings from (Theodoris al., 2023). Such a workflow would take a user hundreds- thousands lines of code to develop. PyTDC allows the user to extract single-cell foundation model embeddings from complex and customized gene expression datasets with than lines of code.
from tdc_ml wtt.:ppreee..eruuuboreeeee from tde_ml import import torch
import Fortuzboutcomo
dataset "scpertuub_drree anug_Aisaaleeevolessseyaaaa
data = Perturboutcome (dataset) import dataset for chemical perturbation response
prediction
adata adata data.adata
tokenizer GeneformerTokeni zer (max_input_size-3) 3 initialize model tokenizer
adata. var "feature_iis i = adata. var Anndx.mapp..
lambda x: tekenizer.co http:/er.ggonenmmmiidddd. get (x, 0)) $ format features using tokenizer
data processing functions
tokenizer..okknizzeeee
vectors (adata, ensembl id= id="feature id", ncount=="count") tokenize custom dataset
cells,
geneformer tdc hf interface ("Geneformer")
model - gene load() $ Toad the Geneformer model
inppt_ttense = torch.tensee (cells)
note you'd typically batch the input tensor
attention_man torch.tensox (llt for t celll for cell batch.)
model (batch,
attontioounas---tttetiinmnmee
states True)
where 4,..., IN. are N, target candidates for the drugs treating the discase. Information modeled for target can- didates can include interaction, structural, and sequence information.
cell-type-specific biological context e E C as input, and the model is tasked to predict:
ÿ = T,ce
(4)
Biological context set. The biological context set includes the -pllsyype-peccffic contexts in which the target candidate set operates. This set is denoted as:
4.2. Context-Specific Metrics in tdc mll f00.mlsscDDNN SCDTN
Context-specific metries (see section 3.3) are defined to measure model performance at critical biological slices, with benchmarks focused measuring cell- type-specific model performance. For single-cell drug- target nomination, we measure model performance at top- performing cell types. We evaluate predictive cell type retrieval using average precision at rank R (AP@R R) and target classification using AUROC, both top predicted cell types. Formally, we define context -pntext-specific AP@ R and AUROC below.
C = {CI, , CN.
where ci..... CN. are N₂ biological contexts which drug- target interactions are being evaluated. Information modeled for cell-type-specific biological contexts can include gene expression and tissue hierarchy. The set is constrained to disease- sppecfcc cell types and tissues.
Drug-target nomination. Drug-Target Nomination binary label y (1, where y 1 indicates the protein candidate therapeutic target. At the same time, means the protein is not such target.
Context-specific AUROC. To calculate the AUROC for the top K performing cell types, we first need to determine which cell types achieve the highest AUROC scores. After selecting the top- cor-perforning cell types, we weigh each top- performing cell type's 's AUROC score by the number of samples in that cell type.
The goal is to train model fe for predicting the probability I E [0, 1] that a protein candidate therapeutic target in a specific cell type. The model learns an estimator for disease-specific function of protein target & E T and
6

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
We denote:
The average precision rank R at Top K cell types can then be defined as:
D = (1. a.), Vi
(5)
CK = f0i, .2,...,,k]] s.t.
Here, D denotes the dataset where x, denotes the feature vector, Yi is the true label, and oj is the cell type for sample i from We further denote C, the set of unique cell types. Then, the AUROC for a specific cell type, AU ROC,, is computed as:
AP@RG 2 APa50y vi K, K
(12)
APo = meen(
Vc, E Ck) (13)
A is robust to varied sizes among cell-type-specific
ROC. = AUROC(D.)
(6)
subgraphs and number of protein targets in those subgraphs (Li et al., 2024; Velez-Arce et al., 2024) and it is used as the retrieval evaluation metric.
Here, De = {(E yilla = of is the subset of the dataset for cell type e and AU represents the AUROC score computed over this subset. Once these are computed, values can be sorted in descending order to select the top X cell type with highest AUROC value.
4.3. Drug-target Nomination Benchmarks
We use curated therapeutic target labels from the Open Tar- gets Platform (Targets, 2023) for rheumatoid arthritis (RA) and inflammatory bowel disease (IBD) (Li al., 2024) (sec- tion A.1.2). Previous work (Velez-Arce al., 2024; Li el., 2024) benchmarked PINNACLE (Li et al., 2024) trained on cell type specific protein-proteiie interaction networks and graph attention neural network (GAT) (Velickovic et al., 2017) trained on a context-free reference protein- protein interaction network- on the curated therapeutic tar- gets dataset. The results (Appendix tables 4 and 3) showed PINNACLE outperforming GAT when it's performance on contextualized metrics (section 4.2) was compared to GAT's aggregate, context-free, performance. However, this eval- uation does not establish the difficulty of performing well on contextualized metrics for drug-target nomination or mo- tivate the need for stronger machine learning models for this task. To our knowledge, PyTDC the first to make this case, evaluate state-of-the-ar methods at the task, and streamline training, evaluation, and inference in single-cell drug-target nomination.
CK = (2.,,,'... s.t. AUROC 2 AUROC
Vi S K.j > K
The weighted AUROC for the top K cell types is given by weighting each cell type's AUROC by the proportion of its samples relative to the total samples in the top K cell types.
AU RUROCTopK EXecK AUROC, x IDel
ExEcK IDal
This measure represents a balance between representation and performance of the cell types, and primarily evalu- ates the model's classification strengths (Velez-Arce al., 2024).
Context-specific Average Precision at rank R (AP@R). In our study, we compute AP@R R for the top K performing cell types. We denote dataset and samples as above.
As a domain-specifer baseline, we run a label propagation algorithm Aridhi et al. (2017) over the Li et al. (2024) cell type specific PPI by labeling the train and validation sets and propagating to the unlabeled test set via a majority vote among the neighborhood of unlabeled node. We find significant portion of the graph remains unlabeled after convergence, resulting in poor performance of the domain- specific baseline illustrated in table 2. We attribute this result to the fragmentation of RA and IBD pathways when overlaid on PPI networks (Agrawal et al., 2018).
D ((1. ca),, V
(9)
Here, D denotes the dataset where x; denotes the feature vector, yı is the true label, and o is the cell type for sample i from S. We further denote C, the set of unique cell types. The samples of each cell type, De (Mii, yi)(e; = c, can be sorted based on the score output by the model for said sample f fx..), with average precision at rank type computed accordingly.
DR. = (21,..., s.t. f(x;) 2 f(zz),
For comparison, we evaluate a state-of-the-eee architecture for representation learning with graphs, Brody et al. (2022). which incorporates attention function. In addition, we evaluate an attention-free architecture for representation learning with graphs, Grover & Leskovec (2016). In both cases, we one-hot encode cell type labels to the context- free PPI graph used to evaluate GAT in Velez-Arce et al. (2024); Li et al. (2024).
(10)
vis R,j> > R24 = C,G =
AP@R AP((y)..... (R).((()))).....)())))))(()))))) (f(x))...........) f(x)))
4; E D
(11)
7

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Table Cell-type specific target nomination for two therapeutic areas, rheumatoid arthritis (RA) and inflammatory bowel diseases (IBD). Cell-type specific context metrics (see 3.3 and definitions in Section 4.2) are used to benchmark (Grover & Leskovec, 2016; Aridhi al., 2017; Brody et al., 2022; Li et al., 2024). Average precision for ranks and 20 for top 20 and cell types is used. AUROC
top 1, 10, 20, and 50 cell types is used.
Model
A Top-20 CT
AP@2 Top-20 CT
AP@5 T0 CT
AP@20 Top-50 CT
PINNACLE (RA)
0.917
0.625 20041
0.628 #0057
0.452 20033
GATv2 (RA)
2.000
0.2466:005
0.25
0.187
Vede2Vee (RA)
0.585
0.448 20.033
0.476 +0,028
0.383 +00020
Label Propagation (RA)
0.105
0.077=20.00
0.042
0.048
PINNACLE (IBD)
0.884
0.634 20.048
0.582 #00077
0.439 10.030
GATv2 (IBD)
0.348
0.261
0.259
0.189
Node2 Vec (IBD)
0.617
0.417:1000
0.406 +0.070
0,295 0.033
Label Propagation (IBD)
0.268
0.228
0.126:20001
0.134::0.003
Model
AUROC Top-1 CT
AUROC Top-10 CT
AUROC Top-20 CT
AUROC Top-50 CT
PINNACLE (RA)
0.765
0.676:1017
0.65220013
0.604 10.00
GATv2 (RA)
0.5
0.5
0.5
0.5
Vec (RA)
0.766 #0030
0.731 #0.024
0.715 +0,05
0.687 #0.024
Label Propagation (RA)
0.451 +0,003
0.405
0.392
0.364
PINNACLE (IBD)
0.935
0,806
0.757
0.672 20.015
GATv2 (IBD)
0.5
0.5
0.5
0,5
Node2 (IBD)
0.883 20.116
0.703 20,036
0.666 100030
0.610
Label Propagation (IBD)
0.683
0.544
0 1
0.457=20.000
Our results (table 2) show Brody et al. (2022); Grover & Leskovec (2016) extract non-random, informative signals by their outperforming label propagation at retrieval, but both methods are far from optimal. SoTA general ML models and domain-specific methods performed poorly at this study's 's single-cell drug target nomination task, supporting the need for stronger models. Li et al. (2024), which outperformed all methods, is a first step in this direction. Still, it is specifi- cally engineered for cell type-specific PPI networks and can only work with apriori-definet and user-specific contexts. In addition, Li et al. (2024)'s contextualized AUROC was strongest at IBD, where label propagation also performed well (better than (Brody et al., 2022)), but weaker and even inferior to Grover & Leskovee (2016) in RA, where label propagation performed worse than random. This sug- gests ample room for improvement at generalization out of subgraph-based view Mao et al. (2024), and a promis- ing avenue of research in this task for GFMs and other peresentation--earing---- baasd transfer learning methods.
a foundation for training and evaluating machine learning models in therapeutics. Introducing an API-first architecture (Beaulieu al., 2022) facilitates continuous data updates, ensuring that biomedical AI models remain adaptable to discoveries. Moreover, PyTDC's model server (figures 2 and 1) streamlines biomedical foundation models' retrieval, deployment, and fine-tuming theettuning aligning new method develop- ment with woaldwrrld therapeutic objectives. Our case study on single-cell drug-target nomination highlights key chal- lenges in model generalization and domain -ppeii- evalua- tion. Despite advances in graph representation learning and context-aware geometric deep learning, existing methods struggle to generalize across unseen cell types and inte- grate additional biomedical modalities. The introduction of context- specific evaluation metrics within PyTDC provides quantitative framework for assessing model performance in the most predictive biological contexts, addressing critical gaps in therapeutic model benchmarking. These results em- phasize the need for multimodal, context-aware foundation models, a research direction that PyTDC is uniquely posi- tioned to support. By defining open challenges, establishing state-of-the-an baselines, and providing tools for method de- velopment and evaluation, PyTDC fosters more rigorous, reproducible, and therapeutically relevant machine learning ecosystem. PYTDC will accelerate innovation in single-cell AI, drive improvements in domain-specifte generalization, and establish new standards for biomedical machine learn- ing research. We invite the community to contribute to this open-source effort and leverage PyTDC as a scalable, adapt- able, and transparent framework for advancing therapeuties
5. Conclusion
PyTDC represents a significant step in accelearting scien- tific research by developing machine learning infrastruc- ture (Kahn al., 2022) for biomedical AI, particularly in single-cell therapeuties (Theodoris, 2024). By integrating multimodal data retrieval (section 3.1), standardized bench- marking (sections and A), and model inference (sections 3.2 and C) into a unified platform, PDTDC enables seamless access to heterogeneous biological datasets and provides
8

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
through AI.
findings that question their effectiveness for this problem. A recent study revealed that deep learning models do not con- sistently outperform simpler linear models across various benchmarks (Ahlmann-Elie et al., 2024). PyTDC datasets, benchmarks, metrics, and foundation model tooling lay the foundation for more thorough study, development, and eval- uation of models this space.
Software and Data
PyTDC is open-source at https:/github..cm/// PyTDC and documentation is provided via https:/ppytdc. apliko.io. In addition, notebooks used to perform the benchmarking discussed in this paper can be found at benchmark.
xyz/
References
Agrawal, M., Zitnik, M., and Leskovec, Large-scale analysis of disease pathways in the human interactome. In Pacific Symposium on Biocomputing 2018, pp. 111- 122. World Scientific, 2018
Acknowledgements
We thank Jesus Caraballo Anaya, a member of the PyTDC team, for his contributions to the technologies discussed in this manuscript. We thank the TDC team for collecting useful datasets.
Agrawal P., Gopalan, V., and Hannenhalli, S. Predicting gene expression changes upon epigenomic drug treatment. bioRxiv, 2023. doi: http:/22222.....5449999 URL http://doi.org/// 101/1100002222222222 20.599955.
We gratefully acknowledge the support of NIH RO1- HD108794, NSF CAREER 2339524, US DoD FA8702-15- D-0001, awards from Harvard Data Science Initiative, Ama- zon Faculty Research, Google Research Scholar Program, AstraZeneca Research, Roche Alliance with Distinguished Scientists, Sanofi iDEA-iTECH Award, Pfizer Research, Chan Zuckerberg Initiative, John and Virginia Kaneb Fel- lowship award Harvard Medical School, Aligning Science Across Parkinson's ' (ASAP) Initiative, Biswas Computa- tional Biology Initiative in partnership with the Milken In- 's Innovation Awards for the Use of Artificial Intelligence, and Kempner Insti- tute for the Study of Natural and Artificial Intelligence at Harvard University. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funders.
Ahlmann-Eltze, C., Huber, W., and Anders, S. Deep learning-based predictions of gene perturbation ef- fects do not yet outperform simple linear methods. bioRxiv, 2024. doi: 10. 11110/20244099.66613333 URL 2024.09. 16.613342.
stitute, Harvard Medical School Dean's
Aissa, A. F., Islam, A. B. M. K., Ariss, M. M., Go, C. C., Rader, A. E., Conrardy, R. D., Gajda, A. M., Rubio- Perez, C., Valyi-Nagy, K., Pasquinelli, M., Feldman, L. E., Green, S. J., Lopez-Bigas, N., Frolov, M. V., and Benevolenskaya, E. V. Single-cell transcriptional changes associated with drug tolerance and response to combina- tion therapies in cancer. Nature Communications, 12(1): 1628, 2021. doi: 10. 10.10330/411477--21-217777 https://doi.org/1000 101111113/114777211111 http://ooo.ooo0.1110000614466660
URL
Impact Statement
Limitations and socictal considerations. PyTDC software open-source and available under MIT license. However, models and datasets exposed by PyTDC released un- der their own licenses, which though exposed, PyTDC is unable to strictly enforce. It's been the case such licenses have been violated by commercial entities developing mod- els on released datasets (Zambrano Chaves et al., 2024). Open-source datasets and benchmark can increase risk of biased or incomplete data, which may lead inaccurate or non-representative AI models. Additionally, the open ac- cessibility of such datasets could lead to misuse, including unethical applications or the proliferation of AI models that reinforce existing biases in medicine. Moreover, reliance on standardized benchmarks may discourage innovation and lead to the over-fitting of models to specific datasets, potentially limiting their generalization in real-world sce- narios. Last, evaluating deep learning models for genetic perturbation tasks requires reconsideration in light of recent
Aridhi, S., Alborzi, S. 7., Smail- M., Devi- gnes, M.-D., and Ritchie, D. Neighborhood- la- bel propagation in large protein graphs. arXiv preprint arXiv: /ao:::/0000774 2017. doi: 10.48550Wrxiv.1700 07074.
Arvind, V., Kim, J. S., Oermann, E., Kaji, D. A., and Cho, S. K. Predicting surgical complications in adult patients undergoing anterior cervical discectomy and fusion using machine learning. Neurospine, 15:329-337, 2018, doi: 10.14245/ns. htt./nss..83.8 1836248.111
Asadi, H., Dowling, R., Yan, B., and Mitchell, P. Machine learning for outcome prediction of acute ischemic stroke post intra-arterial therapy. PLoS ONE, 9:e88225, 2014. doi: 10. how.o//oaall.ooe.00055.
Beacher, F., Mujica- Parood, L., Gupta, S., and Ancora, L. A. Machine learning predicts outcomes of phase iii clinical

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
trials for prostate cancer. Algorithms, 14:147, 2021. doi: (0.3390/a14050144
May 2023. doi: URL 37 371033601 wtn.uuproddnu www.nnnnnneaaadeee www._abiotc.ppicoooooo pewpllarrrpppeeeeliiee License CC BY-NC-ND 4.0.
10226444/chemmxiix-222----ww
Beaulieu, N., Dascalu, S., and Hand, E. Api integrator: A ui design and code automation application supporting api-first design. In Proceedings of the 9th International Conference Applied Computing and Information Tech- nology, 2022. doi: 10. 1145/3544895335499999
Bucanck, J. Learn Objective-CC Cerective-.ooo for Java Developers, chap- ter 20, pp. 353. Apress, 2009. ISBN 978- 978-1-4302-2369. doi: 10. --00-----222222-222222
Bedon, I.., Cecchin, E., Fabbiani, E., Bo, M. D., Buon- adonna, A., Polano, M., and Toffoli, G. Machine learning application in phase clinical trial allows for the iden- tification of clinical-biomoleee biomolecular markers significantly associated with toxicity. Clinical Pharmacology and Therapeutics, 111, 2021. doi: 10. 10110
Cai, M., Bang, S.-J., Zhang, P., and Lee, H. Atm-ter: Ter- epitope binding affinity prediction using multi-head self-attention model. Frontiers in Immunology, 13, 2022. doi: http://mmmum222.88922223
Carlberg, K., Korotkova, M., Larsson, L., Catrina, A., Ståhl, P. L., and Malmström, V. Exploring inflam- matory signatures in arthritic joint biopsies with spa- tial transcriptomics. Scientific Reports, 9, 2019. doi: 10. 01138/41598-019-5555
Beinse, G., Tellier, V., Charvet, V., Deutsch, E.,, Borget, I., Massard, C., Hollebecque, A., and Verlingue, L. Pre- diction of drug approval after phase clinical trials in oncology: Resolved2. JCO Clinical Cancer Informatics, 3:1-10, 2019.
doi: (0.100/CCL.19.0000
Bhasin, M., Lata, S., and Raghava, G. P. S. Tappred prediction of tap-binding peptides in antigens. Meth- ods in Molecular Biology, 409:381-386, 2007. doi:
Chandak, P., Huang, K., and Zitnik, M. Building a knowl- edge graph to enable precision medicine. Scientifie Data, 10(1):67, 2023. URL https://doi.org/10..000 $41597-023-01960-2
Boulanger, D., Eccleston, R. C., Phillips, A., Coveney, P., Elliott, T., and Dalchau, N. A mechanistic model for predicting cell surface presentation of competing peptides by mhe class molecules. Frontiers in Immunology, 9: 1538, 2018. doi: 3.33%//iimm.2018.000
Chen, V. S., Wu, S., Weng, Z., Ratner, A., and Ré, C. Slice-based learning: A programming model for resid- ual learning in critical data slices, 2020a. URL https: hanxiv.org/ab//1909.oo 06349.
Chen, X., Wang, Z., Zitnik, M., Kellis, M.. and Chen, T. Pre- training single-cell language models through genetic pathway learning. Workshop on Accessible and Effi- cient Foundation Models for Biological Discovery, ICML, June 2024. URL httpss//icml..c//ituu////////// cMLBio Poster.
Bowman, E., Banuprakash, S., Nguyen, K.-S., and Marini, M. Machine learning prediction of progression events in oncology recist 1.1 clinical trials. Journal of Clinical On- cology, 2023. doi: 10. www/jco.2023.4..1. suppl.e13589
Ac-
Bradley, Structure-based prediction of cell receptor.peptide--mie interactions. eLife, 12, 2022. doi: 10.1 00.10000022.00.055500000
Chen, Z., Min, M. R., and Ning, X. Ranking- based convo- lutional neural network models for peptide-mhe binding prediction. ArXiv, abs/2012.02840, 2020b.
Brbi, M., Yasunaga, M., Agarwal, P. and Leskovec, J. Predicting drug outcome of population via clin- ical knowledge graph. To be published, 2024. URL http://wwww.iioooott//////////// 1..24303800 Preprint.
Churi, P. P., Wagh, S., Kalelkar, D., and Kalelkar, M. Model- view-controller pattern in bi dashboards: Designing best practices. 2016 3rd International Conference on Com- puting for Sustainable Global Development (INDIACom), pp. 2082-2086, 2016.
Brody, S., Alon, U. and Yahav, E. How attentive are graph attention networks? In International Confer- ence on Learning Representations (ICLR), 2022. URL https//ppennvvieseneefffmumtmddp wtt.e/ffoorrmmmidd//ixmms//
Cock, P. J., Antao. T. Chang, J. T.. Chapman, B. A.. Cox, C. J., Dalke, A., Friedberg, I., Hamelryck, T., Kauff, F. Wilezynski, B., and de Hoon, M. J. L. Biopython: freely available python tools for com- putational molecular biology and bioinformatics. Bioinformatics, 25(11):1422-1422 14233, 2009. URL https: //biopython http:/hhoo.rgg//oos/////////p/pp Entreen..ttatnnddine //hooython..rr/ddc////5555555/// Stt...........mmmm..
Brown, J. S., Mohapatra, S., Lee, M. A., Misteli, R., Tseo, Y., Grob, N. M., Quartararo, A. J., Loas, A.,
R., and Pentelute, B. L. Unsupervised machine learning leads to an abiotic picomolar peptide ligand. ChemRxiv,
10

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Consortium, T. U. Uniprot: the universal protein knowl- edgehase in 2021. Nucleic Acids Research, 49(D1):
Gao, S., Fang, A., Huang, Y., Giunchiglia, V., Noori, A., Schwarz, J. R., Ektefaie, Y., Kondic, J., and Zitnik M. Empowering biomedical discovery with ai agents. Cell- Press, 187, 2024.
D480-D489, 2021. doi: 10. 10.103//ar//kaal 100. URL
https:/wwww.uiiprt.or///eeppppp heades headers.
Cui, H., Wang, C.. Maan, H., Pang, K., Luo, F., Duan, N., and Wang, B. segpt: toward building a foundation model for single-cell multi-omics using generative ai. Nature Methods, pp. 1-11, 2024.
Gao, Y., Gao, Y., and Liu, Q. Pan-peptide meta learn- ing for t-cell receptor- antigen binding recognition. Na- ture Machine Intelligence, ::236-249, 2023. doi: 10. 1038/442356-023--06619---1 URL https://www.ntturs com/aaticles/s42256-0223---000
CZI Single-Cell Biology, et al. Cz cellxgene discover: A single-cell data platform for scalable exploration, analysis and modeling of aggregated data. BioRxiv Preprint, 2023. URL https://doi.orr/110 1Tt.///2.............
Garau-Luis, J. J., Bordes, P., Gonzalez, L.., Roller, M., de Almeida, B. P., Hexemer, L.., Blum, C., Laurent, S., Grzegorzewski, J., Lang, M., Pierrot, T., and Richard, G. Multi-modal transfer learning between biological foun- dation models, 2024. URL https//arssxxorr/abb//200. 14150.
Dai, R., Kannampallil, T., Zhang, J., Lv, N., Ma, J., and Lu, C. Multi-task learning for randomized controlled trials. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 6:1-23, 2022.
Ghadie, M. A., Lambourne, L., Vidal, M., and Xia, Y. Domain-based prediction of the human isoform inter- actome provides insights into the funetional impact of alternative splicing. PLoS Computational Biology, 13, 2017. doi: 1/joun/lj 10057..
doi: 10. 10.1145/353459.
Ektefaie, Y. Shen, A., Bykova, D., Maximillian, M., Zitnik, M., and Farhat, M. Evaluating generalizability of artifi- cial intelligence models for molecular datasets. Nature Machine Intelligence, 2024. URL https://rreu.be//DDD0
Grazioli, F., Machart, P., Mösch, A., Li, K., Castorina, L., Pfeifer, N., and Min, M. R. Attentive variational infor- mation bottleneck for ter-peptide interaction prediction. Bioinformatics, 39, 2022a. doi: 10.1093/boinffommaicc btac820.
Emmerik, M. V., Rappoport, A., and Rossignac, J. Sim- plifying interactive design of solid models: A hypertext approach. The Visual Computer, 9:239 2544, 1993. doi:
10.1007/BF01908847.
Grazioli, F. Mösch, A., Machart, P., Li, K., Alqassem, I., O' O'Donnell, T. J., and Min, M. R. On binding predic- failing to generalize unseen peptides. Frontiers in Immunology, 13:1014255 2022b. doi: 10.3389/fimu.. 2022.1014256 URL 10.3389/fimmu http://wmmmmm........mm 101426//uul..
ESM Team. ESM Cambrian: Revealing the mysteries of proteins with unsupervised learning. EvolutionaryScaal Website, December 2024, URL https:/evoolutionayyccclee hablog/esmmccamrrian. cambrian... Accessed: January 29, 2024.
Fu, T., Huang, K., Xiao, C., Glass, I. M., and Sun, Hint: Hierarchical interaction network for clinical- trial-outcome predictions. Patterns, 3(4):100445, Feb URI. https:
Grover; and Leskovec, J. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discov- ery and Data Mining (KDD), pp. 855-864. ACM, 2016. doi: 10. 10.1145/2336672.29933
2022. doi: 10. p.016//..atterr2222..... //doi.or//10.c 101.0..111/........ pat6//jaateer22222004444 eCollection 2022 Apr 8
Gupta, C., Xu, J., Jin, T., Khullar, S., Liu, X., Alatkar, S., Cheng, F., and Wang, D. Single-cell network bi- ology characterizes cell type gene regulation for drug repurposing and phenotype prediction in alzheimer's dis- ease. PLoS Computational Biology, 18, 2022. doi: 10. 10.1371/journall.p peb. 1010287.
Fujima, N., Shimizu, Y., Yoshida, D., Kano, S., Mizumachi, T., Homma, A., Yasuda, K., Onimaru, R., Sakai, O., Kudo, K., and Shirato, H. Machine learning-based prediction of treatment outcomes using mr imaging-derion derived quanti- tative tumor information in patients with sinonasal squa- mous cell carcinomas: A preliminary study. Cancers, 11: 800, 2019. doi: 10.3390/cancers! Maeesss00000s 1060800.
Hao, M., Gong, J., Zeng, X.. Liu, C., Guo, Y., Cheng, X., Wang. T., Ma, J., Zhang, X., and Song, L. Large-scale foundation model on single-cell transcriptomics. Nature Methods, pp. 1-11, 2024.
Gao, J., Xiao, C., Glass, I.. M., and Sun, J. Compose: Cross- modal pscudo-siamese network for patient trial matching. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. pp- 803-812, 2020.
Hetzel, L., Böhm, S., Kilbertus, N., Günnemann, S., Lotfol- lahi, M., and Theis, F. Predicting cellular responses to

---

M., Almanzar, N., PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
novel drug perturbations a single-cell resolution. arXiv, hns/2204.13545,, 2022. URL https://arrivvorraah/22224 13545.
Ji, Y., Mishra, R. K., and Davuluri, R. In silico anal- ysis of alternative splicing drug-target gene inter- actions. Scientific Reports, 10, 2020. doi: 10.1038/ s4 14159-009-56894-
Hong, D., Fort, D., Shi, L., and Price-Haywood, E. Elec- tronic medical record risk modeling of cardiovascular outcomes among patients with type diabetes. Dia- betes Therapy, 12:2007 2017, 2021. doi: 10.1007/ s13300-021-0109%6
Jiang, Y., Huo, M., and Li, S. C. Teinet: a deep learn- ing framework for prediction of ter-epitope binding specificity. Briefings in bioinformatics, 2023. doi: 10. 10.1093/bibbbbad066.
Hoofnagle, A. N. and Resing. K. A. Proteomics and the analysis of protein phosphorylation. Current Opinion in Biotechnology. 12(6))617-622, 2001.
Jones, R. C.,, Karkanias, J., Krasnow, M., Pisco, A., Quake, S., Salzman, J., Yosef, N., Bulthaup, B., Brown, P., Harper, W., Hemenez, M., Ponnusamy, R., Salehi, A.. Sanagavarapu. B. A., Spallino, E., Aaron, K. A., Concep- cion, W., Gardner, J., Kelly, B., Neidlinger, N., Wang. Z.,, Crasta, S., Kolluru, S., Morri, M., Tan, S. Y., Travaglini, K., Xu, C. A., Micântara-Hernániie
Hu, X. and Liu, S. Deepber:
Deep learning framework for cancer-type classification and binding affinity estimation using b cell receptor repertoires. bioRxiv, 2019. doi: 10.1101/731158
Antony, J., Beyersdorf, B., Burhan, D., Calcuttawala, K., Carter, M. M., Chan, C. K. F., Chang, C. A., Chang, S., Colville, A., Culver, R., Cvijovic, I. D' A'mato, G., Ezran, C., Galdos, F. X., Gillich, A., Goodyer, W., Hang, Y., Hayashi, A., Houshdaran, S., Huang, X., Irwin, J.. Jang, S., Juanico, J. V., Kershner, A. M., Kim, S., Kiss, B., Kong, W., Kumar, M. E., Kuo, A., Leylek, R., Li, B., Loeb, G. B., Lu, W.-.,, Mantri, S., Markovic, M., McAlpine, P. L., de Morrée, A., Mrouj, K., Mukher- jee, S., Muser, T., Neuhöfer, P., Nguyen, T. D., Perez, K., Phansalkar, R., Puluca, N., Qi, Z., Rao, P., Raquer- McKay, H. M., Schaum, N., Scott, B., Seddighzadeh, B., Segal, J., Sen, S., Sikandar, S. S., Spencer, S., Steffes, L., Subramaniam, V., Swarup, A., Swift, M., Van Treuren, W. W., Trimm, E., Veizades, S., Vijayakumar, S., Vo, K. C., Vorperian, S., Wang, W., Weinstein, H. N., Win- kler, J., Wu, T., Xie, J., Yung, A., Zhang, Y., Detweiler, A., Mekonen, H. E., Neff, N., Sit, R., Tan, M., Yan, J.-c., Bean, G., Charu, V., Forgó, E., Martin, B. A., Ozawa, M., Silva, O., Toland, A., Vemuri, V. N. P., Afik, S., Awayan, K., Botvinnik, O., Byrne, A., Chen, M., Dehghannasiri, R., Gayoso, A., Granados, A. A., Li, Q., Mahmoudabadi, G., McGeever, A., Olivieri, J., Park, M., Ravikumar, N., Stanley, G. M., Tan, W., Tarashansky, A. J., Vanheusden, R., Wang, P. L., Wang. S., Xing, G., Culver, R., Deth- lefsen. L.. Ho, P.-y.. Liu, S., Maltzman, J., Metzger, R., Sasagawa, K., Sinha, R., Song, H., Wang. B., Artandi, S., Beachy, P., Clarke, M., Giudice, L.., Huang, F., Huang, K. C., Idoyaga, J., Kim, S. K., Krasnow, M., Kuo, C., Nguyen, P., Rando, T., Red- K., Reiter, J., Relman, D., Sonnenburg. J., Wu, A., Wu, S. M., and Wyss-Corap T. The tabula sapiens: A multiple-organ, single-cell tran- scriptomic atlas of humans. Science, 376, 2022. doi: 10. 1..116//ciencceaab4455
Huang, K., Fu, T., Gao, W., Zhao, Y., Roohami, Y., Leskovec, J., Coley, C. W., Xiao, C., Sun, J., and Zitnik, M. Ther- apeutics data commons: Machine learning datasets and tasks for drug discovery and development, 2021.
Huang, K., Fu, T., Gao, W., Zhao, Y., Roohani, Y., Leskovec, J., Coley, C. W., Xiao, C., Sun, J., and Zitnik, M. Artificial intelligence foundation for therapeutic science. Nature chemical biology, 18(10) (0100)0003-111 1036, 2022.
Huang, Q. Ren, H., Chen, P., Kržmane, G., Zeng, D., Liang P., and Leskovec, J. PRODIGY: Enabling In- context Learning Over Graphs. In Proceeuiiggs of the 37th Conference on Neural Information Processing Systems (NeurIPS), 2023. URL https//arxiv.org/abs/311...1
Huang, S. and Ding, Y. Predicting binding affinity between mhe-i receptor and peptides based on molecular docking and protein-peptide interaction interface characteristics. Letters in Drug Design and Discovery, 2022. doi: 10. 2174/15701808 00000880011166000 19102035.
Initiative, C.
Embedding metries the URL https:
december 2023 Its. 2023.
//chanzuckerbergggihiile jillubioeellleee--enssstaaaaii
(144/202247110embbeedde dee html#
Accessed: 2024-09-14.
Janeway, C. A., Travers, P., Walport, M., and Shlomchik, M. J. Immunobiology: The Immune System in Health and Disease. Garland Science, 2001.
Jensen, M. F. and Nielsen, M. Netter 2.2 improved ter specificity predictions by combining pan- and peptide- specific training strategies, loss-scaling and integration of sequence similarity. bioRxiv, 2023. doi: 10.1 101/2023. 10.12.56.001.
Jourahmad, Z., Habibabadi, J. M., Moein, H., Basiratnia, R., Geranqayeh, A. R., Ghidary, S. S., and Sadegh-Zadeh, S.-A. Machine learning techniques for predicting the
12

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
short-term outcome of resective surgery in lesional-drug resistance epilepsy. ArXiv, abs/2302.10901, 2023.
Li, M. M., Huang, Y., Sumathipala, M., Liang, M. Q.. Valdeolivas, A., Ananthakrishnan, A. N., Marbach, D., and Zitnik, M. Contextual ai models for single-cell protein biology. Nature Methods, 2024. URL htips:
Julkunen, H., Cichońska, A., Gautam, P., Szedmák, S., Douat, J., Pahikkala, T., Aittokallio, T., and Rousu, Leveraging multi-way interactions for systematic prediction of pre-clinical drug combination effects. Nature Communications, 11, 2020. doi: 10.1038/ 291467-020-199500
Li, S., Zhou, J., Xu, T., Huang, L., Wang, F., Xiong, H., Huang, W., Dou, D., and Xiong, H. Structure-awar interactive graph neural networks for the prediction of protein-ligand binding affinity. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2021. doi: 10.1 145/3445548.346731
Kahn, J. D., Pratap, V., Likhomanenko, T., Xu, Q., Han- nun, A., Cai, J., Tomasello, P., Lee, A., Grave, E., Avi- dov, G., Steiner, B., Liptchinsky, V., Synnaeve, G., and Collobert, R. Flashlight: Enabling innovation in tools for machine learning. In Proceedings of the 39th In- ternational Conference on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Re- search, pp. 10557-10574. PMLR, 2022. URL https:
Li, X., Wang, S., Zhang, J., and Zhang. R. Torchprotein: A deep learning library for protein sequence and structure modeling. Bioinformatiis, 38(6))1743- 1745, 2022. doi: 10. 10.1093bionfommaacs//bbba4
Li, Y.. Liu, X., You, Z., Li, L., Guo, J., and Wang, Z. A computational approach for predicting drugetarget inter- actions from protein sequence and drug substructure fin- gerprint information. International Journal of Intelligent Systems, 36:593 609, 2020. doi: 00.10022int.22332
Kalscheur, M. M., Kipp, R., Tattersall, M., Mei, C.,, Buhr, K., DeMets, D., Field, M., Eckhardt, L., and Page, C. D. Machine learning algorithm predicts cardiac resynchro- nization therapy outcomes: Lessons from the companion trial. Circulation: Arrhythmia and Electrophysiology, 11: c005499, 2018. doi: 10. 111111111/EE.III 1 17.005499.
Li, Y.-M., Huang, Y.-A., You, Z., Li, L., and Wang. Z. Drug- target interaction prediction based drug fingerprint information and protein sequence. Molecules, 24, 2019. doi: 10.390//moeeulee244129999
Kim, H. Y., Kim, S., Park, W.-Y., and Kim, D. Tspred: robust prediction framework for ter-epitope interac- tions based on an ensemble deep learning approach us- ing paired chain ter sequence data. bioRxiv, 2023. doi: 10.1101/2023.1 12.3........0000000
Liu, H., P.V.S, A., Patwardhan, S., Grasch, P., and Agarwal, S. Model stability with continuous data updates, 2022.
URL https://arxiv.or//pd//200 2t.../22......0.........
Liu, N. and Salinas, J. Machine learning for predicting outcomes in trauma. SHOCK, 48:504-5106 2017. doi: 10. 2..197/SHK.0000000000000
Klein, D. sciplex dataset. Dataset, 2023. URL https://doi. 0gg/:00.084//m..gghhrrrr.e 146.222............. 1..
Koyaloğlu-Yalgin Z., Lee, J., Niclsen, M., Greenbaum, J., Schoenberger, S., Miller, A. M., Kim, J., Sette, A., and Peters, B. Combined assessment of mhe binding and antigen expression improves cell epitope predictions. bioRxiv, 2020. doi: 010/1//020011.09.37555
Liu, T., Li, K., Wang, Y., Li, H., and Zhao, H. Evaluating the utilities of foundation models in single-cell data analysis. bioRxiv, 2023. doi: 10.1100/2023.00 .9. 00..08..555 55555 URL https:/doi.orr///
Preprint.
Liu, Z., Li, Y., Han, L., Li, J., Liu, J., Zhao, Z. Nie, W., Liu, Y., and Wang, R. Pdb-wide collection of binding data: current status of the pdbbind database. Bioinformutics, 31 (3):405-412, 2015. doi: 10. 1o.1099/boiinfommattsssubb////// URL http:://oiiorr/// http://di..rg/1...11////oiooo//. (0933/ooofoomnaaiss/uu6266
Kuenzi, B., Park, J., Fong, S. H., Sanchez, K. S., Lee, J., Kreisberg, J., Ma, J., and Ideker, T. Predicting drug response and synergy using a deep learning model of human cancer cells. Cancer cell, 2020. doi: 10.1016/j. ccel1.2020.09.014.
Liu, Z., Jin, J.. Cui, Y., Xiong, Z.. Nasiri, A., Zhao, Y., and Hu, J. Deepseqpani: an interpretable recurrent neural network model with attention mechanism for peptide-hla class ii binding prediction. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2021. doi: 10. 10.1109/TCBB. 202.....222292.90
Kwon, J. J., Pan, J., Gonzalez, G., Hahn, W. C., and Zitnik, M. On knowing a gene: A distributional hypothesis of gene function. Cell Systems, 2024. doi: 10.1016/j cels. 2024.04.008.
Lee, I., Keum, J., and Nam, H. Deepconv-dti: Prediction of drug- ttrgetarget interactions via deep learning with convolu- tion on protein sequences. PLoS Computational Biology, 15, 2018. doi: 10. 10. 1os71/jorrall.pb..0000111
Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., and Yosef, N. Deep generative modeling for single-cell transcriptomics. Nature Methods, 15(12):1053-1058,
13

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
2018. doi: 10.0038441192-018-022222 URL https: www..ature..o///arccssss44411222
Membarth, R., Reiche, O., Hannig, F,, Teich, J., Körner, M., and Eckert, W. Hipace: A domain-specifie language and compiler for image processing. IEEE Transactions Parallel and Distributed Systems, 27:210-224 2016. doi: 10.1 1099TTDDS2001522255
Lotfollahi, M., Wolf, F., and Theis, F. J. segen predicts single-cell perturbation responses. Nature Methods, 16:
715-721, 2019. doi: 10.1038.
Meng, F.-.., You, Z.-H., Chen, X., Zhou, Y., and An, J.- Y. Prediction of drug-targe interaction networks from the integration of protein sequences and drug chemical structures. Molecules, 22(7), 2017. ISSN 1420-3049, doi: 10.3390/molccuues2201111 URL https://ww..mdpi. com//1420-3049//27/111
Luecken, M. D., Gigante, S., Burkhardt, D. B., Cannoodt,
R., Strobl, D. C., Markov, N. S., Zappia, L.., Palla, G., Lewis, W. Dimitrov, D., Vinyard, M. E., Magruder, D., Andersson, A.. Dann, E... Qin, Q.. Otto, D. J., Klein, M., Botvinnik, O. B., Deconinck, L., Waldrant, K., Members, O. P. J., Bloom, J. M., Pisco, A. O., Saez-Rodriguez, J., Wulsin, D., Pinello, L.,, Saeys, Y., Theis, F. J., and Krish- naswamy, S. Defining and benchmarking open problems in single-cell analysis. Research Square Preprint, 2023. doi: 10.21203/rs.3... 1rs....33--.441111111
Meslamani, J., Rognan, D., and Kellenberger, E. sc-pdb: database for identifying variations and multiplicity of 'druggable' binding sites in proteins. Bioinformatics, 27 (9): 13:4-321336, 2011.
Lupia, A., Mimmi, S., Iaccino, E., Maisano, D., Moraca, F., Talarico, C., Vecchio, E., Fiume, G., Ortuso, F., Scala, G., Quinto, I., and Alcaro, S. Molecular modelling of epi- topes recognized by neoplastic b lymphocytes in chronic lymphocytic leukemia. European Journal of Medicinal Chemistry, 111838, 2019. doi: 10. 11006//..jmecc..000 111838.
Moris, P., Pauw, J. D., Postovskaya, A., Gielis, S., Neuter, N. D., Bittremieux, W., Ogunjimi, B., Laukens, K., and Meysman, P. Current challenges for unseen-epitope ter interaction prediction and a new perspective derived from image classification. Briefings in Bioinformatics, 22, 2020. doi: 10.1099/bi/bbba331.
Murphy, K. and Weaver, C. Janeway's Immumobiology. Garland Science, 2016.
Lv, H., Yang, X., Wang, B., Wang, S., Du, X., Tan, Q. Hao, Z., Liu, Y., Yan, J., and Xia, Y. Machine learning-dre driven models to predict prognostic outcomes in patients hospi- talized with heart failure using electronic health records: Retrospective study. Journal of Medical Internet Re- search, 23, 2020. doi: 10.2196/24996.
Mysinger, M. M., Carchia, M., Irwin, J. J., and Shoichet, B. K. Directory of useful decoys, enhanced (dud-e): bet- ter ligands and decoys for better benchmarking. Journal of medicinal chemistry, 55(14):6582 6594, 2012.
M. Bran, A., Cox, S., Schilter, O., Baldassari, C., White, A. D., and Schwaller, P. Augmenting large language mod- els with chemistry tools. Nature Machine Intelligence, 2024.
Nair, V., Saleh, R., Toor, S., Taha, R. 7., Ahmed, A., Kurer, M., Murshed, K. A., Nada, M., and Elkord, E. Epigenetic regulation of immune checkpoints and cell exhaustion markers in tumor inmiltriiiiltratigg t cells of colorectal cancer patients. Epigenomics, 12(17):1481-15 2020. doi: 10.22177/ep-2222--027. URL https://doi.or//10.2221 epi-2220-0222
MacKay, E. J., Stubna, M. D., Chivers, C., Draugelis, M., Hanson, W. J., Desai, N. D., and Groeneveld, P. W. Appli- cation of machine learning approaches administrative claims data to predict clinical outcomes in medical and surgical patient populations. PLoS ONE, 16, 2021. doi: 10.137 h133////ummll.ppo//oo wow.eeallppoo../0// 0252558.
Nielsen, M., Lundegaard, C., Lund, O., and Keşmir, C.
The role of the proteasome in generating cytotoxic t-cell epitopes: insights obtained from improved predictions of proteasomal cleavage. Inmunogenetics, 57:33 11133-1,,22 2005, doi: 10. 00107/000001100507788500
Malik, V. K., Pathak, S., Anamika, K., Kaur, A., and Kumar, V. A study of mvc: A software design pattern for web application development on j2ee architecture. Academia.co edu, 2021. URL https://www.academn
Norman, T. M., Horlbeck, M. A., Replogle, J. M., Ge. A. Y.,
edu/5094 11155094163//A STUDY .OF WWVC.A_SOFTTV DESIGN PATTERN FOR WEB APPLICATION DEVELOPMENT ON
Xu, A., Jost, M., Gilbert, L. A., and Weissman, J. S. Exploring genetic interaction manifolds constructed from rich single-cell phenotypes. Science, 365(6455))786-7933 2019. doi: 10.1 126/science.aax4338.
Mao, H., Chen, Z., Tang, W., Zhao, J., Ma, Y., Zhao, T., Shah, N., Galkin, M., and Tang, J. Position: Graph foundation models are already here. In Proceedings of the s1 International Conference on Machine Learning (ICML), 2024. URL https://arii..rr/aa/////0
Özüürk, H., Ozkirimli, E., and Özgür, A. Widedta: pre- diction of drug-target binding affinity. arXiv preprint 19X2. 042.0.16666.0 2019. URL http:://dii.rr/110.8555 1rXiv.1902.044
14

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
O' O'Donnell, T., Rubinsteyn, A., and Laserson, U. Mhhflurr 2.0: Improved pan-allele prediction of mhe class i- presented peptides by incorporating antigen processing. Cell Systems, 11(1):42 ((1):42-48.e7, 2020. doi: 10. 10.1016/j.cels 2020.06.010.
admission with machine learning: Development and val- idation using linked electronic health records. PL.S
Medicine, 15, 2018. doi: 10.137 l/joooraa..pmeeee wwt.lpmmdedd..oo.e.o 1002695.
Rammerstorfer, M. and Mössenböck, H. Data mappings in the model-view-connrollee pattern. Broy, M., Zamulin, A.V. (eds) Perspectives of System Informatics. PSI 2003, pP. 121 2003, doi:
Papadopoulos, S., Datta, K., Madden, S., and Mattson, T. The tiledb array data storage manager. Proc. VLDB Endow. 10:349-360, 2016. doi: 10.14778/3025111. 3025117.
Replogle, J. M., Saunders, R. A., Pogson, A. N., Jost, M., Norman, T. M.. Weissman, J. S., et al. Mapping information-rich genotype-phenotype landscapes with genome-scale perturb-seq. Cell, (55(1))2559-25554)
Parca, L., Pepe, G., Pietrosanto, M.. Galvan, G., Galli, L.,, Palmeri, A., Sciandrone, M., Ferrè, F., Ausiello, G., and Helmer-CCtteeieee M. Modeling cancer drug response through drug-specific informative genes. Scientific Re- ports, 9, 2019. doi: 10. 10.10388/411988.11--57222
2022. doi: 10.1016fj.cell22222.00 20.22.055113... Open access.
Reynisson, B., Alvarez, B., Paul, S., Peters, B., and Nielsen, M. Netmhepan-4, 1 and netmhciipan-4.. improved pre- dictions of mhc antigen presentation by concurrent motif deconvolution and integration of ms mhe eluted ligand data. Nucleic Acids Research, 48:W449- W454, 2020. doi: 10. (0.1093/nar/gkaa339
Park, J. H.. Cho, H. E., Kim, J. H., Wall, M., Stern, Y., Lim, H., Yoo, S., Kim, H.-S., and Cha, J. Machine learning prediction of incidence of alzheimer's disease using large- scale administrative health data. NPJ Digital Medicine, 3, 2020. doi: 10.3/08/ 746-020-0256-0.
Romero-Molina, S., Ruiz-Blanco, Y. B., Mieres-Perez, J., Harms, M., Münch, J., Ehrmann, M., and Sänchez-Garcfa E. Ppi-affinity: A web tool for the prediction and opti- mization of protein protein-peptide and affinity. Journal of Proteome Research, 21:1829 1841, 2022. doi: 10. 10.1021/acs wwwptpooee2c00000000
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J.. Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., De- Vito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-pent perfforrrreeece deep learning li- brary. In Advances Neural Information Processing Sys- tems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL https::/pytorch.. https:/pyorrch.og/oessttaaeeeeee org//dos//stabl/daaa... ham.eaaabeeeeaaaaatttmtthhhch hww.#tocch.utiiiii data.ItrrableDatase.
protein binding
Rongali, S., Rose, A., McManus, D., Bajracharya, A. S., Kapoor, A., Granillo, E., and Yu, H. Learning latent space representations to predict patient outcomes: Model development and validation. Journal of Medical Internet Research, 22, 2020. doi: 10.2196//6374
Roohani, Y., Huang, K., and Leskovec, J. Predicting tran- scriptional outcomes of novel multigene perturbations with gears. Nature Biotechnology, Aug 2023. Open access.
Patil, S. G. Zhang, T., Wang, X., and Gonzalez, J. E. Go- rilla: Large language model connected with massive apis. ArXiv, abs/2305.15334 2023. doi: 10.48855 15334.
Saxena, S., Animesh, S., Fullwood, M., and Mu, Y. Onion- mhe: A deep learning model for peptide - hla-a binding predictions using both structure and sequence feature sets. Journal of Micromechunics and Molecular Physies, 2020. doi: 10.
Peidli, S., Green, T. D., Shen, C., Gross, T., Min, J., Garda, S., Yuan, B., Schumacher, J., Taylor-King. King, J. P., Marks, D. S., Luna, A., Blüthgen, N., and Sander, C. scper- turb: harmonized single-cell perturbation data. Nature Methods, 21:531-5400 Jan 2024.
Sayaman, R. W., Wolf, D. M., Yau, C., Wulfkuhle, J., Pet- ricoin, E., Brown- Swigarr,, L., Asare, S. M., Hirst, G. L., Sit, L.. O'Grady, N., Hedistian, D., Consortium, I.-S. T., Esserman, L. J., LaBarge, M. A., and van 't Veer, I.. J. Application of machine learning to elucidate the biology predicting response in the i-spy 2 neoadjuvant breast cancer trial. Cancer Research, 80(4 Suppl), 2020. doi: 10.1 /58/1538--111111 74488-445..00 www.5..abss99.. 11-pss1--111---..
Piran, Z., Cohen, N., Hoshen, Y., and Nitzan, M. Biolog- ical representation disentangiement of single-cell data. bioRxiv, 2023. doi: 10.1101/2023.03.006 05 05.531195. URL http://0....g///0001122222......
Ploegh, H. L. Antigen processing and presentation. Nature, 353(6342): 1998.
Rahimian, F. Salimi-Khorshidi G., Payberah, A. H., Tran, J.. Solares, R. A., Raimondi, F., Nazarzadeh, M., Canoy, D., and Rahimi, K. Predicting the risk of emergency
Schatz, D. G. and Swanson, P. C. V(d) recombination: mechanisms of initiation. Anmual Review of Genetics, 45:
167-202, 2011.
15

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Schick, T., Schmid, H., and Schütze, H. Toolformer: Lan- guage models teach themselves to use tools. In Pro- ceedings of the 2023 Conference of the North American Chapter of the Association for Computational Linguisties. Association for Computational Linguistics, 2023.
regular functions and generator functions in python. In- ternational Journal of Security, Privacy and Trust Man- agement, 2023, doi: 0..111//jsptm.223..2222
Theodoris, C., Xiao, L., Chopra, A., Chaffin, M., Sayed, Z., Hill, M., Mantineo, H., Brydon, E., Zeng. Z., Liu, S., and Ellinor, P. Transfer learning enables predictions network biology. Nature, 618:1-9, 05 2023. doi: 10. 00088//41186-023-06666666
Schperberg, A. V., Boichard, A., Tsigelny, I., Richard, S., and Kurzrock, R. Machine learning model predict on- cologic outcomes for drugs in randomized clinical trials. International Journal of Cancer, 147:2537-2544 2020. doi: 10. 00.100//jc..32240.
Theodoris, C. Perspectives on benchmarking foundation
models for network biology. Quantitative Biology, 12(4):
Senders, J., Staples, C., Karhade, A., Zaki, M. M., Gorm- ley, W., Broekman, M., Smith, T., and Arnaout, O. Ma- chine learning and neurosurgical outcome prediction: A systematic review. World Neurosurgery; 109:476-486.e1, 2018. doi: 10.1016/j. w.e...00...222209.14..9
335-338, December 2024. doi: 10. 10.1016/jquuamtio.200) 2024.
04.001. URL https://doi.org// wttgg//..011/..uuaooooo htt.:/..quaaniio222220..0
001. Open Access.
Thiago, R., Souza, R., Azevedo, L.., Soares, E., Santos, R., Santos, W., Bayser, M. D., Cardoso, M., Moreno, M., and Cerqueira, R. Managing data lineage of o&g machine learning models: The sweet spot for shale use case, 2020. URL https://axiixorrrabb22000.....
Shamout, F. E., Zhu, T., and Clifton, D. Machine learn- ing for clinical outcome prediction. IEEE Reviews in Biomedical Engineering, 14:116-126, 2020. doi: 10.1 1099/
van der Weide, T., Papadopoulos, D., Smirnov, O., Zielin- ski, M., and van Kasteren, T. Versioning for end-to- end machine learning pipelines. In Proceedings of the Ist Workshop on Data Management for End-to-End Machine Learning, DEEM' 17, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450350266. doi: 10. 00.1145/3076246.3000 34707662246... URL https://doi.org/10.1114 http:p//.....11110/0//6660000040
Siah, K. W., Khozin, S., Wong, C. H., and Lo, A. Machine- learning and stochastic tumor growth models for predict- ing outcomes in patients with advanced non-small-cell lung cancer. JCO Clinical Cancer Informatics, 3:1-11, 2019, doi: 10. 10.1/0//C CEECCCC....000000 9.00046.
Smith, L. M. and Kelleher, N. L. Proteoform: single term describing protein complexity. Nature Methods, 10(3): 186-187, 2013. doi: 10. 10.1088/mmehh.2369
van Os, H., Ramos, L. A., Hilbert, A., van Leeuwen, M., van Walderveen, M. V., Kruyt, N., Dippel, D., Steyerberg, E., van der Schaaf, I., Lingsma, H. F,, Schonewille, W., Majoie, C., Olabarriaga, S., Zwinderman, K., Venema, E., Marquering, H., and Wermer, M. Predicting outcome of endovascular treatment for acute ischemic stroke: Po- tential value of machine learning algorithms. Frontiers in Neurology, 9:784, 2018. doi: 201 8000884.
Springer, I., Besser, H., Tickotsky-Moskovitz, N., Dvorkin, S., and Louzoun, Y. Prediction of specific ter-peptide binding from large dictionaries of ter-peptide pairs. Fron- tiers in Immunology, 11, 2019. doi: 10.3389/immmu.200 01803.
Srivatsan, S. R., McFaline-Figueroa, J. L., Ramani, V.,
Saunders, I.., Cao, J., Packer, J., Pliner, H. A., Jackson, D. L., Daza, R. M., Trapnell, C., et al. Massively mul- tiplex chemical transeriptomics single-cell resolution. Science, 36766433) 2019. doi: 10.1 16/sccncee aax6234.
Velez-Arce, A., Lin, X., Huang, K., Li. M. M., Gao, W., Pentelute, B., Fu, T., Kellis, M., and Zitnik, M. Sig- nals in the cells: Multimodal and contextualized machine learning foundations for therapeutics. In NeurlPS 2024 Workshop on Al for New Drug Modalities, 2024. URL http:://ppnmeveee...eetooottmked
T, R., Demerdash, O., and Smith, J. C. Ter-h: Machine learning prediction of t-cell receptor epitope binding on unseen datasets. bioRxiv, 2023, doi: 10.1101/2023.1..0 569077.
Velickovic, Cucurull, G., Casanova, A., Romero, A.. Lio' P., and Bengio, Y. Graph attention networks. ArXiv, abs/17 10/11710.1003. 2017. doi: 10. (0.11863/CAM.488299
Targets, O.
Open targets platform: Ra and ibd discase drug targets, 2023. URL http::/wwww.opetaarges.orgg Accessed: 2024-05-21.
Wang J., Zhang, I.,, Zeng, A., Xia, D., Yu, J., and Yu, G. Decpiii: Predicting isoform-isoform interactions by deep neural networks and data fusion. JEEE/ACM Transactions on Computational Biology and Bioinformatics, 19:2177- 2187, 2021a. doi: 10. 1..110/TCBBB. CTTT//CBBBBB....... 30268306887
Tasoluk, B. and Tanrikulu, Z. The performance compari- son brute-force password cracking algorithm using
16

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Wang, Y., Zeng, J., Yang, J., Zhou, J., Niu, B., and Guan, J. Prediction of drug-target interaction networks from the integration of protein sequences and drug chemi- cal structures. Molecules, 24(2):321, 2019. doi: 10. 5390mmllecules22400033.. URL https:///oi.orr/11........ molccules24020321.
Yang. M., Huang. Z.-a., Zhou, W., Ji, J., Zhang, J., He, S., and Zhu, Z. Mix-tpi: flexible prediction framework for ter-pmhe interactions based on multi- modal representations. Bioinformaties, 39, 2023. doi: 10. 10.10//hoinnoomaaii/hhaa///5
Yaseen, A., Abbasi, W. A., and ul Amir Afsar Minhas, F. Protein binding affinity prediction using support vec- tor regression and interfecial features. 2018 15th In- ternational Bhurban Conference on Applied Sciences and Technology (IBCAST), pp. 194-198, 2018. doi: 10.1100/BCAAST2228888812
Wang. Y., Carter, B., Li, Z., and Huang, X. Applica- tion of machine learning methods in clinical trials for precision medicine. JAMIA Open, 5, 2021b. doi:
10.10053jamaappea/oob1111..
Wang, Z., Xiao, C., and Sun, J. Spot: Sequential predictive modeling of clinical trial outcome with meta-learning. In Proceedings of the 14th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, 2023, doi: 10.1145/3584337133133000
Ye, X., Lee, Y. C., Gates, Z. P., Ling, Y., Mortensen, J. C., Yang, E. S., Lin, Y. S., and Pentelute, B. L. Binary combi- natorial scanning reveals potent poly-alaniee-sussiiutee inhibitors of protein-protein interactions. Communica- tions Chemistry, 5(1):128, Oct 2022. doi: 10.1038/ s42004-022-00337-ww
Weber, A., Born, J., and Martínez, M. R. Titan: T-cell receptor specificity prediction with bimodal attention networks. Bioinformaties, 37:1237-i244 2021. doi: 10. 10.099/boinfommaicccbaaa222244
You, Y. and Shen, Y. Cross-modality protein embedding for compound-protein affinity and contact prediction. bioRxiv, 2020. doi: 10.1101/202011.2244333666
Wolf, F. A., Angerer, P., and Theis, F. J. SCANPY: large- scale single-cell gene expression data analysis. Genome Biology, 19:1-5, 2018
Yu, H. and Welch, J. D. Perturbnet predicts single-cell responses to unseen chemical and genetic perturbations. bioRxiv, 2022. doi: 10. 10.0000022200...000000444 URL http::/doi..or/100 00. wttp://..r.r/...11222222220 111112022.200111 20.500444.
Wolf, T., Debut, L., Sanh, V.. Chaumond, J., Delangue, C.,, Moi, A., Cistac, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q.. and Rush, A. M. Transformers: State-of-the-arr natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Sys- tem Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. URL https: //www www.aalwwbbogaattooog/g///eeom/m
Zambrano Chaves, J. M., Wang. E., Tu, T., Vaishnav, E. D., Lee, B., Mahdavi, S. S., Semturs, C., Fleet, D., Natarajan, V., and Azizi, S. Tx-llm: A large language model for therapeutics. arXiv preprint arXiv:2406. 06316, 2024. doi: 10. 10.448550/rxxvv.246666 2406.06316.
URL http://arxxvv.rr/abb/
Zeng. J., Yu, G., Wang, J., Guo, M., and Zhang, X. Dmil- ii: Isoform-isoform interaction prediction using deep multi-instance learning method. 2019 IEEE Interna- tional Conference Bioinformatics and Biomedicine (BIBM), pp. 171-176, 2019. doi: 10.1109/BIBM442566 2019.8982956.
Wu, Z., Ramsundar, B., Feinberg, N., Gomes, J., Ge- niesse, C., Pappu, A. S., Leswing, K., and Pande, V. S. Moleculenet: A benchmark for molecular ma- chine learning. Chemical Science, 9:513-530, 2018. doi: 10.1039/C75C026644.
Xu, M., Zhang, Z., Lu, J., Zhu, 7., Zhang, Y., Ma, C., Liu, R., and Tang. J. Peer: A comprehensive and multi-task benchmark for protein sequence understanding, 2022. Ac- cepted by NeurlPS 2022 Dataset and Benchmark Track. arXiv v2: source code released; arXiv vl: release all benchmark results.
Zhang, Q. Yu, F. X., Wu, Y., al. Novel gene ther- apy for rheumatoid arthritis with single local injection: adeno-associated vinus-mediated delivery of a20/tnaaip3. Military Medical Research, 9(1):34, 2022. doi: 10. 1 1186/440799-022-003994 URL https://doi.org/1...111 $40779-022-003993-0.
Yang, F., Wang, W., Wang, F., Fang. Y., Tang, D., Huang, J., Lu, H., and Yao, J. scbert as a large-scale pretrained deep language model for cell type annotation of single-cell ma- seq data. Nature Machine Intelligence, 4(10):852-866 866, 2022.
Zhang. S., Pyne, S., Pietrzak, S. J., Halberg, S., McCalla, S., Siahpirani, A. F., Sridharan, R., and Roy, S. Infer- ence of cell type-specific gene regulatory networks on cell lineages from single cell omic datasets. Nature Com- munications, 14, 2023. doi: 2tt.222222...2.55111..1
17

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Zhang, X., Xiao, C., Glass, L. M., and Sun, J. Deepenroll: patient-trial matching with deep embedding and entail- ment prediction. In Proceedings of the web conference 2020, pp. 1029-1037, 2020.
Zheng, S., Li, Y., Chen, S., Xu, J. and Yang. Y. Predicting drug-protein interaction using quasi-visual question swering system. Nature Machine Intelligence, 2:134-140, 0200044222226000------22
2019. doi:
Zhu, Y., Ouyang, X., Li, P. Jin, Q., Su, J., Zheng, L., and Ye, L. Torchdrug: A powerful and flexible machine learning platform for drug discovery. Journal of Chemical Information and Modeling, 62(9):2204-2211 2022. doi: 10.102 2c05533...
Öztürk, H., Özgür, A., and Ozkirimli, E. Deepdta: deep Bruug-targe binding affinity prediction. Bioinfor- matics, 34(17):881--882, Sep 2018. 8. bioinformatics/ty553...
doi: 10.1093/
18

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
A. New tasks:
datasets, benchmarks, and code
PyTDC (Velez-Arce et al., 2024) introduces four tasks with fine-grained biological contexts: contextualized drug-target identification, single-cell chemical/geentic perturbation response prediction, and cell-type- protein peeteiee binding interaction prediction, which introduce antigen-proceet paattgg-phhwwwaaaaeaeee cell-tye-specific, peptide-specific, and patient- specific biological contexts. Benchmarks for drug target nomination, genetic perturbation response prediction and chemical perturbation response prediction, all at single-cell resolution, were computed with corresponding leaderboards introduced on the TDC website. In addition, a benchmark and leaderboard was introduced for the TCR-Epitope binding interaction task for peptide design at single-cell resolution for T-cell receptors.
Context-specifie metrics (section 3.3 and 4.2). In real-world machine learning applications, data subsets can correspond to critical outcomes, In therapeutics, there is evidence that the effects of drugs can vary depending on the type of cell they are targeting and where specific proteins are acting (Zhang et al.. 2022). We build on the "slice" abstraction (Chen et al., 2020a) to measure model performance at critical biological subsets. Context-specific metrics are defined to measure model performance at critical biological slices, with our benchmarks focused on measuring cell-tyee-specific model performance. In the case of benchmarks for perturbation response prediction and protein-peptide binding affinity, the studies are limited to a particular cell line, however our definition for context-specific metrics lays the foundations for building models which can generalize across cell lines and make context-aware predictions (Kwon et al., 2024). For single-cell drug-targe nomination, we measure model performance at nop-performing cell types. See corresponding sections 3.3 and 4.2 in the main text.
A.1. tdc.ml. aedc.ml.scDDre ScDTI: Contextualized Drug- TargTarget Identification
Motivation. Single-cell data have enabled the study of gene expression and function at the level of individual cells across healthy and disease states (Jones et al., 2022; CZI Single-Cell Ceele-Celll Biology, et al., 2023; Kwon et al., 2024). To facilitate biological discoveries using single-cell data, machine-learning models have been developed to capture the complex, cell- type-specific behavior of genes (Theodoris et al., 2023; Cui et al., 2024; Yang et al., 2022; Li et al., 2024). In addition to providing the single-cell measurements and foundation models, PyTDC supports the development of contextual Al models to nominate therapeutic targets in cell type-specific manner (Li et al., 2024). We introduce benchmark dataset, model, and leaderboard for context- speeext--peciiic therapeutic target prioritization, encouraging the innovation of model architectures (e.g., to incorporate new modalities, such as protein structure and sequences (Li et al., 2020; 2019; Lee et al., 2018; Öztürk et al., 2018; Meng et al., 2017), genetic perturbation data (Ji et al., 2020; Ghadie et al., 2017; Zeng et al., 2019; Wang et al., a0) disease-specific single-cell atlases (Carlberg et al., 2019; Kuenzi et al., 2020; Julkunen et al., 2020), and protein networks (Parca et al., 2019: Zhang et al., 2023; Gupta et al., 2022)). PyTDC's release of tdc_ml. . scDTI is a step in standardizing benchmarks for more comprehensive assessments of context-specific model performance.
A.1.1. DATASET AND BENCHMARK
We use curated therapeutic target labels from the Open Targets Platform (Targets, 2023) for rheumatoid arthritis (RA) and inflammatory bowel disease (IBD) (Li et al., 2024) (section A. 1.2. We benchmark PINNACLE (Li et al., 20an)-trained on cell type specific protein-proteie interaction networks and a graph attention neural network (GAT) (Velickovic et al., on a context-free reference mrotein-pootei interaction network- on the curated therapeutic targets dataset. As expected, PINNACLE underperforms when evaluated on context-agnostic metrics (Table 3) and drastically outperforms GAT when evaluated on context-specific metrics (Appendix Table 4).
To our knowledge, PYTDC provides the first benchmark for context-specifie learning (Kwon et al., 2024). PyTDC's contribution helps standardize the evaluation of single-cell ML models for drug target identification and other single-cell tasks (Yang et al., 2022; Theodoris et al., 2023; Li et al., 2024; Cui et al., 2024).
Model
AP@5
AUROC
PINNACLE (RA)
0.23
0.55
GAT (RA)
0.22
0.58
PINNACLE (IBD)
0.20
0.58
GAT (IBD)
0.20
0.64
Table 3. RA and IBD results for PINNACLE and Graph Attention Network without cell-tyee-specific context metrics. These results obtained after evaluating 10 seeds.
19

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Table Cell-type specific target nomination for two therapeutic areas, rheumatoid arthritis (RA) and inflammatory bowel diseases (IBD), Cell-type specific context metrics (definitions in Section 4.2): AP@5 Top-20 CT average precision at le = 5 for the 20 best performing cell types (CT); AUROC Top-1 CT AUROC for top- performing cell type; AUROC Top-10 CT and AUROC Top-20 CT weighted average AUROC for top-10 10 and top-20 performing cell types, respectively, each weighted by the number of samples in each cell type; AP@ S/AUROC CF context-free AP@S/AUROC integrated across cell types. Shown are results from models run on ten independent seeds. N/A not applicable.
Model
AP@ 5 Top-20 CT
AUROC Top-1 CT
AUROC Top- CO
AUROC AUROCTTpp-00 CT
AP@ CF
AUROC CF
PINNACLE (RA)
0.913-0099
0.765-0a54
0.676=1207
0.647=am4 N/A
0.226=w023
0.510=00m
GAT (RA)
NIA
NIA
N/A
0.580
PINNACLE (IBD)
0.873:1009 N/A
0.935=0006
0.799
0.752aam N/A
0.500
GAT (IBD)
N/A
N/A
0.200=m023
0.640=8017
A. 1.1 (LI, MICHELLE, ET AL.) DATASET
To curate target information for a therapeutic area, we examine the drugs indicated for the therapeutic area of interest and its descendants. The two therapeutic areas examined are rheumatoid arthritis (RA) and inflammatory bowel disease, For rheumatoid arthritis, we collected therapeutic data (i.e., targets of drugs indicated for the therapeutic area) from OpenTargets for rheumatoid arthritis (EFO 0000685), ankylosing spondylitis (EFO 0003898), and psoriatic arthritis (EFO 0003778). For inflammatory bowel disease, we collected therapeutic data for ulcerative colitis (EFO 0000729), collagenous colitis (EFO 1001293), colitis (EFO 0003872), proctitis (EFO 0005628), Crohn's 's colitis (EFO 0005622), lymphocytic colitis (EFO 1001294), Crohn's disease (EFO 0000384), microscopic colitis (EFO 1001295), inflammatory bowel disease (EFO 0003767), appendicitis (EFO 0007 149), ulcerative proctosigmoiditis (EFO 1001223). and small bowel Crohn's 's disease (EFO 0005629).
We define positive examples (i.e., where the label y 1) as proteins targeted by drugs that have at least completed phase 2 of clinical trials for treating specific therapeutic area. As such, protein is promising candidate if compound that targets the protein is safe for humans and effective for treating the disease. We retain positive training examples activated in at least one cell type-specific protein interaction network.
We define negative examples (i.e., where the label y = 0) as druggable proteins that do not have any known association with the therapeutic area of interest according to Open Targets. A protein is deemed druggable if targeted by least one existing drug. We extract drugs and their nominal targets from Drugbank. We retain negative training examples activated least one cell type-specific protein interaction network.
Dataset statistics. The final number of positive (negative) samples for RA and IBD were 152 (1,465) and 114 (1,377), respectively. In (Li et al., 2024), this dataset was augmented to include 156 cell types.
Dataset split. Cold Split: We split the dataset such that about 80% of the proteins are in the training set, about 10% of the proteins are in the validation set, and about 10% of the proteins are in the test set. The data splits are consistent for each cell type context to avoid data leakage.
References. (Li et al., 2024)
Dataset license. CC BY 4.0
Code Sample. The dataset and splits currently available TDC Harvard Dataverse. In addition, you may obtain the protein splits used in (Li et al., 2024) via the following code.
from www.:mrwerssuueeemmmmmmmm dataloader import DataLoader data = DataLoader splits =
A. 1.3. CODE FOR DATASETS AND BENCHMARK
Retrieving training and validation datasets for the tdc_ml.e.... ml.NcDTN task as simple as 3 lines of code, with an additional 2 lines enabling model evaluation. Furthermore, PyTDC exposes the PPI network and cell types used for benchmarking of (Grover& Leskovec, 2016; Velickovic al., 2017; Li et al., 2024; Brody al., 2022) via a handful of lines of code as well.
20

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Listing 1 The below code illustrates how retrieve the train, test, and validation splits and model evaluation for the
sc.. benchmark.
From tdc_ ml. wwnchhark benchmark_group group import scdti_group
group scdtn_greup... ccctn_greoup.SSTTirrrru
train, val group get__tainn i d_sppit a (seed- Train model
test = troup.get_teet () ["test"] # Obtain predictions on the test sot
group. proup.evaluate (preds) Get benchmark results
Listing The below code illustrates how to retrieve the PPI network and cell type labels (Li et al., 2024) used for training and benchmarking of models tdc m.N. Retrieval cell type labels and embeddings from (Li et al., 2024) are also supported.
from tdc_ml resource pinnacle import PINNACLE
graph pinnacle get_ppi () ppi network
embeds pinnacle get_embeds () full pre-computed embeddings dataset
seproteins pinnacle, ket_kess () # protein cell types
A.1.4. TDC.ML. SCDTI: CONTEXTUALIZED DRUG-TARGET NOMINATION (IDENTIFICATION)
PyTDC introduces tde ml SCDTI task. The predictive, non-generative task is formalized learning estimator for discase-specific function f of target protein and cell type outputting whether the candidate protein t is therapeutic target in that cell type
= c),
(14)
Target candidate set. The target candidate set includes proteins, nucleic acids, or other molecules drugs can interact with, producing therapeutic effect or causing biological response. The target candidate is constrained proteins relevant the disease being treated. It is denoted by: T= IN...
(15)
where f1,-.., ,N, are N; target candidates for the drugs treating the disease. Information modeled for target candidates can include interaction, structural, and sequence information.
Biological context set. The biological context set includes the cell-type-specife contexts in which the target candidate set operates. This set is denoted
C= (c, CN.):
(16)
where cI...., CN. are No biological contexts on which drug-target interactions are being evaluated. Information modeled for cell-type-specific biological contexts can include gene expression and tissue hierarchy. The set is constrained to disease- speciffc cell types and tissues. Drug-target identification.
Identification a binary label y (1,0), where indicates the protein is candidate therapeutic target. the same time, 0 means the protein such target.
The goal is to train model fe for predicting the probability á [0, 1] that protein is candidate therapeutic target in a specific cell type. The model learns an estimator for disease-specife function of a protein target 1. T and sell-type-specific biological context e E C as input, and the model is tasked to predict:
= fo E C,CEC).
(17)
A.2. tdc ml. Perturboutccome: Perturbationn Prediction
Motivation. Understanding and predicting transcriptional responses to genetic or chemical perturbations provides insights into cellular adaptation and response mechanisms. Such predictions can advance therapeutic strategies, as they enable
21

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
researchers anticipate how cells will react targeted interventions, potentially guiding more effective treatments. Models that have shown promise this task (Roohani et al., 2023; Hetzel al., 2022) are limited either genetic or chemical perturbations without being able to generalize to the other. Approaches that can generalize across chemical and genetic perturbations (Piran et al., 2023; Yu & Welch, 2022) may be unable to generalize to unseen perturbations without modification.
1. DATASET AND BENCHMARK
We used the scPerturb (Peidli et al., 2024) datasets to benchmark the generalizability of perturbation-responee prediction models across seen/unseen perturbations and cell lines. We benchmark models in genetic and chemical perturbations using metries measuring intra/inter-cell line and scen/unseen perturbation generalizability. We provide results measuring unseen perturbation generalizability for Gene Perturbation Response Prediction using the scPerturb gene datasets (Norman K562, Replogle K562, Replogle RPE1) (Norman et al., 2019; Replogle et al., 2022) with results shown in Table 5. For Chemical Perturbation Prediction, we evaluated chemCPA utilizing cold splits on perturbation type and showed a significant decrease in performance for of 4 perturbations evaluated table (6). We have also included Biolord (Piran et al., 2023) and seGen (Lotfollahi et al., 2019) for comprehensive benchmarking on the well-explored perturbation response prediction on seen perturbation types problem. These tests were run on sciPlex2 (Srivatsan al., 2019).
A.2.2. SCPERTURE DATASET
The scPerturb dataset is a comprehensive collection of single-cell perturbation data harmonized to facilitate the development and benchmarking of computational methods in systems biology. It includes various types of molecular readouts, such as transcriptomics, proteomics, and epigenomics. scPerturb is a harmonized dataset that compiles single-cell perturbation- response data. This dataset is designed to support the development and validation computational tools by providing a consistent and comprehensive resource. The data includes responses to various genetic and chemical perturbations, erucial for understanding cellular mechanisms and developing therapeutic strategies. Data from different sources are uniformly pre-processed to ensure consistency. Rigorous quality control measures are applied to maintain high data quality. Features across different datasets are standardized for easy comparison and integration.
Dataset statistics. 44 publicly available single-cell perturbation-responee datasets. Most datasets have, on average, approximately 3000 genes measured per cell. 100,000+ perturbations.
Dataset split. Cold Split and Random Split defined on cell lines and perturbation types.
References. (Peidli et al., 2024)
Dataset license. CC BY 4.0
Code Sample.
from Edc_ml wwwtuuliiia..m..m per..uuutt.. import PerturbOut.come from pandas import DataFrame
test_loader = Perturboutcome (
rcae- htwp:ubbddduuggiiiaeeeeeeeeeoooe Als.ggggssssseeeellleessaaaaaaa
testdf = test__oaaee get__gettaattalll ()
A.2.3. GENETIC PERTURBATION RESPONSE PREDICTION
We scPerturb gene datasets (Norman K562, Replogle K562, Replogle RPE1) (Norman et al., 2019; Replogle al., 2022). In the case of single-gene perturbations, assessed the models based the perturbation of experimentally perturbed genes not included in the training data. We used data from two genetic perturbation screens, with 1,543 perturbations for RPE-1 1 (retinal pigment epithelium) cells and 1,092 for K-562 cells, each involving over 170,000 cells. These screens utilized the Perturb-seq assay, which combines pooled screening with single-cell RNA sequencing to analyze the entire transcriptome for each cell. We trained GEARS separately on each dataset. In addition to an existing deep learning-bass model (CPA), we also developed baseline model (no perturbation), assuming that gene expression remains unchanged after perturbation.
We evaluated the models' performance by calculating the mean squared error between the predicted gene expression after perturbation and the actual postueerturbation expression for the held-out set. Based on the highest absolute differential expression upon perturbation, the top 20 most differentially expressed genes were selected.
22

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Table Unseen genetic perturbation response prediction. We evaluate GEARS across the top 20 differentially expressed genes. based on the highest absolute differential expression upon perturbation, for MSE (MSE @ 20DEG), Gene expression was measured in log normalized counts, single-cell analysis, standard procedure to normalize the counts within each cell so that they sum to specific value (usually the median sum across all cells in the dataset) and then log transform the values using the natural logarithm (Roohani al., 2023). For both normalization and ranking genes by differential expression, we utilized Scanpy (Wolf et al., 2018). We used the sc.tl.rank. weweeggnnesggggg groups() function with default parameters in Scanpy, which employs t-test estimate scores. This function provides z-score for each gene and ranks genes based on the absolute values the score. Genes showing a significant level of dropout were not
included in this metric.
Dataset
Tissue
Cell Line
Method
MSE@20DEG
Norman K562
K562
lymphoblast lymphoblast
no-perturb CPA
0,341=00001
Norman K562
K562
0.2301..0008
Norman K562
K562
lymphoblast
GEARS
0.176.10.003
Replogle 562
K562
lymphoblast
no-perturb
0. 1.6:
Replogle 562
K5562
lymphohlast
CPA
0. 0.126::0.000
Replogle 562
K:522
lymphoblast
GEARS
0. 0.109
Replogle RPE1
RPE-1
epithelial
no-perturb
0. 0.004
Replogle RPE1
RPE-1
opithelial
CPA
0. 0.162:10.00
Replogic RPE1
RPE-1
opithelial
GEARS
0.1 100+000003
Listing 1. The below code illustrates how to retrieve the train, test, and val splits used for the tdc Pew..reeerrrruttooommee genetic perturbation benchmark
from tdc_ml ml, bwwchharkkgggoup import geneperturb_groop group geneperturb_ http:uuuwwbgroouu..eeeeeeeeeeeee G ()
train_val val
getppp.et_ttaiinn aalid_sppitt___ ()
test group. getptstetttest()
# train your model and test the test set
group.evaluee (preds)
A.2.4. CHEMICAL PERTURBATION RESPONSE PREDICTION
The dataset consists of four drug-based perturbations from sciPlex2 (Srivatsan et al., 2019; Peidli et al., 2024) (BMS, Dex, Nutlin, SAHA). sciPlex2 contains alveolar basal epithelial cells from the A549 (lung adenocareinoma), K562 (chronic myelogenous leukemia), and MCF7 (mammary adenocareinoma) tissues. Results are shown in Table Our experiments rely on the coefficient of determination (R²) as the primary performance measure. We calculate this score by comparing actual measurements with counterfactual predictions for all genes. Assessing all genes essential to evaluating the decoder's s overall performance and understanding the background context. Still, it is also beneficial to determine performance based on top differentially expressed genes (Hetzel et al., 2022). The baseline used discards all perturbation information, adequately measuring the improvement resulting by the models' drug encoding (Hetzel et al., 2022). ChemCPA's performance dropped by an average of 15% across the four perturbations. The maximum drop was 34%. Code for intra/inter cell-line benchmarks for chemical (drug) and genetic (CRISPR) perturbations is in Appendix A.2.4 and Appendix A.2.3, respectively. Using this code, users can evaluate models of their choice on the benchmark.
23

---

PYTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Table Unseen chemical perturbation response prediction. We have evaluated chemCPA utilizzig cold splits on perturbation type and show a significant decrease in performance for 3 of perturbations evaluated. We have also included Biolord (Piran et al.. 2023) and scGen (Lotfollahi al., 2019) for comparison. The dataset used consists of four chemical (drug) perturbations from sciPlex2 (Peidli al., 2024) (BMS, Dex, Nutlin, SAHA). sciPlex2 contains alveolar basal epithelial cells from the A.49 (lung adenocarcinoma), K562 (chronic myelogenous leukemia), and MCF7 (mammary adenocarcinoma) tissues. Our experiments rely on the coefficient determination (R²)
the primary performance measure.
Drug
Method
Rt (seen perturbations)
It (unseen perturbations)
BMS
Bascline
0.620=0.044
N/A
Dex
Bascline
0,603=0.053
N/A
Nutlin
Baseline
0.628_0.036
N/A
SAHA
Baseline
0.617=0.027
N/A
BMS
Biolord
0.939=0.022
N/A
Dex
Biolord
0.942=00.028
N/A
Nutlin
Biolord
0.928_0.026
N/A
SAHA
Biolord
0.980=0.005
N/A
BMS
ChemCPA
0.943=0.006
0.906:10.006
Dex
ChemCPA
0.882=0.014
0.540:10.013
Nutlin
ChemCPA
0.925=0.010
0.835=+0.009
SAHA
ChemCPA
0.825=0.026
0.690:10.021
BMS
scGen
0.903=0.030
N/A
Dex
scGen
0.944=00018
N/A
Nutlin
scGen
0.891
N/A
SAHA
scGen
0.948=0.00
N/A
Listing The below code illustrates how to retrieve the train, test, and val splits used for the tdc. ww..eetturbbOttcmm chemical
perturbation benchmark
from tdc__i m1cm_mm_____ group www.uteeefccuulllgooooo...ttetee train, val group
bewchharkkggguup import counterfactial__groyy
www.pppplltteeeeee...oeeoem (removee_uneeen=Falls
test group.get_test
# train your model and test the test set group.evaluse (preds)
A.2.5. IDC.ML..ERTUREOUUE
PERTURBATION RESSONE PROBLEM FORMULATION
PYTDC introduces Perturbation-Ren SeturbbttionRReponsee prediction task. The predictive, non-generative task is formalized learning an estimator for function of the cell-type-specific gene expression response to chemical or genetic perturbation, taking a perturbation E P, pre-perturbation gene expression profile from the control set ég E Ey, and the biological context c e C under which the gene expression response to the perturbation being measured:
y f(p. cu.
(18)
We center our definition on regression for the cell- sell-type-specife gene expression vector in response to a chemical or genetic perturbation.
Perturbation set. The perturbation set includes genetic and chemical perturbations. denoted by:
P [p (pI...,,N, PN,
(19)
where tp....,PNy PN, are Np evaluated perturbations. Information modeled for genetic perturbations can include the type of perturbation (i.e., knockout, knockdown, overexpression) and target gene(s) of the perturbation. Information modeled for chemical perturbations can include chemical structure (i.e., SMILES, InChl) and concentration and duration of treatment.
Control set. The control set includes the unperturbed gene expression profiles. This set is denoted as:
Ey = ENag)
(20)
24

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
where 601s..... Ex. are Ne. unperturbed gene expression profile vectors. Information models for gene expression profiles can include or normalized gene expression counts, transcriptomic profiles, and isoforn-specific expression levels.
Biological context set. The biological context set includes the cell-yype-specific contexts under which the perturbed gene expression profile is measured. It is denoted by:
C= f=fei....,.N. C..).
(21)
where cI,..., CN2 are the N₂ biological contexts under which perturbations are being evaluated. Information modeled for biological contexts can include cell type or tissue type and experimental conditions (Peidli et al., 2024) as well as epigenetic markers (Agrawal et al., 2023; Nair et al., 2020).
Perturbation-ressoose readouts. Perturbation Responns a gene expression vector éi, where éi, denotes the expression of the i-th gene in the vector. It is the outcome of applying a perturbation, p; E P., within a biological context, oj E C, to cell with measured control gene expression vector, eo2 E E,,.
The Setturabttio-Respppsa Prediction learning task is to learn a regression model fo estimating the perturbation-response gene expression vector ei for a perturbation applied in a cell-type-specific biological context to a control:
ci = fo E E
E Ex,CE
(22)
A.3. tdc.ml. ProteinPeptide: Contextualized Protein-Peptide Interaction Prediction
Motivation, Evaluating protein-pepee peptide binding prediction models requires standardized benchmarks, presenting challenges in assessing and validating model performance across different studies (Romero-Moline et al., 2022). Despite the availability interactions, this is not the case for pooeein-peptide interactions. The renowned
of several benchmarks for multi-task benchmark for Protein sEquence undERstanding (PEER) (Xu et al., 2022) and MoleculeNet (Wu et al., 2018) both lack support for a peteeide-e interaction prediction task. Furthermore, protein-peptidd binding mechanisms vary wildly by cellular and biological context (Janeway et al., 2001; Murphy & Weaver, 2016; 6; Ploegh, 1998; Schatz, & Swanson, 201 I). Current models, as such, tend to be restricted to one task instance (i.e., T Cell Receptor (TCR) and Peptide-MHC Complex or B Cell Receptor (BCR) and Antigen Peptide binding) and do not span protein-peptide interactions (Springer et al., 2019; Chen et al., 2020b; Liu et al., 2021; Bradlcy, 2022; Hu & Liu, 2019; Lupia et al., 2019; Saxena et al., 2020).
We define and evaluate subtask for TCR-Epitope binding interaction prediction applying contextual Al (Section 5.3.5) to
the T Cell cell line.
1. TCR-EPITOPE (PEPTIDE-MHC COMPLEX) INTERACTION PREDICTION
The critical challenge in TCR-Epitope (Peptide-MHC Complex) interaction prediction lies in creating model that can effectively generalize to unseen TCRs and epitopes (Moris et al., 2020). While TCR-H (T et al., 2023) and TEINet (Jiang et al., 2023) have shown improved performance on prediction for known epitopes, by incorporating advanced features like attention mechanisms and transfer learning, the performance considerably drops for unseen epitopes (Cai et al., 2022: Weber et al., 2021). Another challenge in TCR-Epitope interaction prediction lies in the choice of heuristic for generating negative samples, with non-binders often underrepresented or biased in curated datasets, leading to inaccurate predictions when generalized (Kim et al., 2023).
Datasets and Benchmarks. PYTDC establishes a curated dataset and benchmark within its single-cell protein-pepiic binding affinity prediction task to measure model generalizability to unseen TCRs and epitopes and model sensitivity to the selection of negative data points. Benchmarking datasets use three types of heuristics for generating negative samples: random shuffling of epitope and TCR sequences (RN), experimental negatives (NA), and pairing external TCR sequences with epitope sequences (ET), We harness data from the TC-hard dataset (Grazioli et al., 2022b) for the first two types and PanPep (Gao et al., 2023) for the third type. Both datasets use hard (Grazioli et al., 2022b) splits, ensuring that epitopes in the testing set not present in the training set. Our results (Table 7) show the lack of a reasonable heuristic for generating negative samples, with model performance evaluation shown to be unsatisfactory. For two heuristics, all models perform poorly. The best-performing model in ET is MIX-TPI TPI, with roughly 0.70 AUROC. The best-performing model in RN is AVIB-TCR, with approximately 0.576 AUROC. For NA, 4 of 6 models perform near- pear-perfectly as measured on AUROC. Models benchmarked include AVIB-TCR (Grazioli et al., 2022a), MIX-TPI (Yang et al., 2023), Net-TCR2 (Jensen & Nielsen, 2023), PanPep (Moris et al., 2020), TEINet (Jiang et al., 2023), and TITAN (Weber et al., 2021).
25

---

PYTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Table TCR-epitope binding interaction binary classification performance. All models perform poorly under realistic but challenging RN and ET experimental setups. The best- best-performing model in RN is AVIB- TCR, with an average of 0.576 (AUROC). The best- pest-performing model ET MII-TPI, with an average of 0.700 (AUROC). For NA. of models achieve near peareee AUROC.
Methods
Experimental setup
ACC
F1
AUROC
AUPRC
AVIB-TCR
RN
0.570=0.028
0.,08
0.576:t0.049
0.605-10.044
MIX-TPI
RN
0.539=0.039
0,408
0,558=00.028
0,597=40.049
Net-TCR2
RN
0.528=0.050
0.354
+0.042
0.554=+0.075
PanPep
RN
0.507=0.028
0.473=00039
0.535+00021
0.579
TEINet
0.4591..003
0.619 10.036
0.61910.038
0.535.10.022
0,581 1:0.44
TITAN
RN
0.476.10.063
0.338 10.111
0.502.1..066
0.503
AVIB-TCR
0.611100011
10.020
0.683110.010
0.815
MIX-TPI
ET
0.652
0..035
0.703:10.016
0.825-10.014
Net-TCR2
FT
0.621
0.522:10.020
0.674-10.017
0.810:10.016
PanPep
ET
0.556-2.0009
0.506
0.608
0.753-10.009
TEINet
0.356=0.008
0.512++0010
0.571+00000
0.646++0.011
TITAN
ET
0.670=0.013
0.492
0.624:t0.02
0.733-20.018
AVIB-TCR
NA
0.636=0.062
0.197
0.944
0,949+10023
MIX-TPI
NA
0.022
0.937 10,040
0.992:10.002
0.995:10.001
NCR2
NA
0.655 0.001
0.274:10.123
0.973
0.985:10.005
PanPep
NA
0.419=0.011
0.352
0.611
0.499:+0081
TEINet
NA
0.413=0.023
0.582+0.023
0.973+0.001
0.981
TITAN
NA
0.695=0.050
0,404
0.629+0.053
0.661
A.3.2. TCHARD DATASET
The TChard dataset is designed for TCR PCC-peptide/-pMHCCC binding prediction. It includes over 500,000 samples from sources such as IEDB, VDJdb, McPAS-TCR, and the NetTCR-2.0 repository. The dataset is utilized to investigate how state-of-the-ar deep learning models generalize to unseen peptides, ensuring that test samples include peptides not found in the training set. This approach highlights the challenges deep learning methods face in robustly predicting TCR recognition of peptides not previously encountered training data.
Dataset statisties. 500,000 samples
Dataset split. Cold Split referred "Hard" split in (Grazioli al., 2022b).
References. (Grazioli et al., 2022b)
Dataset license. Non-Commercial Use
Code Sample.
from ree_me.ressurcee e import DataLoader
resource
data = DataLoader (name=tchard) self.split aata.get_splic. ()
A.3.3. PANPEP DATASET
PanPep a framework constructed three levels for predicting the peptide and TCR binding recognition. We have provided the trained meta learner and external memory, and users can choose different settings based on their data available scenarios: Few known TCRs for peptide: few-shot setting; No known TCRs for peptide: zero-shot setting; plenty of known TCRs for peptide: majority setting. More information is available in the Github repo
Dataset statisties. Data from multiple studies involving millions of TCR sequences.
Dataset split. Cold Split referred to as "Hard" split in (Gao et al., 2023).
References. (Gao et al., 2023)
Dataset license. GPL-3.0
Code Sample.
from tdc_ml resource dataloader import DataLoader
26

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
data Dataloader (name=pappepp))))))) self f. split data.get_aplic
A.3.4. (YE X AL) DATASET
Affinity selection-mass spectrometry data of discovered ligands against single biomolecular targets (MDM2, ACE2, 12ca5) from the Pentelute Lab of MIT This dataset contains affinity selection-masd massonn--ssssssoom spectrometry data of discovered ligands against single biomolecular targets. Several AS-MS-discovered ligands were taken forward for experimental validation to determine the binding affinity (KD) as measured by biolayer interferometry (BLI) to the listed target protein. If listed as a "putative binder," AS-MS alone was used to isolate the ligands to the target, with KD uM required and often observed in orthogonal assays, though there is some (i 50%) chance that the ligand is nonspecific. Most of the ligands are putative binders, with 4446 total provided. For those characterized by BLI (only 34 total), the average KD is 266 nM; the median KD 9.4 nM.
Dataset statistics. 34 positive ligands, 4446 putative binders, and three proteins
Dataset Split. Stratified Split and N/A Split: We provide stratified 10/90 split train/test well as "test set only" split. References. (Ye et al., 2022; Brown et al., 2023)
Dataset license. CC BY 4.0
Code Sample.
from de ml millimmtii.. pred import ProteinPeptide data = ProteinPeption [name=-rrom__md22__ee2___2 data. data.get_split
Listing The below code illustrates how to retrieve the train, test, and val splits used for the M.l.E.....RRpttppp... benchmark from tdc_ ml.
import ICREpitopeGroup
group TcREpitopeGroopp()
train_val group.get_ttee....
test group petup.gettestt. ()
train your model and test the test set proup.evaluate. (preds)
A.3.5. TDC_ML. PWW.M...ROTTINPPPTTEO PROTEIN-PEPTIDS INTERACTION PREDICTION PROBLEM FORMULATION
PYTDC introduces the Protein-Peptice Binding Affinity prediction task. The predictive, non-generative task is to learn model estimating function of protein, peptide, antigen processing pathway, biological context, and interaction features. It outputs binding affinity value (e.g., dissociation constant Kd, Gibbs free energy AG) or binary label indicating strong or weak binding. The binary label can also include additional biomarkers, such as allowing for a positive label if and only binding interaction is specific (Brown et al., 2023; Hoofnagle & Resing, 2001: Smith Kelleher, 2013). To account for additional biomarkers beyond binding affinity value, our task is specified with binary label. Protein set. The protein includes target proteins. It is denoted by:
P = [pr.- (p PN,
(23)
where ...... PN, are Np target proteins. Information modeled for proteins can include sequence, structural, or post- translational modification data.
Peptide set. The control set includes the peptide candidates. This set is denoted as:
$ (sL, ........8... sN.}.
(24)
where $1, $1...., SN are N. candidate peptides. Information modeled for candidate peptides can include sequence, structural, and physicochemical data.
27

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Antigen processing pathway set. The antigen processing pathway set includes antigen processing pathway profile information about prior steps in the biological antigen presentation pathway processes. It is denoted by:
A (01 ; (N.)
(25)
where a1...., GN, are the Na antigen processing pathway profiles modeled. Information modeled in a profile can include proteasomal cleavage sites (Nielsen et al., 2005), classification into viral, bacterial, and self-protein sources and endogenous vs exogenous processing pathway (Reynisson et al., 2020; O' D'Doneell et al., 2020; Saxena et al., 2020; Boulanger et al., 2018), and larget/iceetoo--peccffee pathway attributes such as transporter associated with antigen processing (TAP) affinity
(Bhasin al., 2007), and endosomal//ysosomall processing efficiency (Kosaloğlu- Soogalojlu-Yalgun et al., 2020).
Interaction set. It contains the interaction feature profiles. The set is denoted by:
I
(26)
where iNN, are the N; interaction feature profiles. Information modeled in an interaction feature profile can include contact maps (Huang & Ding, 2022; Yaseen et al., 2018: Li et al., 2021: You & Shen, 2020), distance maps (Yaseen et al., 2018; Zheng et al., 2019), electrostatic interactions (Huang & Ding, 2022), and hydrogen bonds (Huang & Ding, 2022).
Cell-type-specific biological context set. contains the interaction feature profiles. The set is denoted by:
C= f=,,....,,
(27)
where CN. are the No cell-type-specific biological contexts under which the protein-peptide interaction is being evaluated. Information modeled in the cell-fype-ppeccffi biological context can include transcriptomic and proteomic data. We note, however, that, to our knowledge, single-cell transcriptomic and proteomic data has yet to be used in protein-peptide binding affinity prediction, outlining a promising avenue of research in developing machine learning models for peptide-basee therapeutics.
Protein-peptich interaction. It is a binary label, E {1,0), where = indicates a protein-peptide pair met the target biomarkers and y = 0 indicates the pair did not meet the target biomarkers.
The Protein-Peptide Interaction Prediction learning task learn a binary classification model fo estimating the probability, g. of a protein-pepticn interaction meeting specific biomarkers:
t = fo
(28)
A.4. tdc.ml. TutioOe
PDTDC introduces model framework, task definition, dataset, and benchmark for the Clinical Outcome Prediction task tailored to precision medicine. The framework and definition aim to assess clinical trials systematically and comprehensively by predicting various endpoints for patient aob-popllation. Our benchmark uses the Trial Outcome Prediction (TOP) dataset (Fu et al., 2022). TOP consists of 17,538 clinical trials with 13,880 small-molecule drugs and 5,335 diseases.
Dataset and benchmark. Our benchmark uses the Trial Outcome Prediction (TOP) dataset (Fu et al., 2022). TOP consists of 17,538 clinical trials with 13,880 mmall-molecule drugs and 5,335 diseases. Out of these trials, 9,999 (57.0%) succeeded (i.e., meeting primary endpoints), and 7,539 (43.0%) failed. Out of these trials, 1,787 were in Phase I testing (toxicity and side effects), 6,102 in Phase II (efficacy), and 4,576 in Phase III (effectiveness compared to current standards). We perform a temporal split for benchmarking. The rain/validation and test are time-split by the date January 1, 2014, i.e., the start dates of the test set are after January 1, 2014, while the completion dates of the train/validation set are before January 1, 2014. Here, the HINT model (Fu et al., 2022), benchmarked against COMPOSE (Gao et al., 2020) and DeepEnroll (Zhang et al., 2020) models. Results are shown in table 8.
Table 8. Clinical Trial Outcome Prediction task benchmark model results the TOP dataset (Fu et al., 2022), descrihed in section A.4 (Velez-Arce et al., 2024).
Model
Phase 1 AUPRC
Phase AUPRC
Phase 3 AUPRC
Indication Level AUPRC
HINT
0.772
0.607
0.623
0.703
COMPOSE
0.665
0.532
0.545
0.624
DeepEnroll
0.701
0.580
0.590
0.655
28

---

PYTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
A.4.1. TOP DATASET
TOP (Fu al., 2022) consists of 17,538 clinical trials with 13,880 small-molecule drugs and 5,335 diseases. Out of these trials, 9,999 (57.0%) succeeded (i.e., meeting primary endpoints), and 7.539 (43.0%) failed. For each clinical trial, produce the following four data items: (1) drug molecule information, including Simplified Molecular Input Line Entry System (SMILES) strings and molecular graphs for the drug candidates used in the trials; (2) disease information including ICD-10 codes (disease code), disease description, and disease hierarchy in terms of CCS codes (https://www.ht us u gov/toossootwarrrcsss 10/ces 10jjs)); (3) trial cligibility criteria are in unstructured natural language and contain inclusion and exclusion criteria; and (4) trial outcome information includes a binary indicator of trial success (I) or failure (0), trial phase, start and end date, sponsor, and trial size (i.e., number of participants).
Dataset statisties. Phase 2,402 trials/ Phase II: 7,790 trials Phase III: 5,741 trials.
Dataset split. Temporal Split as defined in (Fu et al., 2022) and Section A.4. References. (Fu et al., 2022)
Dataset license. Non-Commercial Use Code Sample.
from ddc_ml.multtee wat.immlti_pppeee
m1
import TrialOutcome
data = TrialOut (name 'phasel') ') # 'phase2' / 'phase3'
split = data data.get_split ()
A.4.2. CLINICAL TRIAL OUTCOME PREDICTION PROBLEM FORMULATION
The Clinical Trial Outcome Prediction task is formulated as a binary classification problem, where the machine learning model predicts whether a clinical trial will have a positive or negative outcome. is a function that takes patient data, trial design, treatment characteristics, disease, and macro variables as inputs and outputs a trial outcome prediction, binary indicator of trial success (1) failure (0).
Patient set. The patient set includes one or multiple patient sub-populations, with the extreme case representing personaliza- tion. It is denoted as follows:
P = [pI, PNN,hh
(29)
where PI.... .P, are Np patient sub-populations in this trial. The TOP benchmark (Fu et al., 2022) dataset represents patient data as part of the trial eligibility criteria. Patient data can include demographics (Hong et al., 2021; Lv et al., 2020; Rongali et al., 2020; Rahimian et al., 2018; Park et al., 2020), baseline health metrics (Rahimian et al., 2018; Park et al., 2020; Bedon et al., 2021), and medical history (Hong et al., 2021; Lv et al., 2020; Rongali et al., 2020; Rahimian et al., 2018; Park et al., 2020).
Trial design set. The trial design set includes this clinical trial's design profiles. It is denoted as:
D (di....,da dN.).
(30)
where di...., dN are Na eligible trial design profiles for this clinical trial. Trial design profiles can model information including phase of the trial (Fu al., 2022), number of participants, duration of the trial, trial eligibility criteria (Fu et al., 2022), and randomization and blinding methods (Schperberg et al., 2020: Wang et al., 2021b; Dai et al., 2022). Treatment set. The treatment set includes the candidate treatments for the trial. It is denoted as:
T (f1,...,,ty NN...
(31)
where l,,...,44 In. are N, candidate treatments for the clinical trial. The information modeled for treatments can include type of treatment (drug (Fu et al., 2022; Brbi et al., 2024), device (Kalscheur et al., 2018; Fujima al., 2019; van Os et al., 2018), procedure (Asadi et al., 2014; Arvind al., 2018; Senders et al., 2018; Jourahmad et al., 2023; MacKay et al., 2021)). dosage and administration route (Schperberg al., 2020; Bedon et al., 2021; Bowman al., 2023), mechanism of action (Sayaman et al., 2020; Beacher et al., 2021; Siah et al., 2019), pre-clinical and early-phase trial results (Beacher et al., 2021; Bedon et al., 2021; Beinse et al., 2019; Wang et al., 2023).
Macro context set. The macro context set contains the configurations of macro variables relevant to the clinical trial. It is denoted as:
(c1, (cI,...,cN, C..):
(32)
29

---

PYTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
where (1:.... .... CN2 are Nr configurations containing the values for macro variables relevant to the trial, which can include geography (Shamout al., 2020; Beacher al., 2021; Wang et al., 2023; Liu & Salinas, 2017) and regulatory considerations (Beacher et al., 2021; Shamout et al., 2020).
Trial outcomes. The trial outcome is binary label {1,0), where indicates the trial met their primary endpoints, while 0 means failing to meet with the primary endpoints.
The learning task to learn a model fe for predicting the trial success probability 9. where ÿ E [0, 1]:
Dit T,cE
(33)
A.5. tdc ml..sDDD Structure-Based Drug Design
Structure-based drug design aims create diverse new molecules that bind to protein pockets (3D structures) and have favorable chemical properties. These attributes evaluated using pharmaceutically-reeee an ML model learns molecular traits of protein pockets from a comprehensive dataset of protein-ligand pairs. Subsequently, potential new molecules can be generated using the acquired conditional distribution. The generated molecules must exhibit outstanding properties, including high binding effectiveness and structural variety. They must meet other user-specified criteria, such as the feasibility of synthesis (syntesiizbiltyddsiigabitt and similarity to known drugs.
oracle functions. In this task,
1. PDBBIND DATASET
PDBBind is comprehensive database extracted from PDB with experimentally measured binding affinity data for protein- ligand complexes. PDBBind does not allow the dataset to be re-distributed in any format. Thus, could not host it on the TDC server. However, provide alternative route since significant processing is required to prepare the dataset ML. The user only needs to register at http://www.pdbbin..rrgggg download the raw dataset, and then provide the local path. TDC will then automatically detect the path and transform into an ML-ready format for the TDC data loader.
Dataset statistics. 19,445 ligadd pairs
Dataset split. Random Split
References. (Liu al., 2015)
Dataset license. See note in the description on the TDC website. Code Sample.
from tdc_ml generation import SBDD data SBDD (name- 'PDBBind', path-'. /pdbbind') split data. setta.ge_splitt.
A.5.2. DUD-E DATASET
DUD-E provides a directory of valuable decoys for protein-ligand docking.
Dataset statistics. 22,886 active compounds and affinities against 102 targets. DUD-E does not support pocket extraction as protein and ligand are aligned.
Dataset split. Random Split
References. (Mysinger al., 2012) Dataset license. Not specified Code Sample.
from cdc_m1_.00 tdd_mml.enneration generation import SBDD
data SBDD (name= dude') split data. data.get__a....... get__ellt_t_teeeeeeeeeeeeelll
A.5.3. SCPDB DATASET
scPDB is processed from PDB for structure-bart -asse drug design that identifies suitable binding sites for ligaedn-ligand docking.
30

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Dataset statisties. 16,034 protein-ligand pairs over 4,782 proteins and 6,326 ligands
Dataset split. Random Split
References. (Meslamani al., 2011)
Dataset license. Not specified Code Sample.
from tdc__l generation import SBDD
data = SBDD (name='scPDS) scPDB') split data. datalget_split/ ()
A.5.4. STRUCTURE--ASSE DRUG DESIGN PROBLEM FORMULATION
Structure-based Drug Design aims generate diverse, novel molecules with high binding affinity protein pockets (3D structures) and desirable chemical properties. Oracle functions measure these properties. A machine learning task first learns the molecular characteristics given specific protein pockets from a large set of protein-ligand pair data. Then, from the learned conditional distribution, we can sample novel candidates.
Target candidate set. The target candidate set includes proteins, nucleic acids, or other biomolecules drugs can interact with, producing a therapeutic effect or causing a biological response. It is denoted by:
T = (1,,...,ANA NN.):
(34)
where fi,..., iN, are N₂ target candidates for the evaluated set of drugs. Information modeled for target candidates can include interaction, structural, and sequence information. Ligand candidate set. The ligand drug candidate set includes the drug molecules being tested for particular therapeutic effect or biological response. It denoted by:
L IN.
(35)
where l,...., IN. are the Nj ligand/drug molecules being evaluated. Drug modeling can include molecular structure, often represented in formats such as SMILES (Simplified Molecular Input Line Entry System) or InChl (International Chemical Identifier) (Öztürk et al., 2019), physicochemical properties like hydrophobicity and molecular weight (Chen et al., 2020b), and molecular descriptors and fingerprints (Wang et al., 2019).
Scoring function. The scoring function, denoted by S, evaluates the binding affinity of ligand E IL to protein target E T. Drug-likeness function. Function representing the drug-likeness of ligand I E L,, including properties like solubility, stability, and toxicity.
The generative learning task is to generate the ligand / E LL maximizing binding affinity. Sp, and drug-likeness, fo. Given a loss function, ()), for t E T and E L, the first step is to learn model Ma s.t.,
Mg
(LoniingLoss(SAnn)) (), fo))).
(36)
This is followed by the ligand optimization step, which optimizes the ligand for maximum binding affinity and drug-likeness given the trained model. ligand optimization function, F, such as addition or multiplication, is used for the optimization:
argmax err [F(S((t (F E T,T). TT)).
(37)
An example formulation would be as follows:
= argmax (So(t (S
T,() x (o)-
(38)
B. PyTDC Architecture
B.I. API-First Design and Model-View- View-Controllerr
See section 3.1 in the main text. PYTDC (Velez-Arce et al., 2024) drastically expands dataset retrieval capabilities
available in TDC-1 beyond those of other leading benchmarks. Leading benchmarks, like MoleculeNet (Wu al., 2018) and
31

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
TorchDrug (Zhu et al., 2022) have traditionally provided dataloaders access file dumps. PYTDC introduces APl-integrated multimodal data-views (Churi al., 2016; Emmerik et al., 1993; Rammerstorfer & Mössenböck, 2003). To do so, the software architecture of PyTDC was redesigned using the Model-Viee-Conttee Malik et al., 2021). The MVC architecture separates the model (data logic), view (UI logic), and controller (input logic), which allows for the integration of heterogeneous data sources and ensures consisteney in data views (Churi et al., 2016). The MVC pattern supports the integration of multiple data modalities by using data mappings and views (Rammerstorfer Mössenböck, 2003). The WVC-enabled-muliinon retrieval API is powered by PyTDC's Resource Model (Section B.2).
(MVC) design pattern (Bucanek, 2009;
B. TDC DATALOADER (Model)
As per the TDC-1 specification, this component queries the underlying data source to provide processed data to upstream function calls. We augmented this component beyond TDC-1 functionality to allow for querying datasets introduced in PyTDC, such as the CZ CellXGene.
B.2.2. TDC MEANINGFUL DATA SPLITS AND MULTIMODAL DATA PROCESSING (View)
As per the TDC-1 specification, this component implements data splits to evaluate model generalizability to out-of- distribution samples and data processing functions for multiple modalities. We augmented this component to act on data views (Churi et al., 2016) specified by PyTDC's controller.
B.1.3. PYTDC DOMAIN-SPECIFG LANGUAGE (Controller)
PyTDC develops an Depiication-Embeddee Domain-Specific Data Definition Programming Language facilitating the integration of multiple modalities by generating data views from a mapping of multiple datasets and functions for transfor- mations, integration, and multimodal enhancements, while mantaining a high level of abstraction (Membarth et al., 2016) for the Resource framework. We include examples developing multimodal datasets leveraging this MVC DSL in listing 4.
B.2. Resource Model
The Commons introduces a redesign of TDC-1* I's dataset layer into a new data model dubbed the PyTDC resource, which has been developed under the MVC paradigm to integrate multiple modalitics into the API-first model of PyTDC.
B.2.1. CZ CELLXGENE WITH SINGLE CELL BIOLOGY DATASETS
CZ CellXGene (CZI Single-Cell Biology, et al., 2023) is open-source platform for analysis of single-cell RNA sequencing data. We leverage the CZ CellXGene to develop PyTDC Resource Model for constructing large-scale single-cell datasets that maps gene expression profiles of individual cells across tissues, healthy and disease states. PyTDC leverages the SOMA (Stack of Matrices, Annotated) API, adopts TileDB-SOMA (Papadopoulos et al., 2016) for modeling sets of 2D annotated matrices with measurements of features across observations, and enables memory-effiiient querying of single-cell modalities (i.c., scRNA-seq, meq), across healthy and diseased samples, with tabular annotations of cells, samples, and patients the samples come from.
We develop remote procedure call (RPC) API taking the string name (e.g., listing in section B.3.1) of the desired reference dataset as specified in the CellXGene (CZI Single-Cell Biology, et al., 2023). The remote procedure call for fetching data is specified as a Python generator expression, allowing the user to iterate over the constructed single-cell atlas without loading it into memory (Tasoluk & Tanrikulu, 2023). Specifying the RPC as a Python generator expression allows us to make use of memory- querying as provided by TileDB (Papadopoulos et al., 2016). The single cell datasets can be integrated with therapeutics ML workflows in PyTDC by using tools such as PyTorch's IterableDataset module (Paszke et al., 2019).
Knowledge graph, external APIs, and model hub. We have developed framework for biomedical knowledge graphs to enhance multimodality of dataset retrieval via PyTDC's Resource Model. Our system leverages PrimeKG to integrate 20 high-quality resources to describe 17,080 diseases with 4,050,249 relationships (Chandak et al., 2023). Our framework also extends to external APIS, with data views currently leveraging BioPython (Cock et al., 2009), for obtaining nucleotide sequence information for a given non-coding RNA ID from NCBI (Cock et al., 2009), and The Uniprot Consortium's 's RESTful GET API (Consortium, 2021) I) for obtaining amino acid sequences. In addition we've developed the framework to allow access to embedding models under diverse biological contexts via the PyTDC Model Hub. Examples using these
32

---

PYTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
components in sections B.3.2 and B.3.3.
B.3. API-first design code samples
We provide code samples for the components described in sections 3.1, B. and B.2.
Listing above configuration augments protein-peptide dataset with an additional modality, amino acid sequence, and invokes numerous data processing functions tailored the specific needs of the underlying dataset. Added information for this demonstration be found There more complex workflows implemented for current TDC dataviews and all such views leveraging the DSL can be found in the repo
from config import DatasetConfig
from feature_ peature_generators..
frrottiinfeatuur__eeeerrrr import
4 ProteinFFatuureseneraooo
class Brw.aPrresiieeerreeoooootttttttt
http://awoo....o.ooo...ooorrrrrn datsses
def (init_(selff)
super (Brompproteell@@eetttcCCCoo s init_
dat adteppocesiiccassss.cittttttteee
functions_to_run==
autofill_icentti create_range, insert_rrotii_seeueen )
args__or_fuuncioos==
cnnoffillcooumn Name,
key_column: Sequence,
column: (nM) keys: (inder)
subs: [0]
gene_column: Protein_Target
}1,
var [ar_map=i
X1: Sequence,
x2: Battein_o__raasequeeee
ID1: Name,
ID2: Protein,Target,
)
B.3.1. PYTDC MULTIMODAL. SINGLE-CEIL CELLL RETRIEVAL API
focus on the use case of an ML researcher who wishes to train model on a large-scale single-cell atlas. particular, researchers would be familiar with and have trained models on traditional single-cell datasets such as Tabula Sapiens (Jones et al., 2022). Their interest is scale model by training it on more extensive single-cell atlas based on this reference dataset. We build such API. Specifically, given reference dataset available in CellXGene Discover (CZI Single-Cell Biology, et al., 2023), we allow the user perform memory-effiiiee query using TileDB-SOME to expand the reference dataset to include cell entries with non-zero readouts for any of the genes present in the reference dataset. This allows users to build scrge-scale single-cell atlases on familiar reference datasets. The example below illustrates how a user may construct a turge-scale atlas with Tabula Sapiens as the reference dataset. Other use cases include augmenting datasets using knowledge graphs and cell-type-specifie biomedical contexts. These capabilities are all powered by the
33

---

PYTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Model-View-Conrrone framework (section B.
Listing example below illustrates how a user may construct large-scale atlas with Tabula Sapiens the reference dataset using the PYTDC CELLXGENE API.
from tdc__m_ multi perett_peddd___ee pred. single_cell import Cel1XGene
dataloader Cel1XGene (nam=-rabula_Sapieen -aannns____1____ gen = dataloader. get_ value_filter www.fffiitttssseeee - 'brain' Land = 'male'
df next (gen)
Listing 6. In addition to our PYTDC DataLoader API implementation for the CelIXGene RPC API, we provide a simplified wrapper over the CellXGene Census Discovery API, which allows users to perform remote procedure calls to fetch Cell Census data in more machine- machine-learinng learning-friendly formats like Pandas and Scipy. We also maintain support for the AnnData format. Users can query Cell Census counts as as metadata usine this APL. The code sample below illustrates such usage.
from tdc_ml rescurce import cell mellxgene_ccnsee census
initialize Census Resource and query filters resource htt.aaeeneeeensussseesssssssssee
cell_value_filtee = tissue_== 'brain' _ar____an_________________ _and_sex_==_ 'malle' cell cell_column_names = [assay, cell_type, tissue]
'male'
Obtaining cell metadata from the cellxgene census in pandas format obsdf resource, dissurce..ett.cel_mettcoe.
vatie__fiierrreell_aaaaaffeeee columa_ fmt=pandas)
B.3.2. PRIMEKG KNOWLEDGE GRAPH
PrimeKG supports drug-disease prediction by including an abundance of 'indications." 'contradictions', and 'off-label use' edges, which are usually missing in other knowledge graphs. We accompany PrimeKG's graph structure with text descriptions of clinical guidelines for drugs and diseases to enable multimodal analyses (Chandak et al., 2023). The code below depict example use cases of the PyTDC PrimeKG API. Demonstrations are additionally available in wttpa//..ggo.o...ooeoeeeeee/aaan
Listing We illustrate here example utilities for retrieving drug-target-disease associations using the PyTDC PrimeKG API
from tdc ml. resource import PrimeKG
pkg = PrimeKG ()
pkgdf =
def get_all_drug_evidee (disease)
given, ; disass,,
all,,duuge int.racciinng..en........ with, proteins,,reeevant relevant, to,
disease
prots pkgdfl ( (pkgdf
[relation] disease_protein) (pkgdf [x_name] disease
ma_namee.uuiqqe
drugs = p (pkgdd (relation] == drag_protein) & [y_name isin (prots)
relations drugs ddipplayrreaation unique
out = {}
for rel in relations:
out [rel] drugs (drugs (display_relationnn rel] [x_name] .niqqe
34

---

PYTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
return
def get_all_sssccaaeeetttrett (disease):
return pkgdf [ (pkgdf (relation]] == disease_proteing & (pkgdf [x_name] == disease)
+ I[lly_amm,, display_relation
def get_odease_.deesessssciiitss (disease)
return pkgdfl (pggdf disease_disease (pkgdf [x_name]
disease)
[[y_name,, display_reiation]]
def diseases =
(disease):
(disease) [y_name]
out = set ()
for d in diseases:
targets get ppt_all_asseon aaa__assciated__aareeee (d) [y_name] unique ()
out uut.updat (targets)
return 1ist (out)
def return [ [relation] == disease_proteing & (pkgdf [x_name] [.mme].s. str.
:
contains (kw, case=False, na=False)) ] [x name] ..rjj.umn unique ()
if _anaame__ - main_
= all dww.dssssse_____eee_ee_ee_eee__e by keyword (d) for d in x]
[get_all_asociiatee_tareee (d) for d in x]
[get_disseee_ddsssssssssctttttte (d) for d in x]
fromee_s__or_ee__eeeee____eeeeee (d) for in x])
Listing Here we illustrate comhinig the PyTDC PrimeKG API with the networkx module retrieve drug repositioning opportunities.
import networkx as nx
from tdc_ml, resource import PrimekG
Load the PrimekG data kg = PrimekG() ()
data = kg. get_data ()
data = data ( wtt.eeaaatntntt.tooooooooeeeee contains (drug)
Create a graph from the knowledge graph data
nx. wx..fon__andas_edeee
relation') (data, 'x_id', 'y_name', edge_attr /
Example function to find repositioning opportunities for given drug def neighbors list (G. nG.neighbors (drug))
(drug):
diseases = [node for node in neighbors if G[drug] [node] ('relation'] = 4 drug_protein'
return diseases
Find repositioning opportunities for specific drug drug_name = 'DB00945' aepositlooiing_opportttttet
fha.rrppaatiioounnppoortttttttee (drug_name)
35

---

PYTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
B.3.3. PYTDC MODEL HUB
The introduced model hub is composed of the PyTDC Model Hub and set of utilities and endpoints for facilitating model inference and fine-tuning, PyTDC introduces The Commons' HuggingFace Model Hub. It is resource with pre-trained models, including geometric deep learning models, large language models, and other contextualized multimodal models for therapeutic tasks. The models can be fine-tuned using datasets in PyTDC and be used for downstream tasks such as implementations of multi-agent collaborative schemes (Gao et al., 2024) (i.e., expert consultants) our predictive therapeutic tasks (Theodoris et al., 2023; Initiative, 2023). The model hub details and available models can be found at https//huggingffcccooltd
Listing 9.
The below illustrates the basic functionality of the model hub to download a model and perform inference on a precdictive task
as well as fine-tune the model
from tdc_ml import
Idc hf tdc hf interface (BBB Martins-AttentiveFF)
load deeppurpose model from this rеpo
up_model tdc_hf load deeppurpose (' /data')
aal_hh..redicttddeee.
(dp_model, ['YOUR, SMILES, SMILES,STRINGG
# fine-tune ap_model. (train, val, test) for some defined splits
Listing 10. The below illustrates using the tde model hub to download foundation model (Theadoris et al., 2023)
from tdc_ml import
from transformers import BertModel
geneformer tdc_hf_interfer (Geneformer)
model geneformer. () assert isinstance (model, BertModel), type (model)
Listing 11. Beyond downloading foundation model (Thoodoris et al., 2023), the model server facilitates model inference across a range datasets. Below an example integrating PYIDC CelIXGene API model server.
from tdc_ml, resource import cel xellxxene_census from
www.nizers.s..ooo.oo.....o. gen.former import GeneformerTokeni from tdc import idd_hf_interface import torch
# query the CELLXGENE census
adata = relffresource.get.coom anndata var_ har_walue_filter.
feature_id_in_
(Tt.:/01111.1.............0.... 'ENSG0000188829"1, si__value_filter==
sex sex_-=_ ''emalee sedeele___ddeeel_yye___._e hng.goooppppp_nttttooogalal.ell 'neuron'l,
column_namese
obs: [
assay, cell_type, tissue, tissue_general, suspension_type, disease
# tokenize gene expression vectors tokenizer = GeneformerrTkenizzer())
x tokenizer. tokemizer.tokeeiaa_.cllll vectors
vectors (adata,
36

---

PYTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
ncourts=n_measuredd
cells, =
# load the model
geneformer tdc_hf (Geneformez)
model geneformer. load load
()
Custom, pww processin code,,cam, include, padding, and, attention, mask definitions.
input_tensox forch.tenson (cells)
out []
for batch in input_tenser
build an attention mask
attention mask torch.tensor
[[x[0] != 0, x != 0] for x in batch])
run batched inference
out. (model (batch, attentionn_askattteniioooo
C. PyTDC model server: models and code
We have released open source model retrieval and deployment software that streamlines AI inferencing, downstream fine-tuning, and domain-specific evaluation for representation learning models across biomedical modalities. The PyTDC model server (section 3.2) facilitates the use and evaluation of biomedical foundation models for downstream therapeutic tasks and enforces alignment between new method development and therapeutic objectives. We have released support for inference several state-of-the-ard (SoTA) models (Lopez et al., 2018; Theodoris al., 2023; Cui et al., 2024; Li et al., 2024; ESM Team, 2024) well as benchmarking software for 2024).
evaluation (Velez-Arce et al.,
PYTDC presents open source model retrieval and deployment software that streamlines AI inferencing and exposes state- of-the-art, research-ready models and training setups for biomedical representation learning models across modalities. In section 3.2 and figure 2, we describe the architectural components of the model server and its integration into the model development lifecycle with the TDC platform. In figure we show the components of PyTDC enabling the full development lifecycle for transfer learning MI methods in therapeutics. Figure 3 also shares how workflow spanning hundreds-to-thousaadd of lines of code is simplified to below 30 with PyTDC's model server and data retrieval capabilities.
We present PyTDC, maahhne--earming platform providing streamlined training, evaluation, and inference software for single- cell biological foundation models to accelerate research transfer learning method development therapeutics (Theodoris, 2024). PyTDC introduces an API-first (Beaulieu et al., 2022) architecture that unifies heterogeneous, continuously updated data sources. The platform introduces model server, which provides unified access model weights across distributed repositories and standardized inference endpoints. The model server accelerates research workflows (Kahn al., 2022) by exposing state-of-the-are research-ready models and training setups for biomedical representation learning models across modalities. Building upon Therapeutic Data Commons (Huang et al., 2021; 2022; Velez-Arce et al., 2024), we present single-cell therapeutics tasks. datasets, and benchmarks for model development and evaluation.
Below the set of model server models.
C.1. scGPT
scGPT (Cui et al., 2024) is A foundation model for single-cell biology based on a generative pre trained transformer across a repository of over 33 million cells.
Model page. http:://wgginggecc.coo///pploo/p
Abstract. Generative pretrained models have achieved remarkable success various domains such as language and
37

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
computer vision. Specifically, the combination of large-scale diverse datasets and pretrained transformers has emerged promising approach for developing foundation models. Drawing parallels between language and cellular biology (in which texts comprise words; similarly, cells are defined by genes), study probes the applicability of foundation models to advance cellular biology and genetic research. Using burgeoning single-cell sequencing data, we have constructed foundation model for single-cell biology, ScGPT, based on a generative pretrained transformer across repository of over 33 million cells. Our findings illustrate that scGPT effectively distills critical biological insights concerning genes and cells. Through further adaptation of transfer learning, scGPT can be optimized achieve superior performance across diverse downstream applications. This includes tasks such cell type annotation, multi-bate integration, multi-omic integration, perturbation response prediction and gene network inference.
PyTDC Code. See https::/hugginnffcccco/// http://nnn.accccaappooo/////// Illustrated in figure 4.
Figure 4. The below code illustrates workflow using PyTDC's implementation of (Cui et al., 2024). Such workflow would take hundreds-to-thousane of lines of code to develop. PyTDC allows the user to extract single-cell foundation model embeddings from complex and eustomized gene expression datasets with less than 30 lines of code. See https:/huuggingfac..ooolllllssss
from tdo ml. multi preeddpeeeed___ anndata dataset import DataLoader
from tdc ml import tde_ hf interface
from tdc_ml.. loddmmeserrveeee tokenizers ronee scgpt import SCGPTTokenizer import torch
an example dataset adata Datalicader ("cel
"./data",
dataset names= """elllllllll"" sampe saaaeee___mpe__mma__"__
no_convert- -True) adata
code for loading the model and performing inference ("scGPT")
sogpt idc_ht_inters This line can cause segmentation fault on inappropriate setup model scgpt.lcadl)
tokenizer scGPTTokenizer()
gene_ids - adata.var (atttaavrr"eaatueeeaaaaa
to_numpy
& Convert to numpy array
tokenized_dats = tokenizer. wowprrtteeiiiieeell.eleo vectors
adata. tet.aryy), gene_ids)
mask torch. tensor [x != for x tokenized [0] [1]],
dtype-torch.bon
Extract first embedding
first_embed embed = model (tokenized [aa [0],
tokenized_data [0] [1],
attention macioon_mskkmmassa
C.2. Geneformer
Geneformer (Theodoris et al., 2023) a foundational transformer model pretrained on a large-scale corpus of single cell transcriptomes enable context- aware predictions in settings with limited data network biology.
Model page. http://gggiiggaecoo////////reeee
Abstract. Mapping gene networks requires large amounts of transcriptomic data learn the connections between genes, which impedes discoveries in settings with limited data, including rare diseases and diseases affecting clinically inaccessible tissues. Recently, transfer learning has revolutionized fields such as natural language understanding and computer vision by leveraging deep learning models pretrained on large-scale general datasets that can then be fine-tuned towards a vast array of downstream tasks with limited task specifcc data. Here, we developed a context-aware, attention iateetiio-bbaedd deep learning model, Geneformer, pretrained on large-scale corpus of about 30 million single-cell transcriptomes to enable
38

---

PYTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
context-specifie predictions in settings with limited data in network biology. During pretraining, Geneformer gained fundamental understanding of network dynamics, encoding network hierarchy in the attention weights of the model in completely self-supervised manner. Fine-tuning towards diverse panel of downstream tasks relevant to chromatin and network dynamics using limited task-specific data demonstrated that Geneformer consistently boosted predictive accuracy. Applied to disease modelling with limited patient data, Geneformer identified candidate therapeutic targets for cardiomyopathy, Overall, Geneformer represents pretrained deep learning model from which fine-tuning towards a broad range downstream applications can be pursued to accelerate discovery of key network regulators and candidate therapeutic targets.
Code. See http://hungginiieeecoolppllloooo Illustrated in figure 5
Figure 5. The below code illustrates a workflow using PyTDC's implementation of (Theodoris al., 2023). Such workflow would take a user hundreds-tor to- thousands of lines code to develop. PyTDC allows the user to extract single-cell foundation model embeddings from complex and customized gene expression datasets with less than 30 lines of code. See
from tdc ml. model server. tokenizere geneformer reneerrrersggeeeeoomme import GeneFormerrTkennizee from tdc ml import tdc_ hf interface
import torch
Retrieve anndata object. Then, tokenize
tokenizer GeneformerTokeniierr
peeenizer.tckeiig cell vectors (adata,
enserti_ido""ffatire_jdd"
acounts="n (oouurrs""n
cells,
input_tensor = forch.tensen (cells) note that you may need to pad or perform other cust om data processing
retrieve model
geneformer joo_hf_interfacee ("Geneformer")
model - geneformer.lode ()ad
run inforence
aattention mask (
[[x[0] 0, x[1] for input_tensor)) # here we assume used 0/False as special padding token
outputs model (batch,
output_hidden staeeessttttttttteeeeeee True)
layer_to quant quant_layere (model)
Geneformer's second-to-last layer is most generalized
embs_i outputs. hidden_stated [layer_to_quant]
there are "cls", "cell", and "gene" embeddings. will only capture "gene", which is cell type speci fic. for "cell", you'd 'd average out across unmasked gene embeddi ngs per cell
embs embs
C.3. PINNACLE
PINNACLE (Li et al., 2024), a flexible geometric deep-learning approach that is trained on contextualized protein interaction networks to generate context-PINNACIEE protein representations. Leveraging human multi-organ single-cell transcriptomic atlas, PINNACLE provides 394,760 protein representations split across 156 cell type contexts from tissues and organs.
Model page. http://hugginnffeeeoooolllllllpp
39

---

PyTDC: A multimodal machine learning training, evaluation, and inference platform for biomedical foundation models
Abstract. Understanding protein function and developing molecular therapies require deciphering the cell types in which proteins act well the interactions between proteins. However, modeling protein interactions across biological contexts remains challenging for existing algorithms. Here we introduce PINNACLE, geometric deep learning approach that generates context-aware protein representations. Leveraging multiorgan single-cell atlas, PINNACLE learns on contextualized protein interaction networks to produce 394,760 protein representations from 156 cell type contexts across 24 tissues. PINNACLE's embedding space reflects cellular and tissue organization, enabling zero-shot retrieval of the tissue hierarchy. Pretrained protein representations can be adapted for downstream tasks: enhancing 3D structure- representations for resolving cammuooonncolggcee protein interactions, and investigating drugs' effects across cell types. PINNACLE outperforms state-of-the-art models in nominating therapeutic targets for rheumatoid arthritis and inflammatory bowel diseases and pinpoints cell type contexts with higher predictive capability than context-free models. PINNACLE's ability to adjust its outputs on the basis of the context in which it operates paves the way for large-scale context-specific predictions in biology.
Code. See http::/hwhgiinngaaeccc//ppiiihia
Figure The below code illustrates workflow using PyTDC's implementation of (Li et al., 2024). See huts://hugingfacccco/piiks PINNACLE
from tde_ml. resource. pinnacle import PINNACLE
pinnacle = PINNACLE()
embeds pinnacle. get_embeds () extract embeddings dataset
C.4. scVI
Single-cell variational inference (scVI) (Lopez al., 2018) is powerful tool for the probabilistic analysis of single-cell transcriptomics data. It uses deep generative models address technical noise and batch effects, providing robust framework for various downstream analysis tasks. To load the pre-trained model, use the Files and Versions tab files.
Code. See https//hhngiinnffeeccco/aiiiiti
Figure 7. The below code illustrates workflow using PyTDC's implementation of (Lopez et al., 2018). See https//huggingfacccoo/pikkkk scVI
from tdo ml wwt.m..mutt... Brcec___redd.am anndata dataset import DataLoader from tdc import tf interface
adata
DataLoader ("scvi_tedd dataaset_
"./data",
| "sevi_ "est_dataset"]
no_convert-True) adata
scvi tdc hf interface ("soVI")
model scvi (cad
output model (adata)
40


Reviews:
[{"review_id": "QdIPf9cwf1", "reviewer": "Reviewer_ATdV", "summary": "PyTDC is introduced as a cutting-edge multimodal machine learning infrastructure designed to streamline the training, evaluation, and inference of biomedical foundation models. By unifying heterogeneous, continuously updated data sources and providing a model server for seamless access to pre-trained models and inference endpoints, PyTDC addresses the fragmentation in existing biomedical benchmarks. It supports a wide range of therapeutic tasks, including single-cell drug-target nomination, perturbation response prediction, and protein-peptide interaction prediction, while introducing context-specific metrics to ensure model performance aligns with biomedical goals. As an open-source platform, PyTDC accelerates research by offering modular, customizable tools for multimodal data integration, enabling the development of robust, context-aware foundation models for biomedical AI.\n\n## Update after rebuttal\nMy Overall Recommendation remains the same.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ATdV,\nThank you very much for your time dedicated to reviewing our work. Including our responses to your queries below.\n\n**PyTDC doesn't include text data, yet natural language is a key modality in biomedicine. Is it appropriate to claim comprehensive support for biomedical foundation models without this modality?**\n\nThis is fair. Supporting biomedical foundation models is an aspirational goal in this case. We focus this release on single-cell foundation models and are working to add ESM3 to incorporate the protein sequence modality. We could more strongly emphasize this point, or adjust our claim/headline to match “foundation models in therapeutic discovery”.\n\n**As a non-commercial platform, how does the group implement the continuous update of new data?**\n\nThis is addressed via an API-first approach. By invoking external APIs rather than storing static data, we assert up-to-date retrieval. The chosen MVC design pattern and the resulting “data view” allow us to perform data manipulation per dataset definition and, as such, obtain an ML-ready dataset fitting the task or resource definition. \n\n**As an open-source, API-based platform, the user experience of PyTDC will require more feedback from future users to fully evaluate its effectiveness and usability.**\n\nThis is true. It is hard to fully and rigorously assess the success of this release. However, we have some very positive indicators, including a sharp increase in Github stars (now 1000+), a 2x increase in the Pypi package's MAU, and ~5k invocations of the model server APIs in the last month alone.\n\n\n**What managing mechanism is adopted for continuous data updates in PyTDC?**\n\nOur platform emphasizes an API-first approach rather than storing static data, thus ensuring up-to-date retrieval. However, for static datasets, we cannot yet guarantee continuous data updates.\n\n**How does PyTDC manage the use of data with various licenses? For example, if a foundation model is trained using data accessed through PyTDC from sources with different licenses, how does the platform supervise and ensure compliance with these licenses?**\n\nWe don’t yet have rigorous licensing enforcement. However, we expose all licenses for datasets on our website and urge users to review permits for the datasets they are using.", "claims_and_evidence": "All the claims in the paper are sound with corresponding contributions in the platform. However, there are two aspects to be clarified:\n\n- PyTDC doesn't include text data, yet natural language is a key modality in biomedicine. Is it appropriate to claim comprehensive support for biomedical foundation models without this modality?\n- As a non-commercial platform, how does the group implement the continuous update of new data?", "methods_and_evaluation_criteria": "The platform is highly meaningful, providing essential support for the development of biomedical foundation models. The dataset collection, benchmark organization, metric design, and API architecture are well-executed and demonstrate the platform's utility.", "theoretical_claims": "None.", "experimental_designs_or_analyses": "This paper is not based on comparative experiments and analyses. As an open-source, API-based platform, the user experience of PyTDC will require more feedback from future users to fully evaluate its effectiveness and usability.", "relation_to_broader_scientific_literature": "PyTDC provides a fundamental toolkit for the development of biomedical foundation models, addressing a critical gap in the field. The platform's design and capabilities are well-aligned with the needs of modern biomedical AI research.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "The contribution of PyTDC is groundbreaking in this field, offering a comprehensive and modular solution for biomedical AI research.", "other_comments_or_suggestions": "None.", "questions_for_authors": "- What managing mechanism is adopted for continuous data updates in PyTDC?\n- How does PyTDC manage the use of data with various licenses? For example, if a foundation model is trained using data accessed through PyTDC from sources with different licenses, how does the platform supervise and ensure compliance with these licenses?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IObsOHaYBA", "reviewer": "Reviewer_exTK", "summary": "The paper introduces **PyTDC**, a multimodal machine learning platform designed for training, evaluation, and inference of biomedical foundation models. The platform aims to address the limitations of existing biomedical benchmarks by providing end-to-end infrastructure for integrating multimodal biological data and supporting a broad range of machine learning tasks in therapeutics. Key contributions of PyTDC include: (1) integration of single-cell analysis with multimodal machine learning, (2) continuous data updates and heterogeneous data sources, (3) model server for inference and fine-tuning, (4) case study on single-cell drug-target nomination, and (5) context-specific metrics. Overall, PyTDC aims to accelerate research in biomedical AI by providing a unified platform for multimodal data integration, model training, and evaluation, with a focus on therapeutic applications. The platform is open-source and designed to facilitate the development of context-aware, multimodal foundation models for biomedical research.\n\n## Update after Rebuttal\nI agree to accept this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer exTK, thank you very much for your time dedicated to reviewing our work. Including responses to your queries below.\n\n**The only concern for me is that the target of PyTDC is supporting \"biomedical foundation models\", which may be somewhat overclaiming. Biomedicine encompasses a vast range of disciplines, for which various tasks, datasets, and benchmarks have been proposed. There have also been many models claimed to be \"biomedical foundation models\", which actually focus on very different applications, such as clinical NLP, medical image processing, and drug discovery. Therefore, as an AI platform, PyTDC can hardly cover the whole field of \"biomedical foundation models\".**\n\nThis is true. Supporting biomedical foundation models is an aspirational goal in this case. We focus this release on single-cell foundation models and are working to add ESM3 to incorporate the protein sequence, structure, and function modalities. We can either make a stronger note of this limitation or rephrase our claim to be “foundation models in therapeutic discovery”.\n\n**For the structure-based drug design component of PyTDC, beyond the scoring functions already included, there are two recent popular aspects to be considered in evaluating the molecules:\nThe physical plausibility of 3D structures, such as PoseCheck [1].\nThe diversity of generated molecules, new metrics including NCircles [2] and HamDiv [3].\nThe SBDD component may not be the main contribution of PyTDC, but as a comprehensive benchmarking platform, I believe these metrics would help.**\n\nWe agree. We are still working on the final release of the codebase, which could be released simultaneously to a camera-ready version. This suggestion has been added to the list of tasks for that release.\n\n**Tiny improvements in writing could be made:\nIn Section 4.3, \"PPI\" and \"ppi\" are mixedly used.\nSome citations in-line act as components of sentences, where \\citet should be used instead of \\citep. For example, the last line of page 7.\n\"Table\" vs. \"table\", \"Figure\" vs. \"figure\", \"Section\" vs. \"section\" are mixedly used in many places, which should be unified.**\n\nThis will be done for any camera-ready version.\n\n**How many single-cell data items are there in the dataset? Is this amount enough for training a robust model?**\n\nOur website and appendix show the total number of items for all datasets introduced in the paper. For the case study on the single-cell drug target nomination task, we have 394760 (cell, gene) pairs in the dataset. We will mention this explicitly when we release the documentation for the pinnacle resource. This dataset was large enough to train PINNACLE and Node2Vec to satisfactory results. It is possible a larger dataset would have yielded better performance from the tested transformer models. However, we note the dataset presents what is realistically feasible to curate for these diseases based on the current literature.", "claims_and_evidence": "As an infrastructure paper, the claims of contributions of PyTDC are concrete. The only concern for me is that the target of PyTDC is supporting \"biomedical foundation models\", which may be somewhat overclaiming. Biomedicine encompasses a vast range of disciplines, for which various tasks, datasets, and benchmarks have been proposed. There have also been many models claimed to be \"biomedical foundation models\", which actually focus on very different applications, such as clinical NLP, medical image processing, and drug discovery. Therefore, as an AI platform, PyTDC can hardly cover the whole field of \"biomedical foundation models\".", "methods_and_evaluation_criteria": "The PyTDC platform offers multiple good features for such kinds of infrastructure, including open-source datasets and plug-and-play APIs. The comparisons in Table 1 demonstrate the significant contribution of PyTDC.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "The only experiments in Section 4.3 are completely newly built and act as a case study rather than the main contribution of this paper.", "relation_to_broader_scientific_literature": "PyTDC reviews related works in detail, and its contribution compared to the existing works is significant.", "essential_references_not_discussed": "For the structure-based drug design component of PyTDC, beyond the scoring functions already included, there are two recent popular aspects to be considered in evaluating the molecules:\n\n1. The physical plausibility of 3D structures, such as PoseCheck [1].\n2. The diversity of generated molecules, new metrics including NCircles [2] and HamDiv [3].\n\nThe SBDD component may not be the main contribution of PyTDC, but as a comprehensive benchmarking platform, I believe these metrics would help.\n\n[1] Harris C, Didi K, Jamasb A, et al. PoseCheck: Generative Models for 3D Structure-based Drug Design Produce Unrealistic Poses[C]//NeurIPS 2023 Generative AI and Biology (GenBio) Workshop.\n\n[2] Xie Y, Xu Z, Ma J, et al. How Much Space Has Been Explored? Measuring the Chemical Space Covered by Databases and Machine-Generated Molecules[C]//The Eleventh International Conference on Learning Representations.\n\n[3] Hu X, Liu G, Yao Q, et al. Hamiltonian diversity: effectively measuring molecular diversity by shortest Hamiltonian circuits[J]. Journal of Cheminformatics, 2024, 16(1): 94.", "other_strengths_and_weaknesses": "None. See above parts.", "other_comments_or_suggestions": "Tiny improvements in writing could be made:\n\n1. In Section 4.3, \"PPI\" and \"ppi\" are mixedly used.\n2. Some citations in-line act as components of sentences, where \\citet should be used instead of \\citep. For example, the last line of page 7.\n3. \"Table\" vs. \"table\", \"Figure\" vs. \"figure\", \"Section\" vs. \"section\" are mixedly used in many places, which should be unified.", "questions_for_authors": "1. How many single-cell data items are there in the dataset? Is this amount enough for training a robust model?\n2. Will the PyTDC API be freely available to academic researchers?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xHtljuTNSL", "reviewer": "Reviewer_xzws", "summary": "This paper introduces PyTDC, an open-source machine learning platform designed to support the training, evaluation, and inference of multimodal biological AI models, with a strong focus on the single-cell domain. PyTDC facilitates integration of diverse biological data sources from single-cell gene expression, perturbation responses, protein-peptide interactions, and clinical trial data into contextualized machine learning tasks. Key tasks include single-cell drug-target nomination, perturbation response prediction, and cell-type-specific protein-peptide interaction prediction.\n\nA central contribution is the adoption of an API-first architecture, implemented using a Model-View-Controller (MVC) design pattern. This architectural choice allows seamless integration of heterogeneous and continually updated data sources.\n\nThe paper also introduces a benchmarking and model retrieval infrastructure, which allows researchers to evaluate state-of-the-art models, fine-tune them on task-specific data, and assess performance on out-of-distribution samples. Notably, the paper highlights the poor performance of existing graph-based and domain-specific methods on the newly introduced tasks, particularly in out-of-distribution contexts, underscoring the need for context-aware, multimodal foundation models in the biomedical domain.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**However, the necessity for adopting the specific 'API-first' architecture and Model-View-Controller (MVC) design pattern are underexplained. While these approaches are well-established in software engineering, the paper does not clearly justify why they are uniquely advantageous for this use case compared to alternative architectures, including comparison with the previous versions. Further explanation is needed to clarify how this design improves the implementation of the pyTDC server, in perspectives including modularity, scalability, or reproducibility.**\n\nWe ensure up-to-date retrieval by invoking external APIs rather than storing static data. The chosen MVC design pattern and the resulting “data view” allow us to perform data manipulation per dataset definition and, as such, obtain an ML-ready dataset fitting the task or resource definition. The single-cell gene expression data retrieval API and the shown DSL-based dataset are examples of data views leveraging this pattern, which we can more clearly emphasize in the paper.\n\n**Minor comments:\nThere are several technical issues with references throughout the manuscript, particularly in the Appendix. Compilation errors resulting in ?? citations appear in multiple locations, including: Page 3 line 129 (right column), Page 21, Page 24 lines 1316~1318, Page 38\n...**\n\nThank you. We will address all of the minor comments raised in the camera-ready version of the paper.\n\n**While the authors highlight the novelty of implementing the ‘API-first’ architecture via MVC design as one of the main contributions, there are limited demonstration or explanation to support the statement throughout the paper. How does this architecture differ from other versions or databases? In which way might the users benefit from such design? A section for supporting such claim, either quantitatively or qualitatively, may clarify the readers in the technical contribution of pyTDC architecture.**\n\nWe contrast this architecture difference with other benchmarks in B.1, noting that alternatives, including the previous version (which we can note as well), provide access to “file dumps.” We can reword and expand this to more thoroughly distinguish between static datasets provided by alternatives and our “API-first dataset.” The MVC allows us to implement data views (https://www.w3schools.com/mysql/mysql_view.asp), our chosen implementation of the “API-first dataset” abstraction. \n\nIn B.1.3 we state “[PyTDC] develops an Application-Embedded Domain-Specific Data Definition Programming Language facilitating the integration of multiple modalities by generating data views from a mapping of multiple datasets and functions for transformations, integration, and multimodal enhancements, while…”.\n\nWe agree that while all components for answering your query are present in the paper, and shared above, a more distinguishable narrative would be helpful and will write an addition to our discussion of the architecture more thoroughly detailing: The API-first architecture and the benefits of invoking external APIs, the “API-first dataset” abstraction, the choice of MVC design pattern, how that implements data views and the overarching “API-first dataset”, and illustrating via existing data views, would be helpful. We cite works demonstrating individual benefits and our contribution is composing these into an innovative design.\n\n**Another area of concern relates to the redistribution and usage of pre-trained models and datasets through the PyTDC platform. Given the diverse licensing terms associated with pre-trained models (some under open-source licenses, others with academic or non-commercial restrictions), how does PyTDC plan to: - Clearly surface licensing terms and permissible use when users retrieve models especially through python editors? -  Ensure that models with restrictive licensing (e.g., non-commercial, academic use only) are not inadvertently used in ways that violate their original licenses? - Including a transparent licensing and provenance management layer would greatly enhance trust, reproducibility, and adoption in both academic and commercial settings, particularly for users in regulated industries like pharmaceutical research.**\n\nWe surface license terms on the website for all datasets and will do so for models on our model hub pages as well. We can explore the possibility of augmenting classes to expose licenses more easily. Beyond exposing licenses, it is likely impractical for PyTDC to prevent license violations in its current state as a PyPI package (and Harvard dataverse repository). However, deploying a transparent licensing and provenance layer is certainly a requirement of interest for deploying a distributed service based on the current PyTDC platform. This has been added to our product backlog for future releases.", "claims_and_evidence": "The paper provides reasonably convincing evidence for the overall utility of PyTDC as a multimodal biomedical benchmarking platform. This is supported by a comprehensive case study on the single-cell drug-target nomination task, where several state-of-the-art models (including graph representation learning methods and domain-specific approaches) are benchmarked against the newly introduced contextualized tasks. The case study effectively highlights gaps in current model performance, particularly in out-of-distribution settings, supporting the need for improved multimodal, context-aware methods. The webserver and model retrieval infrastructure appear to be well-implemented and accessible.\n\nHowever, the necessity for adopting the specific 'API-first' architecture and Model-View-Controller (MVC) design pattern are underexplained. While these approaches are well-established in software engineering, the paper does not clearly justify why they are uniquely advantageous for this use case compared to alternative architectures, including comparison with the previous versions. Further explanation is needed to clarify how this design improves the implementation of the pyTDC server, in perspectives including modularity, scalability, or reproducibility.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria seem appropriate for the problem domain. The datasets are well-curated across diverse modalities, and the evaluation metrics appear aligned with both machine learning performance assessment and domain-specific biological relevance, such as out-of-distribution generalizability across cell types.", "theoretical_claims": "As I have understood, this manuscript does not present or rely on any formal theoretical claims or mathematical proofs, which is appropriate for a system and benchmarking paper of this nature. The focus is on the development of infrastructure, integration of multimodal data, and empirical benchmarking of biological machine learning tasks.", "experimental_designs_or_analyses": "The experimental design and case study focused on the single-cell drug-target nomination task were reviewed in detail, including inspection of the provided code examples for loading datasets, training workflows, and model benchmarking results. The experiments are generally sound and well-documented, with clear pipelines for data processing, training, and evaluation.", "relation_to_broader_scientific_literature": "The contributions of this paper align with and extend several active research domains at the intersection of multimodal biomedical machine learning, biological data platforms, and therapeutic AI modeling. Specifically, PyTDC expands existing biomedical data platforms by adding multimodal contextualization, foundation model evaluation, and task-specific benchmarking infrastructure, filling a critical gap for researchers developing the next generation of multimodal foundation models for therapeutic discovery.", "essential_references_not_discussed": "None to be addressed.", "other_strengths_and_weaknesses": "**Strengths**\n\nThe overall contribution of this paper is significant, particularly in the context of accelerating multimodal AI research for therapeutic discovery and precision medicine. By combining diverse biological data modalities with a unified benchmarking and inference platform, PyTDC has the potential to serve as a foundation for interdisciplinary studies across computational biology, cheminformatics, and clinical data science.\n\nOne of the key strengths is its integration of multimodal data with model retrieval, fine-tuning, and evaluation capabilities, which is often fragmented across different tools and platforms. The open-source nature and availability of a web interface may further lower the barrier to entry for both computational researchers and domain experts.\n\n**Weaknesses**\n\nSee `Claims And Evidence` section paragraph 2.", "other_comments_or_suggestions": "Minor comments: \n\n1.\tThere are several technical issues with references throughout the manuscript, particularly in the Appendix. Compilation errors resulting in `??` citations appear in multiple locations, including: Page 3 line 129 (right column), Page 21, Page 24 lines 1316~1318, Page 38\n\n2.\tIn Table 1, the check marks and cross marks could benefit from improved visual contrast, such as color-coding or distinct icons, to improve readability. Currently, the marks are difficult to distinguish at a glance.\n\n3.\tThere is confusion between the terms \"pyTDC\" and \"TDC-2\" across the manuscript and appendix. This includes Figure 1 and multiple locations throughout the appendix. The paper needs to be clearer in whether TDC-2 refers to a previous version, a distinct system, or a legacy project in each referenced points for its appearances throughout the manuscript, and ensure terminology is consistent across all sections of the paper.", "questions_for_authors": "1.\tWhile the authors highlight the novelty of implementing the ‘API-first’ architecture via MVC design as one of the main contributions, there are limited demonstration or explanation to support the statement throughout the paper. How does this architecture differ from other versions or databases? In which way might the users benefit from such design? A section for supporting such claim, either quantitatively or qualitatively, may clarify the readers in the technical contribution of pyTDC architecture.\n\n2.\tAnother area of concern relates to the redistribution and usage of pre-trained models and datasets through the PyTDC platform. Given the diverse licensing terms associated with pre-trained models (some under open-source licenses, others with academic or non-commercial restrictions), how does PyTDC plan to:\n    - Clearly surface licensing terms and permissible use when users retrieve models especially through python editors?\n    - Ensure that models with restrictive licensing (e.g., non-commercial, academic use only) are not inadvertently used in ways that violate their original licenses?\n\n    Including a transparent licensing and provenance management layer would greatly enhance trust, reproducibility, and adoption in both academic and commercial settings, particularly for users in regulated industries like pharmaceutical research.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LimkO7lFo8", "reviewer": "Reviewer_ckR1", "summary": "This paper describes PyTDC, an open-source software platform for training, evaluation, and use of biological foundation model. This provides API access to multimodal biological data-sets, a variety of tasks and associated metrics, and APIs for model retrieval and deployment. The utility of this system is demonstrated through the comparison of different models on a context-aware drug target prediction task.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reviewer ckR1,\nThank you very much for the time dedicated to reading the paper and providing your review. Addressing your points below:\n--\n\"The paper describes a useful system. The key weakness is a lack of scientific novelty, which makes it more suited to a systems-focused conference or workshop.\" and \"The paper describes a system which looks to be of general use, however I question whether this is the right venue. It would be more suited to a systems-focused workshop, e.g., MLSys.\"\n\n-> The  ICML 2025 Call for papers lists the following as of interest for the main conference: Machine Learning Systems (improved implementation and scalability, hardware, libraries, distributed methods, etc.). Our paper is inline with this interest.\n\n-> The ICML 2025 Call for papers lists the following as of interest for the main conference: Application-Driven Machine Learning (innovative techniques, problems, and datasets that are of interest to the machine learning community and driven by the needs of end-users in applications such as healthcare, physical sciences, biosciences, social sciences, sustainability and climate, etc.). Our paper is inline with this interest.\n\n-> The paper establishes novelty on the fronts of model evaluation criterion, datasets, task definitions, and machine learning system design.\n\n-> As an example of precedent for ICML publication, our paper cites the ICML paper (Kahn et al., 2022) . It is also an ML systems paper and is a good example we'll cite for some of your other expressed concerns : https://proceedings.mlr.press/v162/kahn22a/kahn22a.pdf\n\n--\n\n\"This paper is largely descriptive, rather than making specific scientific claims and providing evidence for them. The stated characteristics and utility of the system is evidenced through high-level diagrams and an example application.\"\n\n-> This is inline with MLSys publications at ICML such as (Kahn et al., 2022) https://proceedings.mlr.press/v162/kahn22a/kahn22a.pdf \n\n-> Papers presenting new software (models, algorithms, systems, etc) are necessarily descriptive, including those published at ICML. To illustrate this, an ICML paper chosen at random from last year is also largely descriptive with results being showed starting on page 6. https://openreview.net/pdf?id=HsseRq2FAx \n\n--\n\n\"This work does not present novel methods or evaluation criteria, rather it describes a system for evaluating existing methods under existing evaluation criteria.\"\n\n-> The comparisons in Table 1 demonstrate the significant novel contribution of PyTDC.\n-> The novel dataset collection, benchmark organization, metric design, and API architecture are well-executed and demonstrate the platform's utility.\n-> The proposed methods and evaluation criteria seem appropriate for the problem domain. The datasets are well-curated across diverse modalities, and the evaluation metrics appear aligned with both machine learning performance assessment and domain-specific biological relevance, such as out-of-distribution generalizability across cell types.\n\n--\n\nOverall: This review is correct in stating we don't emphasize novel theoretical claims. However, it appears to over-generalize this and argue the work does not present novelty aligned with ICML interests. In this rebuttal, we re-emphasize novelty and how it aligns with ICML Call for papers.\n\n--\n\nIn light of this, we kindly request the reviewer re-evaluate their claims and recommendation, and share any further reservations. A revision of the claims made in the review, given inconsistency with ICML criteria and precedent as well as with the content of the paper, and, consequently, the overall recommendation, should be considered.", "claims_and_evidence": "This paper is largely descriptive, rather than making specific scientific claims and providing evidence for them. The stated characteristics and utility of the system is evidenced through high-level diagrams and an example application.", "methods_and_evaluation_criteria": "This work does not present novel methods or evaluation criteria, rather it describes a system for evaluating existing methods under existing evaluation criteria.", "theoretical_claims": "There are no theoretical claims in this work", "experimental_designs_or_analyses": "The evaluation experiment describe is valid and explained clearly.", "relation_to_broader_scientific_literature": "The paper describes a system which looks to be of general use, however I question whether this is the right venue. It would be more suited to a systems-focused workshop, e.g., MLSys.", "essential_references_not_discussed": "The relevant related works are discussed, as far as I can tell", "other_strengths_and_weaknesses": "The paper describes a useful system, which provides a basis for further development and optimisation of multimodal biological foundation models. This should be of general interest to the ICML community and provide a foundation for the development of biological foundation models.", "other_comments_or_suggestions": "No other comments", "questions_for_authors": "No questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces PyTDC, an open-source platform for training, evaluating, and deploying multimodal biomedical machine learning models. It integrates diverse biological data sources, supports a wide range of therapeutic tasks, and provides infrastructure for benchmarking and inference. Reviewers agree that the work addresses a timely and important need in the biomedical AI community. While the contributions are primarily system-oriented, the paper introduces several useful components, including task definitions, evaluation metrics, and a case study that illustrates practical challenges. Some concerns are noted regarding the scope of the claims and architectural clarity, while the overall value of the platform is recognized, especially in facilitating future research in multimodal and context-aware biomedical modeling.
Decision: Accept (poster)