SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Design

Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 01:47:47

Abstract: Predicting synergistic drug combinations can help accelerate discovery of cancer treatments, particularly therapies personalized to a patient's specific tumor via biopsied cells. In this paper, we propose a novel setting and models for *in-context drug synergy learning*. We are given a small "personalized dataset" of 10-20 drug synergy relationships in the context of specific cancer cell targets. Our goal is to predict additional drug synergy relationships in that context. Inspired by recent work that pre-trains a GPT language model (LM) to "in-context learn" common function classes, we devise novel pre-training schemes that enable a GPT model to in-context learn "drug synergy functions". Our model---which does not use any textual corpora, molecular fingerprints, protein interaction or any other domain-specific knowledge--- is able to achieve competitive results. We further integrate our in-context approach with a genetic algorithm to optimize model prompts and select synergy candidates to test after conducting a patient biopsy. Finally, we explore a novel task of inverse drug design which can potentially enable the design of drugs that synergize specifically to target a given patient's "personalized dataset'". Our findings could have an important impact on precision cancer medicine, and also raise intriguing questions on non-textual pre-training for LMs.

Main paper: 
 introduction: 
 Drug combination therapy is a standard practice for diseases including cancer (Mokhtari et al., 2017) and HIV. It is based on identifying multiple single agent therapies that, when used together, lead to synergistic effects. Predicting such combinatorial synergies is challenging, especially given the wide range of multiple different mutations as well as different genetic backgrounds typically found in different patients' cancer cells (Mroz & Rocco, 2017) . Many drug combinations can also cause increased toxicity (Zapata et al., 2020; Juurlink et al., 2004 ) in a manner that may depend on specific patient backgrounds (O'Donnell & Dolan, 2009) , adding further complexity to the problem. To enable the safest and most effective implementation of combination therapy in cancer care, it is thus important to personalize the prediction of drug synergies.Since the number of drug combinations scales exponentially, differentiating between synergistic and antagonistic pairings is very expensive to test in large quantities in laboratory conditions. Thus, considerable interest has recently grown in using machine learning for predicting synergistic and antagonistic effects between pairs of drugs in silico (Liu et al., 2020; Preuer et al., 2018; Rozemberczki et al., 2022a) . These approaches are typically not evaluated in the few-shot setting, where only a few training examples are given. This is particularly relevant in the personalized setting described above, and more generally for cancer tissue types for which there is limited training data for synergy learning models. Additionally, these efforts use a variety of features to categorize the drugs, from molecular fingerprints (Preuer et al., 2018) to protein interactions (Yang et al., 2021) . Obtaining these features often requires integrating external knowledge sources (e.g., from drug databases), which often results in findings being restricted to the limited subsets of drugs for which this information is available and also requires specialized engineering in model design. Finally, it is unclear if these external sources are actually needed for current models.In this work, we address these limitations by exploring the ability of transformer language models (LMs) to learn drug synergy relations. We devise approaches that leverage transformers (1) without any external knowledge required to be integrated into the model (i.e., no protein interaction networks or patient cell line features); (2) in the few-shot setting with an in-context learning approach that can generalize to novel unseen drugs and patient cell lines; and (3) for designing novel synergistic drug structures in the context of a specific patient's data.Transformer LMs are Strong Drug Synergy Learners-Even Without Textual Representations First, we consider drug synergy prediction using transformer language models without enriching drugs/cells with information from external knowledge bases. We find these "feature-less" models are able to achieve results that are better or competitive in comparison to knowledge-enhanced state-ofart drug synergy models (e.g., BERT models achieve 84.1% ROC-AUC to GraphSynergy's 83.4%) Furthermore, in contrast to recent work that uses language models pre-trained on scientific corpora (Nadkarni et al., 2021) , we discover an intriguing finding: using randomized (i.e. uninformative) tokens instead of drug/cell names is able to rival models that use textual names of those entities. This suggests that external information coming from pre-training on scientific corpora (e.g., as in SciBERT (Beltagy et al., 2019) ) or the web (e.g., Wikipedia) has negligible impact on fine tuned models in this setting. These findings motivate us to explore the power of transformer models without external information, and to study generalization beyond memorization capacity by evaluating on novel drugs/cells that were unseen during training.SynerGPT: A New In-Context Drug Synergy Setting & Model We take inspiration from recent work (Garg et al., 2022) that showed how a GPT model architecture can be trained to "in-context learn" function classes such as linear functions (e.g., linear regression/classification) and neural networks. We pre-train a GPT model from scratch on known drug synergies-using no textual corpora-and explore its ability to generalize in the few-shot setting to drugs and patient cell lines unseen during training. We find that our model, dubbed SynerGPT, is able to achieve strong competitive results without any external knowledge sources. In particular, we introduce a new setting of In-Context Learning for Drug Synergy (ICL-DS). In-Context Learning (ICL) (Dong et al., 2022) has emerged as a powerful paradigm for few-shot learning (Brown et al., 2020) . In ICL, trained model parameters are never explicitly updated after pre-training, and adaptation to each task is done on the fly given contextual examples. This is particularly appealing in settings where it is prohibitively costly to perform parameter updates for each incoming new task and context (e.g., for each new patient in a hospital setting). We devise novel pre-training approaches for ICL-DS, including strategies for optimizing the language model prompt selection with a genetic algorithm. Prompts comprise specific combinations of drugs tested for synergy on specific patient cell lines; optimizing prompt selection in this setting has potential implications for the design of a standardized assay panel of drugs and cells to be tested for a patient's particular tumor. While specific patient data at this level is not readily available, we re-purpose existing drug combination data to lay the foundations for formalizing and studying our approaches from a machine learning perspective.Designing New Molecules to be Synergistic in the Context of a Specific Patient Finally, in our third major contribution we propose an additional new task of Inverse Synergistic Drug Structure Design (ISDSD): using a GPT transformer model for generating or retrieving drug molecules that are synergistic in the context of a specific cancer patient's information (i.e., molecules that are synergistic with other drugs administered to a patient with specific cancer cells). This approach may in the future provide a new methodology for personalized drug candidate discovery. 
background and problem setting: 
 In the last few decades, combination therapy has emerged as an effective method to target genetically unstable diseases (Mokhtari et al., 2017) , with dramatic success in treating HIV (Moore & Chaisson, 1999) and more recently HCV (Liang & Ghany, 2013) . Unlike HIV and HCV which encode only 10-15 proteins (Frankel & Young, 1998; Dubuisson, 2007) , cancer is radically more complex. Since cancer has an unstable genome, combination therapy is often considered necessary (Mokhtari et al., 2017) and is commonly used in practice, with varying degrees of success.Generally, drugs work by affecting cellular pathways-chain interactions of molecules which lead to changes in a cell. In drug synergy prediction, our goal is to predict whether combining drugs will have positive or negative outcomes in the complex system of these interacting pathways. Generally, synergy lab experiments are conducted in cell lines, which are a population of cells from a multi-cellular organism (for example, human lung cancer cells). In this work, we also investigate inverse design of drug molecules. Traditionally, the idea behind inverse design of molecules is to predict or retrieve a molecular structure which has some desired chemical property or protein target (Sanchez-Lengeling & Aspuru-Guzik, 2018) . In our work, we seek to explore inverse design at a higher level-the "interactome" of drug interactions in complex cellular pathways.General Problem Formulation Given k input drugs d 1 , d 2 , . . . , d k ∈ D along with a cell line c ∈ C, the goal of drug synergy prediction is to predict a synergy value y for the interactions between the drugs in the given cell line. In existing datasets, only the pairwise k = 2 setting is considered. Thus, we focus our experiments on pairwise drug synergy, the most commonly researched setting, but our methods can naturally be extended to n-ary synergies. This problem can be considered as either a regression (y ∈ R) or a binary classification problem (synergistic (True) or not (False); y ∈ [0, 1]). Synergy data comes from a dataset of tuples (d 1 , d 2 , c, y) ∈ D.We also consider the few-shot setting in our formulation, which has applications for predicting synergies when there is scarce training data such as in tumor-specific synergy prediction, uncommon cancer tissues, or newly introduced single-agent therapies. In the few-shot setting, we assume there are n synergy tuples available which contain an unknown entity h (unknown cell line c h or unknown drug d h ). Define these tuples asx i := (d 1 , d 2 , c, y) i for i ∈ [1..n]where one of d 1 , d 2 , or c is the unknown h. Each x i can then be used for training in addition to the existing training data. In our proposed method SynerGPT, we don't use these tuples x i in trainingrather, we use them as the prompt for in-context learning. Here, we are particularly interested in synergy prediction based on extremely small datasets (e.g. tested synergies from a patient's specific cancer cells), which makes traditional supervised approaches less effective. In section 3.2.3, we detail our training strategies for in-context learning with unknown h from limited examples.Example 1Example n Unknown Combination We propose a new task where the goal is to predict the structure of a molecule given a context of drug synergy tuples (e.g., we might be given 20 synergy tuples). We train a model to predict the structure of some unknown drug d h from its synergy relations with other drugs. This has two important uses. First, this may enable scientists to predict new molecules which have desirable synergies or similar synergies to existing drugs, which is a novel way to consider drug discovery. This can potentially enable the design of drugs that synergize specifically to target a given patient's unique cancer cells. Secondly, this can support explainability of the synergy prediction model as a function of the context it is fed, by "visualizing" SynerGPT's understanding of the unknown drug given the context. Section 4.3 shows that we can observe the structure of the drug evolving towards the ground truth as more context examples are given. As this is a novel and difficult problem; we initially frame it as a retrieval task, effectively constraining the output space, though from an implementation perspective it is trivial to instead predict structures by using a pretrained generative model for molecules (Jin et al., 2020) with no architectural differences, as both the retrieval and generation of drug structures requires generating a latent vector.Uniformly select examples from the nodes adjacent to x in G.3. Unknown-First: Uniformly select nodes adjacent to x which share an edge of type e h , i.e. prioritizing selection of nodes that contain the masked unknown h.Note that these strategies are hierarchical-Unknown-First falls back to Graph when there aren't enough examples which falls back to Random. Examples from Random are put earlier in the context than Graph which is again put before Unknown-First. In order to train the model to correctly use the [UNKNOWN] token, we need to artificially create unknown drugs or cells during training. Given training example x, we uniformly select d 1 ∈ x or d 2 ∈ x to be the hidden drug d h . For the unknown cell line setting, c ∈ x is always set to c h because there is just one cell line per example. We replace all occurrences of h in the prompt with [UNKNOWN]. We note that our sampling strategy is related to retrieval augmented models (Mialon et al., 2023) . Here, however, we note that the model is also in-context learning synergy functions for unknown drugs based on Definition 1 in (Garg et al., 2022) . 
methodology: 
 In this section, we will consider the four components of our paper. First, we detail how drug synergy tuples are input to encoder-only language models ( § 3.1). Next, we extend this idea to the few-shot ICL setting and propose training methodologies to do so ( § 3.2). We then discuss optimization of the "prompt" used for ICL ( § 3.2.3). Finally, we extend our methodology to inverse drug design ( § 3.3).Initially, we explore the efficacy of BERT-style language models (Devlin et al., 2019; Beltagy et al., 2019; Yasunaga et al., 2022) for drug synergy prediction. We modify the task input to be in natural language using a simple formulation:[CLS] d 1 [SEP] d 2 [SEP] c [SEP]where d 1 and d 2 are drug names (e.g., imatinib, 5-FU), and c is the name of a cell line (e.g., MCF2, Ishikawa). The model is then trained to predict the output value y from the [CLS] token representation.We also investigate to what extent pretraining knowledge is responsible for the model's performance. To do so, we evaluate the impact on performance when the drug and cell names are replaced with 'random' tokens. Given the ordered (by frequency) vocabulary V of the LM, we select the tokens{v i ∈ V | i ∈ [k..(k + |C| + |D|)]} to represent our drug and cell lines. Note we start at a threshold k to avoid the most common tokens which might have specialized representations in the language model's latent space. We uniquely map each cell line and drug to a token in this set, which we use as input to the BERT LM. Essentially, this experiment is used to determine whether knowledge from pretraining or the transformer architecture itself is responsible for performance on the drug synergy task. An example input from this strategy is: [CLS] Recent work trained transformer models to "in-context learn" function classes (Garg et al., 2022) . A function class is a set of functions that satisfy specific properties, such as linear functions or neural networks. In-context learning of a function class F is defined as being able to approximate f (x query ) for "most" functions f ∈ F given a new query x query when conditioned on a prompt sequence (x 1 , f (x 1 ), . . . , x n , f (x n ), x query ). We define a prompt prefix P n (x 1 , f (x 1 ), . . . , x n , f (x n ), x n+1 ) as the first n in-context examples followed by the n + 1th input. A model M θ parameterized by θ is trained to minimize the loss averaged over all prefixesmin θ E        1 n + 1 n i=0 w i ℓ M θ (P i ), f (x i+1 )        (1)given some appropriate loss function ℓ. Weights w n := 1 unless otherwise noted.For in-context prediction of drug synergy, we redefineP n = (d 1 1 , d 2 1 , c 1 , y 1 , . . . , d 1 n , d 2 n , c n , y n , d 1 n+1 , d 2 n+1 , c n+1) as the prompt prefix (as discussed in Section 2, we refer to this as the "context" or "input context"). Here, y can be considered the output of a function measuring synergy on (d 1 , d 2 , c). As in (Garg et al., 2022) , we consider a GPT-2 family (Radford et al., 2019 ) decoder-only model, which we call SynerGPT. Here, the prediction of the synergy value y j is made using a linear transformation of the contextualized output representation of c j (note that this includes d 1 j and d 2 j due to self-attention). Model inputs-drugs d, cell lines c, and labels y-are initialized using a learnable embedding layer (i.e. no external features). To evaluate the model's ability to predict synergies of unknown entities, we hold out either m drugs or m cell lines and remove their synergy relations from the training set (see Section 4). We use a subset of the held out tuples as a pool of context examples. We now turn to selecting the context (prompt prefix) from this pool in a manner that increases predictive performance.A central question about using language models without external features-including textual namesis how to teach the model to understand unknown drugs or cell lines. We propose using a masking strategy-every unknown drug d h or cell c h is represented by [UNKNOWN] and the model must use in-context learning to understand it based on contextually-related known drugs and cell lines. In this setting, we assume that we are given a set of synergy tuples to sample from to construct a prompt. During training, it's simply the training set. During evaluation, we consider a special held-out "context" set D c ⊂ D (thus named because we sample the context/prompt P n from this set). To sample from this context set, we propose a context-selection strategy based on constructing a graph G on this D c . Specifically, we construct G by creating a node for every synergy tuple x := (d 1 , d 2 , c, y) ∈ D c . We construct a drug edge e d between two nodes x 1 and x 2 if they share drugd (i.e. d ∈ x 1 ∧ d ∈ x 2 ).Similarly, we construct a cell line edge e c if they share cell line c. See Figure 1 for an example and Appendix Figure 8 for more details. We employ the following context selection strategies to sample a context with n examples given some node x containing unknown h which is either drug d h or cell c h :1. Random: Uniformly select n context examples from D c .We further study whether the context can be optimized to best enable predictions for some unknown drug or cell line h (see Figure 7 for an example). The purpose of these experiments is to enable the eventual development of a standardized assay for drug synergy prediction. Thus, as output, these optimization algorithms produce a set of context tuples for each h. To do this optimization, we assume that we have four splits of data, which are constructed as follows. Given a set of p "unknown" drugs/cells H, all synergy tuples not containing any h ∈ H are put into a training set D T r . The remaining tuples are randomly partitioned into three equal sized sets: a context bank D c , a validation set D v , and a test set D T e . We first train a model on D T r following the Unknown-First strategy (where contexts are sampled from D T r itself). Following this, for each unknown entity h i , we select n context examples from D c which maximize the model's score on the validation set D v . This is a combinatorial optimization problem which can be considered related to the best subset selection problem (Bertsimas et al., 2015; Miller, 2002) . We consider a genetic algorithm (Gad, 2021): a metaheuristic method which is useful for black box optimization of systems containing complex interacting parts (Mitchell et al., 2007) , which is suitable for the complex interactions between cellular pathways required for drug synergy prediction. As output, we get a set of context tuples for each h. Optimization algorithm details are given in Appendix C.To train the model to retrieve relevant drug structures in-context, we use the same architecture as for synergy prediction ( § 3.2.2), so that we can use the same data split and optimized contexts from Section 3.2.4 to understand how the model interprets them. For effective retrieval, we need a strong base molecular representation that makes it possible to effectively distinguish molecules. So, we choose to use MegaMolBARTv2 (NVIDIA Corporation, 2022) representations, which were trained on 1.45 billion molecular SMILES strings and thus have a relatively comprehensive (in terms of drug classes) latent space. We train a SynerGPT model from scratch to predict representations using a linear transformation on the output [UNKNOWN] representation. We use this final representation to retrieve the desired drug using cosine similarity with the MegaMolBARTv2 representations of the drugs in our synergy dataset. The training context is selected using the Unknown-First strategy. Finally, we train the model using a minibatch contrastive loss (Radford et al., 2021; Edwards et al., 2021) between the L2-normalized ground truth representations D g (here MegaMolBartv2) and predicted representations D p (output from our model's prediction head):ℓ(D g , D p ) = CE(e τ D g D pT , I b ) + CE(e τ D p D gT , I b ) (2)where CE is categorical cross-entropy loss, b is the mini-batch size, I b is the identity matrix, and τ is a learnable temperature parameter. We use this loss for ℓ in equation 1. 
results: 
 BERT can do Drug Synergy? In this section, we experiment with finetuning BERT on drug synergy data where all drugs and cell lines are seen during training (data splits detailed in Appendix B.1). As discussed earlier, there has been recent work using external network datasets capturing interactions between drugs, proteins and cell lines (Yang et al., 2021) for synergy prediction. To evaluate the impact of these external datasets, we compare against a strong and recent model, Graphsynergy (Yang et al., 2021) that uses over a dozen different network datasets and achieves state-of-the-art on its subset of DrugCombDB. A natural question here is whether the model has learnt the required knowledge during pre-training. Surprisingly, replacing drug and cell names with random tokens ( § 3.1) resulted in no drop in performance. This suggests that the transformer architecture may be the dominant factor explaining BERT's performance on the task. However, if we use a randomly-initialized BERT model without any pre-training, we find the performance is worse (by 3 ROC-AUC pts).To verify our findings, we consider the ChemicalX framework (Rozemberczki et al., 2022b) , which implements several baselines and provides a standardized subset of DrugCombDB (Liu et al., 2020) with drug and cell line features. This standardization allows us to compare different baseline methodologies on the same dataset.The ChemicalX DrugCombDB dataset has 2,956 drugs, 112 cell lines, and 191,391 synergy tuples. We compare against baselines DeepSynergy (Preuer et al., 2018) , MR-GNN (Xu et al., 2019) , SSI-DDI (Nyamabo et al., 2021) , and DeepDDS (Wang et al., 2022) , which we train using default hyperparameters from the original papers for 50 epochs as in (Rozemberczki et al., 2022b) . These baselines (details in Appendix B.2) represent the most popular approaches to drug synergy prediction and allow us to compare against transformer architecture performance. Remarkably, SciBERT with random tokens outperforms all baselines except DeepDDS in this setting (Table 1 ). We see similar results on the DrugComb dataset (this database is larger but is continuously modified by volunteers; see Appendix L). We note that, while this performance is surprising in this domain, it follows from results from other domains. For example, language models are able to learn complex grammar and interactions just by observing how words co-occur.We conjecture this may be related to the observation that pre-training on a nonsense corpus (Krishna et al., 2021) can provide good weight initializiations for downstream tasks. We further discuss related work in Appendix F.We now evaluate models on the few-shot and zero-shot setting, i.e, when a new drug or cell line is introduced with limited or no interaction data. We use the same architecture used in H := {h i | i ∈ [1..m]}). To construct our split, we remove all synergy tuples containing h ∈ H from the dataset D so that the remaining dataset only contains tuples with known drugs/cells (this is our training set D T r ). Then, for each h, we select n synergy tuples randomly to form the "context" bank/split D c . All other "unknown" synergy tuples are put into D T e .For comparison, we use the same baselines trained in zero-shot and few-shot settings. We also test SetFit (Tunstall et al., 2022 ) (a few-shot LM approach), k-nearest neighbors, off-the-shelf pre-trained GPT-2 (using entity names as input, similar to CancerGPT (Li et al., 2023a )), and MAML with DeepDDS (details in Appendix B.2). In the few-shot setting, the context bank D c is considered part of the training set, and in the zero-shot setting it is not used. Our model, SynerGPT, however, is not trained on the context bank but uses it as context (prompt) examples for evaluation. Examples are selected using the Random, Graph, or Unknown-First strategies. We separately investigate the setting where drugs are unknown and where cell lines are unknown.Unknown Drugs To construct the dataset split, we set m = 50 unknown, i.e.,"held-out" drugs and context n = 20 synergy tuples. Hence, our context bank contains 50 × 20 = 1, 000 tuples. Overall, we find that our SynerGPT can perform better in the few-shot setting than existing baselines on on this task, as shown in believe this is caused by the relatively small number of patient cell lines in the data vs. 2,956 drugs, making it harder to learn higher-level types of drug-cell line interaction. In other words, we are trying to learn a complex function class (drug synergy in an unknown cell line) without a significant number of example functions f ∈ F . To alleviate this issue, we use 6 layers, batch size of 128, and only 30 epochs. Nonetheless, the issue still exists-performance decreases for baselines DeepDDS and MR-GNN and our strategies Unknown-First and Graph. We experiment with interpolating between training initially with Random to Unknown-First at the end (see Appendix B.2.2), which helps in the unknown cell line case. We believe this creates an exploration-exploitation effect. As we have shown in the previous section that the context selection strategy is very important for SynerGPT performance, the natural next question is to what extent the context can affect model performance. To test this, we conduct a different split. Like before, we select 50 unknown drugs and 20 cell lines; with their respective tuples, we create three uniform splits: context, validation, and test. We train a Syn-erGPT Unknown-First model using hyperparameters as in our above experiments.In context optimization, our goal is to select examples from the context and train splits which maximize some metric on the validation split. For our experiments, we maximize ROC-AUC for our trained model using the validation set. Overall, we consider two strategies: Unknown-First, and a genetic algorithm (GA). For the genetic algorithm, we use the implementation and hyperparameters from PyGAD (Gad, 2021) with a population of 8 for 50 epochs. Here, we consider each example in the context split to be a potential gene. For comparison, we also select the context at random according to the Unknown-First strategy. To ensure comparability, we evaluate Unknown-First the same number of times as the genetic algorithm and select the best context. Our results (Table 3 ) show that the genetic algorithm optimizes the context from a starting average AUC of 79.2% up to 81.5% for unknown drugs and from 85.2% to 86.1% for unknown cells. Appendix D visualizes this and shows error bars. We further analyze the results by different tissue types (Appendix G). For example, we find that for unknown drugs, synergy prediction in ovarian cancer is effective, but for both unknown drugs and cell lines predictive performance on bone cell lines is low. Explainability is one of the most challenging problems in deep learning. With transformer language models, the contrast between remarkable performance gain and lack of explainability becomes even more striking. Here we propose a novel drug design task to better understand the model's "thought process." As shown in Figure 2 , essentially, we look at SynerGPT's prediction as it gains more information via synergy tuples. While this is a useful step, we do recognize that retrieval doesn't fully address explainability and hope to inspire further work. We refer to Appendix A for more discussion.In this novel task, we evaluate SynerGPT's ability to retrieve the structure of an unknown drug.We use the same splits as before but replace the classification head with a vector output trained using the loss in Equation 2. Using the same splits allows us to visualize the optimized context from the genetic algorithm. Experimentally, we achieve the best performance with the weight value from equation 1 set to w i := i/k. Two examples of the model retrieving drugs which match the context synergy pairs are shown in Figures 2 and 5 . These show the retrieved drug after i context examples have been observed by the model. Additionally, we show overall retrieval performance as the number of context examples shown to the model increases in Appendix E. Figure 4 . For the weighted strategy, mean rank for seen drugs decreases from ∼1,500 to ∼400 as context increases.Qualitatively, we find that we can retrieve the relevant drug or a similar structure from synergy relationships in multiple cases. This is considerably more effective for drugs observed during training, but performance is also better than random for unknown drugs. This ability to visualize the model's understanding is helpful for explaining what the model predicts from observing a given context. Second, it enables retrieving drugs which have a desired set of synergies, which can help inform drug candidate discovery, including patient-specific scenarios. We note that we worked off a broad definition of drug design as discovering new candidate medications. While retrieval is currently a challenging version of this, future work can expand the search space with generative models. 
conclusions and future work: 
 As demonstrated by HIV, HCV, and now cancer, combination therapy is a critical option for disease treatment. Yet, difficulties arise in regards to understanding drug-drug interactions and patient-specific genetic differences. To tackle this, we show that encoder-only language models are effective for drug synergy prediction. We then build on these results by proposing SynerGPT, a decoder model with a novel training strategy for in-context learning which can produce strong results for few-shot drug synergy prediction. We additionally show that the model context can be optimized using non-linear black-box approaches, which has exciting implications for the design of a standardized drug synergy testing panel for creating patient-specific synergy datasets. Finally, we explore a novel task of inverse design using desired drug synergy tuples. Performance on this challenging task is low for unknown drugs; nonetheless, it shows promise for future work that may enable personalized drug discovery. 
 
 Appendix: While we are able to achieve strong performance without additional cellular or drug data, our approach is very much a black box akin to most deep learning methods. To address this, we propose the task of inverse design from drug synergy examples, which allows the visualization of the model's structural understanding as it gains more information. While this is a useful step, we do recognize that further research on mechanistic explainability would be valuable. We hope our contribution on synergy-based inverse design can inspire further work on explainability and that SynerGPT's predictions can be useful inspiration for clinical researchers. We would also like to note that regardless of using deep learning models, pharmaceutical researchers are in many cases unable to explain the mechanisms of many important drugs on their own (e.g., Modafinil, Metformin, general anesthetics) (Stahl, 2020; Rena et al., 2013; Brown et al., 2011) -let alone explain their interactions with each other. These drugs are prescribed to hundreds of millions of patients. Recent studies (Lin et al., 2019) suggest that many purported protein drug targets may not be the actual target at all. Important progress with life-saving modern drugs can be made with limited visibility into underlying mechanisms, yet certainly improved mechanistic understanding would be highly useful.While we show that strong performance is possible without features, future work will still likely want to integrate external database features into drug synergy prediction; however, they will likely need to be integrated in a more thoughtful manner in order to ensure an actual benefit.It would also likely be interesting for future work to investigate the internal connections language models are learning and what it might mean for understanding the fundamental biology of how cellular pathways interact. It is also worth noting that designing molecules using drug synergy tuples is a somewhat atypical task, so there may exist a wall in terms of the information content inherent in the context. While we do analysis by separating model performance into different tissue types in this work (as done in multiple prior studies), we note that for future research it is likely too limiting and simplistic to separate cell lines into tissues types.Full results for the BERT input method and GraphSynergy tests are in Table 4 . We compare on the specific subset of DrugCombDB Liu et al. (2020) which was selected to match Graphsynergy's network data (i.e. selecting the subset of DrugCombDB with drugs/cells that can be matched with external protein-protein interaction, drug-protein association, and cell-protein association networks) and a 7:1:2 train:validation:test split. This data subset also contains useful surface names (the common natural language name of the drug; e.g. dasatinib), which allows us to compare the effect that drug names have on language model synergy prediction performance.We consider three BERT training variations: the original BERT Devlin et al. (2019) , SciBERT Beltagy et al. (2019), and BioLinkBERT Yasunaga et al. (2022) . SciBERT was trained on a corpus of scientific documents which would be considerably more focused on drugs than a general corpus.BioLinkBERT is a biomedical BERT model additionally trained using document relation prediction (e.g. citation links). We would like to reiterate the rather remarkable finding that pre-training on scientific literature does not necessarily help the model perform drug synergy prediction any better.Overall, these results indicate that models using external data may not be behaving how we think they are.We report full results on the subset of DrugCombDB Liu et al. (2020) DeepSynergy is a popular feedforward model which uses cell line features and drug fingerprints. MR-GNN is a graph convolutional network (GCN) Kipf & Welling (2016) fed into an LSTM Hochreiter & Schmidhuber (1997) which takes the drug structure into account. SSI-DDI uses a graph attention network (GAT) Veličković et al. (2017) with a final co-attention layer. DeepDDS uses both a GAT and GCN, which are fed into a fully connected feed forward network.Real GPT-2 We train a GPT-2 model2 in the few-shot setting (as opposed to SynerGPT's zero-shot) using random context and the same hyperparameters to mimic SynerGPT's training settings as much as possible. We use names of the drugs obtained from linking to PubChem Kim et al. (2023) as input in the form "Are drugs [DRUG1] and [DRUG2] synergistic in cell line [CELL]?".SetFit Furthermore, we test finetuning a few-shot language-model baseline, SetFit Tunstall et al. (2022) , on our few-shot data. We follow the original paper in using batch size 16, R = 20 text pairs generated for contrastive learning, and 1 epoch. Inputs to the model follow the same format as BERT in Section 3.1. We test using four models.1. SetFit-SBERT: paraphrase-multilingual-mpnet-base-v2 from Reimers & Gurevych (2019) with names as input. This model was trained to create semantic embeddings via Siamese networks.2. SetFit-C: recobo/chemical-bert-uncased-simcse from Recobo.ai3 with names as input. This model was trained using SimCSE on chemistry text.3. SetFit-S2: allenai/specter2 from Singh et al. (2022) with names as input. This model was trained on multiple scientific classification and regression tasks, such as MeSH descriptors classification.4. SetFit-SMILES: DeepChem/ChemBERTa-77M-MTR from Ahmad et al. (2022) with SMILES strings as input. This model was pretrained by predicting 200 molecular properties for a molecule given its SMILES string.Model-Agnostic Meta-Learning We also consider a meta-learning formulation of our problem setting. We use MAML Finn et al. (2017) to train a DeepDDS model. Since MAML4 does few-shot classification using episodes sampled from different learning tasks, we reframe our problem to match this. We consider predicting synergy for each drug to be a task. Then, we sample an episode for training from a random task for each mini-batch. We aggregate rare drugs without enough samples to form an episode into the same task until there are enough samples for an episode. Additionally, since we are dealing with binary classification here, we use N = 2-way. We sample the "validation" portion of each episode from our training set like in SynerGPT. We use the same context bank (and context size) for "adaptation" during evaluation. The same learning rate (1e -3), batch size (512), and number of steps/epochs as DeepDDS is used. We report few-shot (first-order) and zero-shot (no adaptation) versions. Overall, we find that the MAML training procedure produces poor results, and adaptation produces insignificant performance increases. We attribute this to the episode-based sampling strategy neglecting important information in training.Protonets As another meta-learning baseline, we consider Protonets Snell et al. (2017) . We use the same meta-learning framework as for MAML. Because we don't have drug task meta-data, we only consider the few-shot setting.k-Nearest Neighbors We also consider a k-Nearest Neighbors baseline using scikit-learn Pedregosa et al. (2011) similar to Nadkarni et al. (2021) . We construct embeddings for each synergy pair by concatenating (Drug1, Drug2, Cell) embeddings. In the training set, we also include (Drug2, Drug1, Cell). We consider two embedding sources. For the first, kNN-Features, we consider the drug and cell fingerprint features from ChemicalX. For the second, kNN-S2, we use name embeddings from the Specter2 model. We report both zero-shot and few-shot versions. In the few-shot setting, the context bank is added to the training data. We set k equal to the context number (20 and 10 for drugs and cell lines, respectively). We find performance on cell lines to be surprisingly effective, although still less than SynerGPT.In the Unknown cell line setting, we observe that Random has an interesting effect where it performs better after examples (although still worse than Unknown-First (no-ex)), so we consider a fourth strategy: interpolating between Random Unknown-First. Essentially, for each data mini-batch in epoch e of E total epochs, we select either the Random strategy with probability max(0.25, 1 -e E ) otherwise we use the Unknown-First Strategy. This is analogous to an exploration-exploitation approach where we are pretraining with Random and transitioning to Unknown-First. We use a threshold of 25% to ensure the benefits of Random are kept until the end of training. We find that this interpolation strategy is effective (with p < 0.05, see Table 6 ) in dealing with the unknown cell line case.In the unknown drug setting, to allow for tuples with multiple unknown drugs, we use both a [UNKNOWN] and [UNKNOWN2] token (e.g. a tuple containing two unknown drugs would be ([UNKNOWN], [UNKNOWN2], c)).Unknown BERT models use random token inputs. Results are the average of 5 runs. The difference between Unknown-First with and without context has p < 0.05 for unknown drugs based on a paired t-test. On unknown cell lines using the interpolate strategy, p < 0.05. Similarly, both are statistically significant from the best baseline.For the inverse design experiments, in some cases, context examples do not contain the unknown entity h and therefore no [UNKNOWN] tokens, so we use -→ 0 as a replacement for the ground truth representation when calculating our loss function. We use the same model, splits, and training hyperparameters as in the context optimization setting.In the case of the genetic algorithm, each context bank synergy tuple x ∈ D c is considered as a gene which can be selected by the algorithm. Given p "unknown" drugs or cell lines, each has n slots for context examples in its prompt, which makes for np total genes. We also enforce that each x contains the relevant unknown drug d h or cell line c h . We disallow each context example from being selected multiple times; the reasons for this is two-fold. First, in early experiments we found that if we use the same example for the entire context (e.g. 20 repeats of x), then the model performs poorly. This is likely because the model is not trained on duplicate input, so it is trying to make meaningless connections between the same x. Second, repeating x in the context provides no new information to the model. Although we enforce this constraint, in practice without it the model will likely do the same thing on its own. For the genetic algorithm in context optimization, we use a population of 8 for 50 epochs. We use steady-state parent selection with 4 parents, single-point cross-over, 10% gene mutation, and elitism. Each example in the context bank is considered a gene and we disallow repeated genes. This results in 351 evaluations on the validation set.Using the Unknown-First strategy, we sample a context for some heldout tuple in the validation set. We then calculate the absolute error ϵ for the heldout tuple. For each context example x c in the heldout tuple's input context P n , we store ϵ and the relevant heldout entity, h. After some number (for fairness we use the same number of times as the genetic algorithm evaluates ROC-AUC on the validation set-351) of epochs on the validation set, we calculate a mean error εh (x c ) for that context example x c . Finally, for each heldout drug or cell h, we select the n context examples x c i with the lowest εh (x c i ). As shown in Table 7 , this strategy produces poor performance. This indicates that simply selecting all of the most individually informative context examples is not useful. Rather, there is a more complex, non-linear interaction between examples which is informative to the model. This is intuitive, because the interaction between cellular pathways in complex and still not well understood. The ability for the context to be optimized by a genetic algorithm but not error reduction indicates that data collection strategies which emphasize diversity may be important to consider for constructing new drug synergy datasets.Unknown The ground truth drug is not retrieved, but the model instead identifies two almost structurally identical drugs which would potentially be candidates for interacting with the same drug target. This shows how our method can identify related drugs which behave in similar, synergistic ways.In recent years, advances in machine learning and NLP have been applied to molecule representations. Several efforts (Fabian et al., 2020; Chithrananda et al., 2020; Vaucher et al., 2021; Schwaller et al., 2021; NVIDIA Corporation, 2022; Tysinger et al., 2023) show excellent results training on string representations of molecules (Weininger, 1988; Weininger et al., 1989; Krenn et al., 2020; Cheng et al., 2023) . Interest has also grown in multi-modal models (Edwards et al., 2022; Zeng et al., 2022) and multi-encoder models (Edwards et al., 2021; Vall et al., 2021; Xu & Wang, 2022; Su et al., 2022; Liu et al., 2022; Seidl et al., 2023; Xu et al., 2023b; Zhao et al., 2023) with applications to chemistry and biology. Existing work (Edwards et al., 2022; Su et al., 2022; Xu et al., 2023a; Christofidellis et al., 2023 ) also builds on this to "translate" between these modalities, such as MolT5 (Edwards et al., 2022) , which translates between molecules and language.With the success of models such as GPT-3 (Brown et al., 2020) and GPT-4 (OpenAI, 2023) , interest has grown in the theoretical properties of in-context learning. (Garg et al., 2022) , which we follow in this work, investigates the ability of transformers to learn function classes. (Olsson et al., 2022) investigates whether in-context learning is related to specific "induction heads". (von Oswald et al., 2022) shows that transformers do in-context learning by gradient descent. (Li et al., 2023b) frames in-context learning as algorithm learning to investigate generalization on unseen tasks.Very recently, considerable interest has grown in using language models, particularly GPT-4 (OpenAI, 2023) , for uncovering chemical knowledge and molecular discovery (Hocky & White, 2022; White et al., 2022; Bran et al., 2023; Boiko et al., 2023; White et al., 2023; Castro Nascimento & Pimentel, 2023) , including work in the few-shot setting (Ramos et al., 2023; Jablonka et al., 2023) . CancerGPT (Li et al., 2023a) , a related contemporaneous preprint, was recently released which explores a similar few-shot approach to drug-drug synergy prediction. It explores training literature-aware text-based GPT models on drug synergy data. The use of GPT models pretrained on massive textual corpora from the web also makes rigorous evaluation and comparison difficult. We believe our work is complementary, since we largely explore the transformer architecture without language and we consider in-context learning which they do not. We also consider extensions such as inverse design and context optimization. Due to the recency of (Li et al., 2023a) , we leave additional comparisons beyond our real GPT2 baseline to future work. Applying language models to knowledge graphs has been investigated in the general (Yao et al., 2019; Kim et al., 2020; Youn & Tagkopoulos, 2022) and scientific domains (Nadkarni et al., 2021; Safavi et al., 2022) . They can be considered similar to our tests of BERT language models applied to a drug synergy hypergraph ( § 4).As discussed above, there are several approaches (Preuer et al., 2018; Xu et al., 2019; Nyamabo et al., 2021; Wang et al., 2022; Kuru et al., 2021; Sun et al., 2020; Rozemberczki et al., 2022b) which can predict synergy scores given cell line and drug features. There has also been interest in learning representations for these settings (Scherer et al., 2022) . Recently, work (Yang et al., 2021; Rozemberczki et al., 2022a; Lin et al., 2022) has begun to incorporate additional data sources such as drug-protein interactions. This can help improve results, but it often requires creating a subset of the original synergy dataset which can bias results towards the proposed method. (Yang et al., 2023) extracts additional training data from the literature to improve synergy prediction results, which may relate to our results in Appendix H. Research also investigates the application of few-shot (Ma et al., 2021) and zero-shot (Huang et al., 2023) machine learning to drug response prediction-we extend this idea to drug synergy prediction. (Yang et al., 2020) and (Kuenzi et al., 2020) are related but have different focuses compared to our paper; neither compare against any other synergy baselines or do large-scale evaluation. (Yang et al., 2020) focuses on a mechanistic understanding of (drug, tumor) activity-a different task. They use this understanding to rank subsystems and predict a limited number of drug combinations to evaluate. (Kuenzi et al., 2020) does database and experimental testing with small numbers of cell tissues and drugs.We further analyze the results of context optimization by separating the results for unknown drugs into their effects on different tissue types. To obtain tissue types, we use the COSMIC Tate et al. ( 2019) cancer mutation database. Results (Table 8 ) show that performance varies between different tissue types, but that the context optimized by genetic algorithm outperforms default Unknown-First and the model with no context in all cases with exception of pleura. For example, the model excels predicting synergies in ovarian cancers, but results are lower than average in bone and lymphoid cancers. For example, the ROC-AUC of ovarian cancer increases from 77.6 to 81.6% with examples selected using the default Unknown-First strategy. Our context optimization strategy also shows to be important-the ROC-AUC further increases significantly to 87.5 using the examples selected by the genetic algorithm. In the unknown cell line case, we see improvements on all cell lines except skin and bone. Interestingly, performance on bone-derived cell lines is low in both settings.While we do analysis by separating model performance into different tissue types in this work (as done in multiple prior studies), we note that for future research it is likely too limiting and simplistic to separate cell lines into tissues types. Future studies may look at better bucketing approaches, such as primary cancer-driving mutations. An excellent example are KRAS mutations, which occurr in up to 25% of human tumors and in many different tissue types (for KRAS: pancreatic, thyroid, colorectal, and lung carcinomas, among others) Kranenburg (2005) . Further, we note that while our work focuses on few-shot applications to mono-clonal cell lines and tumor biopsies, there is growing evidence that intra-tumor heterogeneity is a driving factor in cancer growth and is also responsible for drug resistance Black & McGranahan (2021) . Future work can investigate the effect that this heterogeneity may have on patient-specific drug synergy prediction. Stankevicius et al. (2016) . Images are for visualization purposes-in practice a learnable embedding is used as described in 3.2.2.All experiments were done on an internal cluster of GPUs. Each experiment was conducted on a single NVIDIA RTX A6000 with 48 GB VRAM. Notably, multiple experiments can fit on the GPU at one time. Our BERT model experiments done within the ChemicalX framework take roughly 2.5Code will be made available upon publication."gpt2" from HuggingFace.www.recobo.aiWe use the implementation from https://github.com/cnguyen10/few_shot_meta_learning 


Decision: The paper proposes a new approach for drug synergy prediction using fewer features/sources of pre-training knowledge compared to prior work. The results are interesting and surprising in the context of prior work.

It also defines an additional task to generate drug molecules that are synergistic in the context of a specific patient’s context and evaluates the language model on it.

This is an interesting application of Transformer models to in-context learning on sequences of drugs for a cell line.

Pros
* The paper presents interesting findings which can have implications for precision cancer medicine
* The additional experiments included after the author response addressed reviewers' comments

Cons
* The organization of the presentation is not optimal

Reviews: review: paper_summary: paper_summary: In-depth analysis of drug synergies is an important basis for cancer drug design and selection. Based on language models, this paper proposes a novel method for predicting drug synergies, namely in-context learning for synergy prediction. Without introducing many drug and cell line features, this work trains SynerGPT based on GPT-2, which can leverage the capabilities of the Transformer framework to achieve more competitive prediction performance than previous baseline models. This paper also uses algorithm to modify and optimize the context. Finally, this paper proposes the task of inverse drug design to discover medication regimens for specific patients. In short, this study cleverly leverages the capabilities of language models to provide a new, efficient, low-cost, and easily implementable solution for drug synergistic research.
strength_weakness: strength_weakness: 1. This paper fails to adequately consider the intrinsic features of drugs and cell lines. It may be overly simplistic to achieve drug and cell line representations solely through a learnable embedding layer. In current research, there are doubts about whether this method can be effective across a broader range of drugs and cell lines.
2. This paper only simply considers synergy between two drugs; no experiments are conducted regarding synergy among more drugs.
questions: questions: 1. In Section 4, the paper concludes that “replacing drug and cell names with random tokens resulted in no drop in performance.” Does this conclusion in terms of Encoder-only models similarly apply to Section 5? Does the method of representing drug names with random tokens remain effective in the training of GPTs?
2. In Section 5, the inputs of SynerGPT, such as drugs and cell lines, are obtained from a learnable embedding layer. Compared to directly fine-tuning GPT-2 using real drug and cell line names as training data, how much difference in evaluation results will there be with the current SynerGPT?
paper_summary: paper_summary: The paper introduces SynerGPT, a novel machine learning framework that leverages a transformer model (specifically GPT) to predict drug synergies without the need for domain-specific data like molecular fingerprints or protein interactions. This approach is notable for its potential to simplify and accelerate the process of identifying effective drug combinations for cancer treatment, a crucial area in personalized medicine. The authors propose an innovative way to pre-train the model on drug synergy functions and use in-context learning for new, unseen drugs and cell lines. The significance of this work lies in its ability to move toward practical applications of AI in enhancing personalized therapy strategies, which could lead to more efficient and tailored treatment options for patients with complex diseases like cancer.
strength_weakness: strength_weakness: - Lack of Detail: The paper could be improved by providing more detailed technical descriptions of the model architecture and training processes. This would help in replicating the study and understanding the nuances of the model's performance.
- Limited Evaluation: The evaluation focuses primarily on specific types of cancer. Expanding the scope to include other diseases where drug synergy is critical could make the case stronger for the model's applicability and robustness.
- Concerns Over Generalizability: The paper does not sufficiently address how the model performs across different sets of drugs and diseases beyond those tested. This is crucial for understanding the model's utility in broader clinical settings.
questions: questions: - Data Preprocessing and Splits: Could you provide more detailed information on how the data was preprocessed and split for training, validation, and testing? Understanding this aspect is crucial for assessing the model's robustness and the potential for overfitting.
- Model Generalizability: How does SynerGPT perform when applied to other diseases or drugs not included in your study? Are there limitations in its applicability to other areas of drug discovery?
- Integration into Drug Discovery Pipelines: How do you envision integrating SynerGPT into real-world drug discovery pipelines? What are the challenges, and how might they be overcome? Additionally, how does the model handle the complexity of multi-drug interactions in more genetically diverse patient populations?
paper_summary: paper_summary: This paper proposes to use transformer-based language models to solve the problem of drug synergy prediction. In particular, the few-shot DDS problem is studied with the technique of in-context learning. In addition, authors also study the task of drug retrieval with the proposed model.
strength_weakness: strength_weakness: 1. The main problem of this paper is the organization of methodology.
First, Section 4 seems unnecessary and redundant. a) There are several works in the literature showing that BERT can do DDI prediction, DDS prediction, and link prediction on graphs. b) The experiments in this part are on encoder-only model while the remaining parts are on decoder-only model. c) The main problem setup in this paper is for few-shot DDS, while the experiments in section 4 are not about this.
Second, it will be much better if sections 5-7 can be organized as a whole framework. Currently, the DDS and drug design parts are too separate, with different algorithms and different training strategies.
Third, the results are given for each section, again making the methodologies look separated.

2. The authors mentioned several times about patient's personalized dataset (also in the title). But how to understand the relationship between the proposed method and the work "personalized"? There is little description and motivation for this point in the method design. In addition, I also fail to find how such a dataset looks like. How many patients are there and how many samples each patient has?

3. It is hard for me to understand the results in Figure 2. In other words, how can I see that the retrieved molecules are meaningful? Why does the proposed method work better than other retrieval methods?

4. There lacks an important baseline/related work studying the few-shot DDS problem: "Few-Shot Drug Synergy Prediction With a Prior-Guided Hypernetwork Architecture" in TPAMI 2023.

Overall, even though this paper introduces something novel and interesting, there is a huge space to improve in this paper.

Comments: title: After rebuttal comments
comment: Thank you for your responses. I have increased the score as my appreciation for the responses.

The remaining concerns, as indicated by other reviewers, are the presentation issues and the limited experiments. Hope the authors will take these concerns into account and carefully revise the presentations if accepted.
Thank you for the response. These all make sense and I assume there will be (very minor) updates to the manuscript. My (positive) recommendation remains unchanged.