LINT: LLM Interaction Network for Clinical Trial Outcome Prediction

Authors: chufan  gao
tianfan  fu
jimeng  sun
Venue: KDD_org_2024_Workshop_AIDSH
Published: 2024-04-24 14:07:52

Abstract: Clinical trial outcome prediction aims to predict the success probability of a clinical trial that reaches its desirable endpoint. Most of
the effort focuses on developing machine learning models for making accurate predictions with diverse data sources, including clinical
trial descriptions, drug molecules, and target disease conditions. Accurate trial outcome prediction helps trial planning and asset portfolio prioritization. Previous works have focused on small-molecule drugs; however, biologics are a quickly growing intervention type
that lacks information that is traditionally known for drugs, like molecular properties. Additionally, traditional methods like graph
neural networks are much more difficult to apply to biologics data which are a fast-growing type of drug. To address these points, we
propose a Language Interaction Network (LINT), a novel method for trial outcome prediction using only free-text descriptions. We validate the effectiveness of LINT with thorough experiments across three trial phases. Specifically, LINT obtains 0.770, 0.740, and 0.748 ROC-AUC scores on phase I, II, and III, respectively, for clinical trials with biologic interventions.

Main paper: 
 related work on trial outcome prediction: 
 Existing works often focus on predicting individual patient outcomes in a trial instead of a general prediction about the overall trial success. They usually leverage expert, hand-crafted features. Previous work has taken advantage of many aspects of the trial-related features [18, 21, 28, 39, 43, 54, 55] .Deep learning techniques have also seen widespread usage in learning representations from clinical trial data to support downstream tasks such as drug repurposing [2, [14] [15] [16] 57] , patient retrieval [12, 19, 41] and enrollment [1, 56] . Still, none have fully integrated the textual features along with the tabular trial information (such as allocation, primary purpose, and more) in a natural way.Additionally, unlike previous work, the dataset used to train and test our proposed model is much larger than before, including both small molecule trials and biologic trials, extending the previous dataset used by Fu et al. [12, 13] . Also, these works generally optimize the representation learning for a single component in a trial, whereas LINT models a diverse set of trial text and tabular data jointly. Furthermore, previous methods will not work for biologics, which lack information that small-molecule drugs generally possess. [30] enhance the accuracy of [13] by quantifying uncertainty of prediction.  
methodology 2.1 formulation and data featurization: 
 In this section, we formulate the clinical trial outcome prediction problem into a binary classification problem. Additionally, we review all the data features, including trial eligibility criteria, drug descriptions, and disease codes.Let the set of drug or biological interventions be denoted as: where 𝑐 𝑖 is the 𝑖𝑡ℎ inclusion/exclusion criterion sentence, 𝑠𝑢𝑚 is the text summary of the trial, 𝑑 𝑖 is the 𝑖𝑡ℎ associated ICD code, 𝑚 𝑖 is the 𝑖𝑡ℎ associated drug/biologic intervention, and x = {ec_gender, ec_min_age, ec_max_age, allocation, intervention_model, primary_purpose, masking, sponsors, continent} represents a feature vector of tabular features in the trial, described in the following section. Tabular Trial Data: Specifically, input is a list of text. First, each sentence 𝑐 𝑖 is separated out in the trial criteria {𝑐 𝑖 , . . . , 𝑐 𝑁 𝑐 } by splitting into new lines in the raw trial criteria text; these sentences are added to the list. Next, the brief summary information is appended to the list. Finally, additional tabular table information is processed and combined with the previous list (details in Appendix A.5).𝑀 = {𝑚 1 , 𝑚Note that each of the categorical variables also has an additional value for missing values. Finally, each of these categorical features is converted into text by the simple function template linearization(x) = "[feature_name] [feature_value]; . . . ; [fea-ture_name] [feature_value]". Data Processing for ICD Codes: To obtain a list of relevant ICD Codes, we use the API provided by https://clinicaltables.nlm.nih. gov/ by inputting the conditions associated with each trial.2 https://dev.drugbank.com/guides/fields/drugsLINT learns a classifier ŷ = 𝑓 𝜃 (𝑇 ), where 𝑇 is the trial with the associated drug intervention information as above. More specifically, LINT has 2 main modules, the transformer module, and the GRAM module.𝑓 𝜃 (𝑇 ) = 𝑓 𝜃 ′ (𝐶𝑂𝑁𝐶𝐴𝑇 (ℎ 𝑡𝑒𝑥𝑡 , ℎ 𝐺𝑅𝐴𝑀 )),where 𝑓 𝜃 ′ is a classifier trained on the output of LINT's inner components, a transformer model that obtains embeddings ℎ 𝑡𝑒𝑥𝑡 = 𝑓 𝜃 𝑡𝑒𝑥𝑡 (𝑐 𝑖 , . . . , 𝑐 𝑛 𝑐 , 𝑠𝑢𝑚, 𝑚 𝑖 , . . . , 𝑚 𝑛 𝑚 , 𝑥) and a GRAM model that obtains embeddings respectivelyℎ 𝐺𝑅𝐴𝑀 = 1 𝑛 𝑑 𝑛 𝑑 ∑︁ 𝑖=1 𝐺𝑅𝐴𝑀 (𝑑 𝑖 )for a specific trial. Note that indexes for the arbitrary number of codes, eligibility criteria, and interventions are omitted. LINT Encoder: Text Data: This module handles all the text information in our input data and outputs embedding ℎ 𝑡𝑒𝑥𝑡 . We use a pretrained BERT model to extract the text features. BERT (Bidirectional Encoder Representations from Transformers) is a pretraining technique that captures language semantics and exhibits state-ofthe-art performance in various NLP tasks [9] . Specifically, we use BioBERT [27] as implemented in the HuggingFace library [51] . BioBERT is pretrained on biomedical corpora-namely-PubMed abstracts and PubMed Central (PMC) full-text articles. This offers higher performance on domain-specific tasks such as biomedical relation extraction.Due to the small number of trials, we use the transformer encoder to obtain a weighted mean over the original BioBERT embeddings. This allows LINT to take advantage of the powerful PLM while adding an additional layer of attention.Specifically, 𝑓 𝜃 𝑡𝑒𝑥𝑡 is consists of a PLM 𝐿𝐿𝑀 () and a transformer encoder 𝐸𝑁𝐶𝑂𝐷𝐸𝑅(). 𝐿𝐿𝑀 () simply takes in a varying-length text input and outputs a 768-dimensional embedding. 𝐸𝑁𝐶𝑂𝐷𝐸𝑅() is a transformer encoder 3 combined with a simple Linear layer that takes in a varying-length list of 768-dimensional inputs and outputs a scalar attention weight for all inputs, i.e., 𝐸 = 𝐿𝐿𝑀 (𝑐 𝑖 , . . . , 𝑐 𝑛 𝑐 , 𝑠𝑢𝑚, 𝑚 𝑖 , . . . , 𝑚 𝑛 𝑚 , 𝑥)𝑊 = 𝐸𝑁𝐶𝑂𝐷𝐸𝑅(𝐸) 𝑓 𝜃 𝑡𝑒𝑥𝑡 = 𝐸 • 𝑊 ,where 𝐸 ∈ 𝑅 𝐵×𝑛 ,𝑊 ∈ 𝑅 𝑛 . 𝐵 is 768 in this case, and 𝑛 = 𝑛 𝑐 + 1 + 𝑛 𝑚 + 𝑙𝑒𝑛(𝑥), i.e., the total length of the list of text inputs where 𝑛 𝑐 is the number of eligibility criteria, 1 represents the summary embedding, 𝑛 𝑚 is the number of molecules, and 𝑙𝑒𝑛(𝑥) the length of the tabular data of the specific trial. GRAM Encoder: Disease Code Representation: This module handles all the ICD code information in our input data outputs embedding ℎ 𝐺𝑅𝐴𝑀 .Disease codes are typically hierarchically organized in a directed tree. For example, ICD-10 [36, 53] codes consist of three to seven characters. The beginning 3 characters represent the category to which the code belongs to: e.g., the range of A00-B99 consists of "Certain Infectious and Parasitic Diseases". The latter characters represent the specific condition: e.g., A15.0 represents "Tuberculosis of lung".We leverage graph attention-based model (GRAM) [7] to represent disease code. Specifically, suppose 𝑑 is the disease code of interest, the ancestors of 𝑑 are D. Then, GRAM yields an embedding for a given ICD code by the formulation: 𝐺𝑅𝐴𝑀 (𝑑) = 𝑑 𝑗 ∈ D∪{𝑑 } 𝛼 𝑗 𝑑 𝑗 , where 𝛼 𝑗 are the learned attention weights for each level of ICD code.Cross entropy loss is leveraged to guide the training 𝐿 = 𝑁 𝑖=1 -𝑦 log ŷ, where 𝑦 ∈ {0, 1} and ŷ ∈ (0, 1) denote groundtruth and prediction, respectively.LINT Classification: Finally, the outputs of the 2 previous subsections are concatenated and input to a multilayer perceptron (MLP) for binary classification. Because our data is imbalanced, we use a weighted binary-cross-entropy loss, where the weight is calculated based on the label distribution in the training data. ℓ (𝑥, 𝑦) = -2 𝑐=1 𝑤 𝑐 log ŷ𝑐 𝑦 𝑐 Where 𝑦 𝑖 is the predicted probability of class 𝑖 and 𝑤 𝑖 is the class weight of 𝑖. 
experiments: 
 In this section, we present the experimental results. We start by describing the experimental setup, including dataset and splitting (Section 3.1). Finally, we present the trial outcome prediction results and analysis (Section 3.2).All the historical clinical trial records can be downloaded from https://clinicaltrials.gov/. The trial success information is based on the benchmark from Fu et al. [13] 4 .Table 1 : A breakdown of the trials after preprocessing of the 426,368 found on clinicaltrials.gov. No Label indicates the lack of success or failure label found. Non-Biological or Small-Molecule indicates that the drug does not have either of these interventions (i.e., those clinical trials may involve medical devices and behavioral interventions). We follow the labels and data splits provided by Fu et al. [13] . A further breakdown of the labeled data is shown in Table 2 . [5, 13, 14] . This split strategy is to circumvent the information leakage because later trials are typically relying on the knowledge from the earlier trials. All trials whose completion date is confirmed to be before the cutoff are considered training and validation data; otherwise, they are considered test data. Finally, we also split by Phase and Intervention Type. We consider predicting trial outcomes from phases 1, 2, and 3. There are 3 interventions that we consider: only small molecule drugs, only biologics, or both. Table 2 shows the full information regarding splits based on drug type and phase.Information regarding baselines-Logistic Regression (LR), Support Vector Machine (SVM), Decision Tree (DT), AdaBoost (AB), Random Forest (RF), and HINT-are shown in Appendix A.4.3.In this section, we present and analyze the experimental results. Table 4 reveals that LINT excels at Phase 3 prediction, garnering Test AUC scores of 0.770, 0.740, and 0.745 for biologic, drugs, and combined predictions, respectively. While the model delivers a solid performance in Phase 1 prediction, its overall performance dips in Phase 2. This downturn is expected, given that Phase 2 has the highest trial volume and is generally the most challenging task.Table 3 comparison reveals LINT surpasses all baseline models in all metrics. The simple Logistic Regression with our BERT embedding input, the second-best model, achieves Phase 3 Test F1 scores of 0.865, 0.800, and 0.806, which still lags considerably behind LINT. Note that HINT is solely applicable in Drugs mode, as it doesn't cater to Biologics.From Figure 2 , we can visualize the probability of actual success vs. model-predicted success for phase 3, and from Figure 6 we see the number of predictions by probability. Most of the predictions have higher scores (which makes sense as there are more positive true labels). We generally see a positive correlation between the predicted vs. actual success probability, which confirms that our model is well-calibrated, even without explicit tuning for calibration. One notable exception can be seen with the predictions greater than 0.9. This is because there are very few predictions for biologics that have normalized probabilities larger than 0.9, as shown in Figure 6 (exactly one trial). This could be because of the noise level in the combined labels of all drugs and biologics; regardless, further work should aim to factor in calibration as a metric for model performance to improve human trust. Additionally, we observe that model predictions below 0.4, 0.2, and 0.4 yields entirely true negative predictions. Performance Breakdown by Disease Type:  
conclusion: 
 Clinical trial outcome prediction is vital for predicting the safety of new drugs and biologics. In this paper, we focus on developing a machine learning model to predict the outcome of clinical trial that can account for biologics, a quickly growing intervention type. Specifically, we propose an open-source, flexible framework that is built on top of pretrained language models-LINT-a method that supports the accurate prediction of success in clinical trials.Thorough empirical studies are carried out to validate the effectiveness of the proposed method, which achieves state-of-the-art ROC-AUC scores on predicting approval of phase III trials, beating many traditional and recent baselines. We validate the effectiveness of LINT with thorough experiments across three trial phases. Specifically, LINT obtains 0.770, 0.740, and 0.748 ROC-AUC scores on phase I, II, and III, respectively for clinical trials with biologic interventions. We also show that LINT is generally well calibrated and demonstrated LINT's performance on the top-5 most popular categories of ICD codes. Additionally, using Shapley values, we visualize the portions of the input text that are the most significant for the prediction of success/failure. 
: 
 others, informing their scientific and investment decisions. Inaccurate risk evaluation can lead to grave mistakes in drug development choices [52] . Moreover, given the high costs and generally low success rates of trials, it is crucial to prioritize correctly. For example, approval rates for oncology drugs that enter clinical development are estimated to be as low as 3.4-19.4%, 8.7-25.5% for Cardiovascular, 8.2-15% for Central Nervous System, etc [4, 10, 52, 53] .Drug research usually involves two phases: drug discovery and drug development. The goal of drug discovery is to design diverse and novel drug molecular structures with desirable pharmaceutical properties, while the goal of drug development is to evaluate the effectiveness and safety of the drug on human bodies via clinical trials.A drug needs to pass three phases of clinical trials to be approved and enter the medical market. Specifically, Phase I trials mainly focus on the safety and dosage of the drug molecules to human bodies (20 to 80 participants, several months, 70% of drugs pass this phase), Phase II focuses on efficacy and side-effects (100 to 300 participants, could take several months to 2 years, 33% of drugs pass this phase), and Phase III focuses on efficacy and monitoring of adverse reactions on broader population in treating disease (200 to 3,000 participants, 1 to 4 years, 25-30% of drugs pass this phase) [8] .From a financial perspective, creating a novel drug-based treatment typically requires around 13-15 years and more than 2 billion dollars in research and development [3, 11, 33, 42] . From 2009 to 2018, the FDA approved 355 new drugs and biologics. As a concrete example to see the scale of the cost it takes to bring a drug to market: to treat heart failure, Novartis sponsored phase III of Entresto (a small-molecule drug), which recruited 4822 patients and spanned five years (2014-2019) which eventually yielded results that beneficial, but not statistically significant effects 1 [23, 29, 44, 45, 47] . If machine learning models can predict the approval rate of a clinical trial before it starts, we could circumvent running high-risk trials that are likely to fail, which would save a significant amount of time and resources. Some of the main challenges for building accurate machine learning models for predicting trial outcomes are 1) limited training data and 2) diverse trial types. Although clinical trial summaries are publicly available on clinicaltrials.gov, the limited number of labeled trials (success and failure) may be insufficient for training sophisticated machine learning models. Additionally, clinical trial descriptions vary significantly, with biologics often lacking typical molecular property information compared to conventional small-molecule drugs.Despite these limitations, there are numerous opportunities for machine learning in this space. For example, a plethora of unstructured text is available that describes the various aspects of the trial and the drugs. Additionally, there also exists literature that describes the pharmaceutical properties of drug molecules (Absorption, Toxicity, etc).In recent years, there have been numerous breakthroughs in the natural language processing (NLP) community for clinical applications. For example, models can learn semantic knowledge from massive unlabeled data [27] , classify unstructured medical text with no human labels [17] , process clinical trial tabular data [48, 49] , and more [46] . Inspired by recent trends in the NLP space, we propose a novel methodology: LLM-based Interaction Network (LINT). Our model builds on pretrained language models (PLM) to predict trial outcomes by jointly considering the text descriptions of the trial, its associated drugs, and the corresponding medical codes.We formally define LINT, a deep learning framework for clinical trial outcome prediction that can predict outcomes on both small molecule drugs and biologics on Phase I, II, and III clinical trials taken from the largest labeled trial dataset. It leverages text features and International Classification of Diseases (ICD) Codes to accurately predict the approval of interventional trials with both small-molecule drugs and biologics. These text features include trial eligibility criteria, trial design specifications, drug property descriptions, and implicit PLM knowledge. In short, LINT learns a function on a weighted combination of pretrained large language model (LLM) embeddings to predict clinical trial outcomes.• Experimental results. The proposed method achieves state-ofthe-art performance and beats traditional models. Specifically, LINT obtains ROC-AUC scores of 0.723, 0.702, 0.770 (and F1 scores of 0.643, 0.654, and 0.740) on phase I, II, and III, respectively for clinical trials with biologics interventions, which are significantly better than other baselines (Section 3). • Open Source Code. LINT is an open-source, flexible framework that is built on top of pretrained language models (PLM); thus, it is easily adapted when a novel PLM is released. The code will be publicly available at https://github.com/chufangao/LINT • Interpretability and Validation. Although LINT is a deep neural network, we may use Shapley values (a measure of feature importance based on the average contribution of that word to the final prediction probability) to interpret its decisions and visualize which portions of the input text are the most significant [20] . Furthermore, we will show in later sections that LINT's predicted score generally corresponds to the actual approval rate (see Section 3.2). 
 
 Appendix: Future Work: Future research should address the substantial lack of clear outcome labels in clinical trial datasets, an issue currently unaddressed, with potentially over 100,000+ unlabeled trials, as indicated in Table 1 . Unsupervised learning strategies, such as masked language modeling, weak supervision, or semi-supervision, could be instrumental in resolving this. It's also crucial to improve label quality, a challenging task due to the complex language in result descriptions. Endeavors to identify human-interpretable automatic labels could help expand the dataset. Interpretability is also key for decision-makers. Understanding why a trial is predicted to fail or succeed, rather than simply knowing the outcome, is valuable. While LINT can be paired with Shapley values to highlight text sections affecting prediction confidence, it doesn't directly account for human interpretability. Future research should focus on creating interpretable models, which could optimize clinical trial design and enhance success rates.Table 5 shows relevant notation used in the paper.In this section, we provide additional Tables that are not featured in the main text due to length limitations. For example, Table 6 shows LINT performance on popular ICD categories.In this section, we briefly review the related literature. In recent years, there have been several attempts to use machine learning to predict clinical trial outcomes.A.4.1 Traditional Machine Learning Methods. Lo et al. [28] applied traditional machine-learning techniques (penalized logistic regression (PLR), random forests (RF), neural networks (NN), gradient boosting trees (GBT), support vector machines (SVM) [37] , and decision trees C.50 [26] ) to predict drug approvals using drug and clinical trial data. Payvert et al. [21] introduced PrOCTOR, a random forest-based model to predict drug toxicity using 10 molecular properties, 34 target-based properties, and 4 drug-likeness rule features. Hong et al. [24] designed an ensemble classifier based on weighted least squares support vector regression (LS-SVR) to predict the success/failure of clinical trials.Wu et al. [54] developed a two-stage SVM classification method to identify genes and genetic lesion statuses in clinical trials. Raj et al. [39] used Gradient-Boosted Decision Trees (GBDT [55] ) to predict patients who responded to treatment on various depressive symptoms utilizing pretreatment symptom scores and electroencephalographic features. Siah et al. [43] created an open challenge and compared over 3000 various machine learning models for clinical trial outcome prediction. They found that the best-performing model was an ensemble consisting of two XGBoost models [6] and one Bayesian logistic regression (BLR) model [22] .Our proposed model LINT differs from the previously mentioned methods in several ways. Most do not consider drug molecule features and trial protocol information jointly, rather generally [14, 25] , patient retrieval [19, 58] and enrollment [1] .Doctor2Vec [1] , a recently proposed hierarchical clinical trial embedding where the unstructured trial descriptions were embedded using Bidirectional Encoder Representations from Transformers (BERT) [9] .DeepEnroll [58] leverages a hierarchical embedding model to represent patient longitudinal electronic health record (EHR) and aligns it with eligibility criteria (EC) via a numerical information embedding and entailment module to reason over numerical information in both EC and EHR.Gao et al. [19] proposed a patient-trial matching model to find qualified patients for clinical trials given structured EHR and unstructured EC text with both inclusion and exclusion criteria. The core of this model consists of a convolutional highway network and a hierarchical memory network that generates a contextualized word embedding for each word of the trial protocol. Multiple onedimensional convolutional layers with varying kernel sizes capture semantics at different granularity.Qi et al. [38] designed a Residual Semi-Recurrent Neural Network and took phase 2 results as features to predict the phase 3 outcome. This network consists of an RNN with a residual connection from the first input and performs significantly better than RNNs. The trough concentration (Ctrough) and Phase 2 subject-level baseline characteristics were used to build an individual treatment effect (ITE) model for Phase 3 trial patients.Fu et al. [13] designed a Hierarchical Interaction Network (HINT) to capture the interaction between multi-modal features (drug molecules, disease codes, eligibility). It uses an interaction graph module on embeddings produced via domain knowledge to capture various relations between EC, molecule structure, trial protocol, and more to predict trial outcomes. However, this work does not support biologics-related interventions because the lack of protein structures and molecule properties such as absorption, distribution, metabolism, excretion, and toxicity (ADMET) are not known.A.4.3 Baseline Methods. We employ the following baseline methods for clinical trial outcome prediction and compare them with our proposed LINT method. Each baseline method has the average text embedding of the input texts as well as the GRAM embeddings of the ICD codes for diseases addressed in the clinical trials.• Logistic Regression is a common model that models the logodds for a class through a linear combination of the input features, similar to a simple one-layer neural network with a logistic activation function [28] . • Supporting Vector Machine (SVM) is another common linear model that attempts to fit a maximum-margin hyperplane between the input features (often using a nonlinear kernel function) in order to separate classes [54] . • Decision Tree is a hierarchical, rule-based model that's generally trained using algorithm that attempts to iteratively split on a feature using an information-theoretic measure like label entropy at each branch of the tree [28] . • AdaBoost is a meta-estimator that iteratively fits a decision tree and then fits additional copies of the classifier on the re-weighted dataset (where weights of incorrectly classified instances are increased to emphasize them) [13] . • Random Forest is an ensemble of decision trees trained on different sub-samples of the input data (usually via sampling with bootstrapping) [28] . • Hierarchical Interaction Network (HINT) is the previous state-of-the-art model we compare against. It is a complex model, consisting of a graph attention network, highway networks, and more to combine drug structure, eligibility criteria, and ICD codes in order to make a binary trial success prediction [13] .We describe the processing for each of the tabular features in x.• ec_gender: The eligible patient genders (male/female / or either) that the trial considers from eligibility criteria. • ec_min_age, ec_max_age: The minimum and maximum ages of patients selected via the eligibility criteria. Notes that we convert the valid age range of the eligibility criteria to 4 bins following the Research Inclusion Statistics Report from the NIH [35] . Ages below 6 are considered children, ages 6-18 are considered adolescents, and ages 18-65 are considered adults. Finally, ages higher than 65 are considered older adults. I.e., 4 bins of (<6, 6-18, 18-65 and >65). • allocation: the treatment allocation, which can be randomized or nonrandomized.• intervention_model: The general design of the strategy for assigning therapies and can be Crossover, Factorial, Parallel, Sequential, or Single Group Assignment. • primary_purpose: Describes the trial purpose, including Basic Science, Diagnostic, Educational/Counseling/Training, Health Services Research, Prevention, Screening, Supportive Care, Treatment, or Other. • masking: The type of method (single, double, triple, or quadruple masking) used to keep the study group assignment hidden after allocation between parties (Participants, Care Providers, Investigators, and Outcomes Assessors). • sponsors: The organization that oversees the trial. Since there are hundreds of possible sponsors, we simply denote separate sponsors into Large or Small, where large sponsors are the top 10 most common sponsors over all trials 5 . • continents: The continents in which the study was performed were converted from the raw trial "country" data. We conduct ablations by excluding text features from the trial summary, trial tabular data, eligibility criteria, and drug information, as shown in Table 7 . Interestingly, removing certain text features marginally boosts ROC-AUC performance, but at the expense of F1 and Accuracy. Furthermore, excluding the trial summary significantly impairs LINT's performance, with all metrics dropping. This could elucidate why removing other features doesn't impact the results, as the model appears to predominantly rely on the trial summary.Despite this, LINT might still find other outcomes important, yet it primarily hinges on the trial summary, thereby heavily influenced if it's absent. Future studies should delve into this dependency since other text features also significantly contribute to trial outcome prediction.A.6.1 Case Studies. In this section, we take a closer look at 2 different biologic drugs. Secukinumab and Botulinum toxin type A, two random examples chosen from the test set, for the fair evaluation of LINT.Case 1: Secukinumab (NCT02404350)6 [34, 53] . First, let us take a look at a successful test prediction. As a brief summary, Novartis Pharmaceuticals sponsored this study, and the main goal of this study was to demonstrate efficacy on inhibition of progression of structural damage of secukinumab in subjects with active Psoriatic Arthritis (PsA) as measured by improvement in physical function measured by Health Assessment Questionnaire and skin and nail improvement for psoriasis signs. This study was deemed successful and had P-value of less than 0.0001 in its Estimation Parameter of Odds Ratio. LINT predicted success with a normalized score of 0.79 (from 0 to 1).Case 2: Botulinum toxin type A (NCT02660359) 7 Here, we take a look at an incorrect test prediction. This study was sponsored by Ipsen, and the primary purpose of this trial was to analyze safety and efficacy of two Dysport [40] (Similar to Botox) doses (600 units U and 800 U), compared to placebo in reducing urinary incontinence (UI) in adult subjects treated for neurogenic detrusor overactivity (NDO) due to spinal cord injury (SCI) or multiple sclerosis (MS). This study was annotated to be unsuccessful due to the lack of participants; however, upon further analysis of the trial, all p-values were significantly less than 0.05, indicating statistical significance. This could indicate that if given more participants, a successful outcome may have been possible. LINT predicted success with a normalized score of 0.38 (from 0 to 1). 8 shows a full example text input to LINT.Figure 7 shows the text Shapley (SHAP) value importance. The base value as shown in the upper left of the plot is the model logit output when the entire input text is masked. Using Shapley values [31, 32] from the package shap.readthedocs.io, we are additionally able to visualize the portions of the raw input text that affect model output the most (See Figure 7 ). The SHAP values additively explain the impact of unmasking each word on the model output-from the base value (where the entire input is masked) to the final prediction value (no mask). In short, the darker the color of the highlighted text, the more attention the model pays to it in its final classification. In short, Shapley values are the average marginal contribution of a feature value across all possible coalitions (combinations of features).Features are masked out by random sampling from the existing dataset when not considered for traditional tabular feature classification. In this case, masking is done by replacing words with the [MASK] token by the SHAP package. Section A.6.1 shows an example of this. From this, we see that the model generally does rely on informative portions of text. The purpose of this study is to provide confirmatory evidence of the safety and efficacy of two Dysport® doses (600 units [U] and 800 U), compared to placebo in reducing urinary incontinence (UI) in adult subjects treated for neurogenic detrusor overactivity (NDO) due to spinal cord injury (SCI) or multiple sclerosis (MS). Galantamine is a tertiary alkaloid and reversible competitive inhibitor of the acetylcholinesterase AChE enzyme which is a widely studied therapeutic target used in the treatment of Alzheimers disease.A1018 First characterized in the early 1950s galantamine is a tertiary alkaloid that was extracted from botanical sources such as Galanthus nivalis.A201968 Galantamine ...Galantamine is a competitive and reversible inhibitor of acetylcholinesterase that works to increase acetylcholine levels.L13571 Galantamine acts both centrally and peripherally to inhibit both muscle and brain acetylcholinesterase thereby increasing cholinergic tone.A201968 Galantamine is also a positive allosteric modulator of neuronal nicotinic acetylcholine receptors.A1022A201968 As dementia is a progressive neurodegenerative ...The oral LDsub50sub of the active ingredient galantamine hydrobromide in rats is 75 mgkg.L13709 Symptoms of overdose are expected to be similar to those of cholinomimetics which involve the central nervous system the parasympathetic nervous system and the neuromuscular junction. Effects of a cholinergic crisis include severe nausea vomiting gastrointestinal ...In vitro study findings suggest that about 75 of the drug is metabolized by CYP2D6 and CYP3A4. CYP2D6 promotes Odemethylation of the drug to form Odesmethylgalantamine and the CYP3A4mediated pathway forms the galantamineNoxide.A182993 Important metabolic pathways also include Ndemethylation epimerization and sulfate conjugation.A203444 Other metabolites include norgalantamine Odesmethylgalantamine Odesmethylnorgalantamine epigalantamine ...Over a dose range of 832 mgday galantamine exhibits a doselinear pharmacokinetic profile. The oral bioavailability of galantamine ranges from 90100. Following oral administration the Tmax is about 1 hour.L13571 Following 10 hours of administration the mean galantamine plasma concentrations were 8297 gL for the 24 mgday dose and 114126 ...https://clinicaltrials.gov/ct2/show/study/NCT01920711Top 10 sponsors: GlaxoSmithKline, Merck Sharp & Dohme LLC, Sanofi Pasteur, a Sanofi Company, Amgen, Pfizer, National Cancer Institute (NCI), Novartis Pharmaceuticals, Abbott, Bristol-Myers Squibb, Novartis Vaccineshttps://clinicaltrials.gov/ct2/show/study/NCT02404350https://clinicaltrials.gov/ct2/show/results/NCT02660359 



Reviews: review: title: title: Review for Paper #3
main_review: main_review: Summary

The author proposed the Language Interaction Network (LINT) method. This method utilizes natural language processing techniques and only uses the text descriptions of clinical trials for prediction. Specifically, LINT consists of two modules: one is a text encoder based on the pre-trained language model for processing text data; the other is a disease encoder based on the graph attention mechanism for processing disease codes. The outputs of these two modules are concatenated and input into a multi-layer perceptron for binary classification.

Advantages: 
1. A new method for predicting the clinical trial results of biologics has been proposed, addressing the shortcomings of existing methods in this regard.
2. The experimental results show that this method has good performance in predicting clinical trial results.
3. The method is simple and feasible, and easy to expand and apply.

Disadvantages: 
1. There is a lack of a framework diagram in the manuscript to help readers understand the structure of LINT. 
2. Insufficient novelty. The performance improvement of this method is merely achieved by using more modalities of data. The processing of each modality is based on existing methods, and the interaction between modalities is a simple concat, lacking more novel technical contributions. 
3. The baseline models for experimental comparison are very strange. The author mentions in the paper, "Each baseline method is upgraded with the same input embedding used in LINT", which means the difference between the baseline models and LINT is only the final classifier (i.e., the MLP of LINT). The author should compare with more related works, such as Doctor2Vec and DeepEnroll mentioned in the manuscript.

Other issues: 
1. The author should provide the full name of GRAM when it first appears in the manuscript. 
2. Some double quotation marks have formatting problems, such as in lines 237 and 300. 
3. The author needs to maintain the consistency of wording in the manuscript. For example, the "Mode" column in Tables 2, 3, 4, and 5 uses completely inconsistent writing styles.
title: title: Review for Paper #3
main_review: main_review: **Summary:**
The paper introduces the Language Interaction Network (LINT), a framework designed to predict clinical trial outcomes using free-text descriptions, particularly for biologics lacking traditional molecular data. LINT leverages pretrained language models and a graph attention model to process trial descriptions, drug properties, and disease codes.

**Pros:**
- The writing and presentation are nice and clear, easy to follow.
- Transparency: the code is also clear.
- The significance of this work is substantial, as illustrated in the introduction.

**Cons:**
- It would be better to include a model architecture figure.
- The GRAM Encoder could be written in the preliminaries. The phrase "See Section 2.2" is also in Section 2.2.
- How is the validation set split? The table shows the partitions of the training set and test set, but a more detailed validation set split strategy should be included.
- Though the paper claims it beats the latest baselines, are LR, SVM, DT, AdaBoost, and RF the latest baseline methods?