SlowGuess/NAACL_2021_OCR huggingface dataset:
Random model: couples' similarity was randomly scored.
529 Metrics
530 Metrics calculated in this experiment were Precision-Recall Area Under the Curve (PR-AUC) and Area Under Receiver Operating Curve (ROC-AUC), F1-score, precision, recall. 534 Accuracy was an additional metric calculated during the training, on the balanced validation set.
536 1-shot evaluation assay
A similarity score was assigned to the 32 couples in the batch. If the couple scored most similar was the only couple of trials belonging to the same pathway the batch assay was positive, otherwise negative. The final score was calculated as the percentage of positive hits.
Analysis of the model's feature contribution
544 To identify the impact of each feature on the   
545 overall EnSidNet prediction, a SHAP analysis has   
546 been performed on a subset of 10 positive and 10   
547 negative test data.
548 Prediction pipeline
One of the greatest challenges in implementing a Siamese neural network setting to identify new drug-development pathway (de-novo or completing existing ones) is the number of trials that need to be matched. With more than 140,000 trials, many of which started in the last 20 years, it would be impractical to compare all trials against each other.
The first step of the proposed pipeline was the selection of relevant trials. Trials may be stratified based on the type of sponsor (research institute or pharmaceutical company), the purpose of the trial (e.g. treatment, diagnostic, basic science), phases (phase 4 trials are beyond the scope of this research, so they would be excluded). This first step can reduce the number of trials to match by a factor of 10.
The trials were then divided in buckets based on their therapeutic area. We follow the Medical Dictionary for Regulatory Activities (MedDRA) terminology. The MedDRA System Organ Class (SOC) term was used to represent the therapeutic area. It is rare for trials from the same pathway to include patients affected by pathologies from different MedDRA SOC terms. Dividing the trials into therapeutic area decreased the algorithm complexity. Trials belonging to multiple therapeutic areas were duplicated.
577 If previous pathways exist for the therapeutic area the algorithm tried to expand them with new trials.
Trial expansion was performed in a setting like 1-shot evaluation. One unmatched trial was compared with 2 trials chosen randomly from all the pathways. The trial was considered to belong to the pathway if the prediction obtained for both trials was higher than a threshold (e.g. 0.8).
586 Corner cases in which trial A and B were matched below the threshold but trial C matched with trial A above the threshold as well as trial B and trial C, were considered a pathway (consisting of trial A, B, and C); this assumption may increase the false positive rate trials in pathway but ensures that all possible clinical trials matching are grouped; the human labelling step would exclude the clinical trials not matching the pathway.
The following step grouped the remaining trials into pathways. To increase the matching probability trials were sorted (for example based on popularity of lead sponsor or condition), then they were divided into batches (in the experiments the batches had 200 trials). Trials within a batch where completely matched. Positive matching was considered for the couples with predictions above a threshold (e.g. 0.8). Matched couples with one trial in common were then grouped into a possible pathway.
To allow grouping of matched trials across batch 1 trial for all possible pathways was matched in an 'all-versus-all' setting, and inter-batch grouping was performed again.
The matching step was repeated 3 times, to ensure the maximum matching of trials.
Once all possible pathways for all therapeutic areas were obtained, the results could be submitted to the human labelers for pathway confirmation.
The false positive couples would be paramount for a second re/training of the algorithm.
618 Human evaluation of predicted pathways
619 A subset of the predicted possible pathways across 620 the therapeutic areas (1-3 predicted pathways for 621 each therapeutic area) was sent to human scorers. 622 Trials in the correct pathway kept the drug 623 development pathway identification number, 624 while trials belonging to a different or new 625 pathway changed the drug-development pathway 626 identification number accordingly. The statistics 627 of the predicted and confirmed pathways can be 628 found in Supplementary Table 4.
NCT ID	Intervention	Condition	Phase	Sponsor	Lead Sponsor	Countries	Date (dd/mm/yy)
Start	End
NCT02632708	cytarabine, AG-221, mitoxantrone, daunorubicin, etoposide, idarubicin, AG-120	Newly Diagnosed Acute Myeloid Leukemia (AML), AML Arising From Myelodysplastic Syndrome (MDS), AML Arising From Antecedent Hematologic Disorder (AHD), AML Arising After Exposure to Genotoxic Injury, Untreated AML	1	Agios Pharmaceuticals, Inc., Celgene Corporation	Agios Pharmaceuticals, Inc.	Germany, Netherlands, United States	31/12/15	1/7/23
NCT02073994	AG-120	Cholangiocarcinomas, Gliomas, Chondrosarcomas, Other Advanced Solid Tumors	1	Agios Pharmaceuticals, Inc.	Agios Pharmaceuticals, Inc.	France, United States	1/3/14	1/6/21
NCT02489513	[14C]-AG-120	Healthy Volunteers	1	Agios Pharmaceuticals, Inc.	Agios Pharmaceuticals, Inc.	United States	1/6/15	1/10/15
NCT02677922	Azacitidine, AG-120, AG-221	Leukemia Acute Myeloid Leukemia (AML)	2	Celgene	Celgene	Australia, Canada, France, Germany, Italy, Republic of Korea, Netherlands, Portugal, Spain, Switzerland, United Kingdom, United States	3/6/16	31/10/21
NCT02831972	Itraconazole, AG120	Healthy Volunteers	1	Agios Pharmaceuticals, Inc.	Agios Pharmaceuticals, Inc.	United States	1/6/16	1/10/16
NCT02989857	AG-120 matched placebo, AG-120	Metastatic Cholangiocarcinoma, Advanced Cholangiocarcinoma	3	Agios Pharmaceuticals, Inc.	Agios Pharmaceuticals, Inc.	United States	1/1/17	1/8/20
Supplementary Table 1: Example of a drug-development pathway. Different trials are conducted by pharmaceutical companies to obtain proof of safety and efficacy of the drug before submitting the results to regulatory agency for drug approval
Number of parameters to train	Average training time (seconds/epoch)
SNN	2,074,497	185.525
SiD NN	1,069,473	96.35
EnSidNet	1,079,681	98.175
Supplementary Table 2: Complexity of the models used for the experiment
629 B. Supplementary Tables.
FDA Snapshot	Hand-curated
# of pathways	116	20
# trial/pathway range	1 - 7	1 - 14
25 percentile # trials	1	2
50 percentile # trial	1	4
75 percentile # trials	2	7
Supplementary Table 3: Statistics on the datasets
Predicted	Confirmed
pathways count	73.000	165.000
trials in pathways	mean	3.616	1.600
std	5.619	1.258
min	2.000	1.000
25 percentile	2.000	1.000
50 percentile	2.000	1.000
75 percentile	3.000	2.000
max	49.000	11.000
Supplementary Table 4: Difference in the distribution of the trial number in predicted vs human checked (confirmed) pathways
NCT ID	Intervention	Condition	Phase	Sponsor	Lead Sponsor	Countries	Date (dd/mm/yy)
Start	End
Matched	NCT01340872	ST10-021, Placebo Comparator	Ulcerative Colitis, Iron Deficiency Anemia (IDA), Inflammatory Bowel Diseases (IBD)	3	Shield Therapeutics	Shield Therapeutics	Austria, United Kingdom	1/8/11	1/10/14
NCT02968368	Placebo, Ferric maltol	Iron-Deficiency Anemias, Renal Insufficiency, Chronic	3	Shield Therapeutics	Shield Therapeutics	United States	1/12/16	1/8/18
Not Matched	NCT02946463	Eculizumab, Ravulizumab	Paroxysmal Nocturnal Haemoglobinuria (PNH)	3	Alexion Pharmaceuticals	Alexion Pharmaceuticals	France, Japan, Republic of Korea, United States	20/12/16	1/1/23
NCT01711359	Baricitinib, Baricitinib Placebo, Folic Acid, MTX Placebo, Methotrexate	Rheumatoid Arthritis	3	Eli Lilly and Company	Eli Lilly and Company	Argentina, Austria, Belgium, Brazil, Canada, Germany, Greece, India, Italy, Japan, Republic of Korea, Mexico, Portugal, Puerto Rico, Russian Federation, South Africa, Sweden, United Kingdom, United States	1/11/12	1/8/15
Supplementary Table 5: Example of a trial couple belonging to the same drug-development pathway (NCT01340872 and NCT02968368) and a trial couple belonging to different drug-development pathway (NCT02946463 and NCT01711359)
# total couples	# positive couples	# positive couples' pathways	# positive couples from snapshot pathways	# positive couples from oncology pathways
Training and validation set	2720	1360	112	101	11
32-way 1-shot validation set	1056	33	33	27	6
32-way 1-shot test set	1056	33	33	29	4
Supplementary Table 6: composition and origin of the datasets
Supplementary Figure 1: Representation of the 3 Neural Network architectures and modules: numerical inputs in gold dashed rectangle (present in the architecture of SNN), additional inputs and a concatenation layer in green dashed rectangle (architecture of SiD NN) and the fully connected layer as last layer before prediction in dark purple dashed rectangle (together with the green dashed module constitute the EnSidNet architecture). BiLSTM = Bidirectional Long-Short Term Memory; FC = Fully connected.
633 C. Supplementary Figures.
Supplementary Figure 2: Scheme of the matching pipeline. Bold trials in pathways are selected to match to trials not in pathways (here for simplicity only one was selected, in the algorithm proposed they were 2) (1). Couples are built (2) and matching prediction is given (3). Matched trials are combined into existing (primed, e.g. Pathway 1 which included Trial C) or new (de-novo) pathways (e.g. Pathway 3 composed by Trials D and E) (4)
Supplementary Figure 3: Feature contribution analysis