SlowGuess/NAACL_2021_OCR huggingface dataset:
Lucia Pagani
Analytics Center of Excellence, IQVIA
Via Fabio Filzi 29, Milan, Italy
lucia.pagani@iqvia.com
Siamese Neural Networks have been widely used to perform similarity classification in multi-class settings. Their architecture can be used to group the clinical trials belonging to the same drug-development pathway along the several clinical trial phases. Here we present an approach for the unmet need of drug-development pathway reconstruction, based on an Enhanced hybrid Siamese-Deep Neural Network (EnSidNet). The proposed model demonstrates significant improvement above baselines in a 1-shot evaluation setting and in a classical similarity setting. EnSidNet can be an essential tool in a semi-supervised learning environment: by selecting clinical trials highly likely to belong to the same drug-development pathway it is possible to speed up the labelling process of human experts, allowing the check of a consistent volume of data, further used in the model's training dataset.
25 1 Introduction
Siamese Neural Networks (SNN) were developed in the early 1990s (Bromley et al., 1994) to obtain a similarity score from examples of signatures with the goal of identifying forgery. From then many applications used SNN, primarily on image recognition tasks (Chopra et al., 2005). The basic architecture of SNN consists of two identical networks able to learn the hidden representation of the inputs. A similarity function would then compare the inputs hidden representations. The similarity score was taken advantage of in contexts like 1-shot learning in multiclass-
38 classification problems, where a single example 39 of a class was seen by the algorithm only once 40 before making inference (Koch et al., 2015).
Different architectures of SNN were developed in time: Simo-Serra and colleagues developed a 3- inputs SNN (Simo-Serra et al., 2015), where the neural network learned to rank the outputs and identify whether the reference's hidden representation is more similar to a positive or a negative sample.
Another example involves the insertion of an intermediate stage between the similarity score layer and the final prediction layer (Subramaniam, Chatterjee, and Mittal, 2016), allowing to increase performance in person re-identification task despite partial occlusion and difference in point of view or illumination.
The first applications of SNN were based on Convolutional Neural Networks (CNN) to obtain similarity score on images (Simo-Serra et al., 2015), seeing SNN involved in different tasks such as patch identification (Simo-Serra et al., 2015), person identification (Ahmed et al., 2015), image matching from different angles (Vo and Hays, 2016). SNN was also explored in Natural Language Processing (NLP) contexts in tasks like identifying sentence similarity (Mueller and Thyagarajan, 2016) and support relation for argumentation (Gema et al., 2017). These applications highlight the flexibility of SNN to identify similarities in different contexts. Here we apply this architecture on an unmet healthcare task: grouping clinical trials belonging to the same drug-development pathway. Before being released on the market a new drug needs to go through several expensive and time-consuming experiments, involving testing the pharmacological characteristics of the drug in
Association for Computational Linguistics: Human Language Technologies, pages 254-266
biochemical, cellular, and animal models (preclinical phase) and then on human volunteers (clinical stage). The clinical stage is divided into 3 pre-approval phases (safety, efficacy, regulatory proof) and a fourth post-market phase (Corr and Williams, 2009). The experiments performed by research or pharmaceutical companies to study a drug in human subjects are called clinical trials. A drug-development pathway is defined as all the clinical studies performed on a drug for an indication to obtain approval from the regulatory agency. Example of a drug-development pathway is presented in Supplementary Table 1. From starting a phase 1 clinical trial to obtaining approval from a regulatory agency, a drug can be tested for over 10 years, and the process can cost hundreds of millions of dollars, involving thousands of subjects, including patients, doctors, nurses and other personnel, with an approval rate of around  $10\%$  (Wong, Siah, and Lo, 2019).
Information on most clinical trials is publicly available. Pharmaceutical companies are asked to share their information on ClinicalTrials.gov, a U.S. National Library of Medicine resource. Other companies such as DrugBank (Wishart et al., 2006) or Citeline (Wong, Siah, and Lo, 2019) parse the information from ClinicalTrials.gov and add a hand-curation process in which human labellers cross-reference certain information and add additional labels to the trials, resulting in a similar but more accurate database.
Although having information on the clinical trials related to the development of a drug may seem a very straightforward process, there are many confounding factors:
- Very often several trials of the same phase are run, to obtain statistical power or on slightly different protocols (country, population, sample size, ...)
- The same trial can belong to two different phases (e.g. phase 1-2 or 2-3)
- The company may not share on public databases the information of the trials it is performing, or may share partial information or not update them
Some phases may be skipped
- Often subsequent trial phases from the same drug-development pathway may address slightly different diseases
- The disease and the drug can be referred to from different nomenclatures in different trials
Grouping of clinical trials to the same drug-development pathway is a requirement for many different applications, such as analyzing the success of a pharmaceutical company performing trials and marketing new drugs, or calculating the probability of success of a drug for a therapeutic area, evaluating the number of pathway in a therapeutic area, and investigating the futility of a pathway.
Although there is a strong need for a large freely-available dataset, only proprietary hand curated datasets exist (Wong, Siah, and Lo, 2019). A relatively small dataset of regulatory agency approved pivotal trials could be parsed from Food and Drug Administration Drug Trials Snapshots (FDA Snapshot)
(https://www.fda.gov/drugs/drug-approvals-and-databases/drug-trials-snapshots). The lack of large publicly available datasets may be one of the reasons why to our knowledge no algorithms to group clinical trials in drug-development pathways have been described in the literature.
The contributions of this paper are: (a) a novel approach to group clinical trials in drug-development pathways; (b) an iterative semi-supervised learning pipeline to optimize the grouping of clinical trials to the pathway.
The model proposed here is based on a SNN architecture. The model learned the similarity of trials belonging to the same pathway. The advantage of using the proposed model in a semi-supervised learning pipeline would lead to decreased human-labelling effort; the proposed pipeline can work in a de-novo mode (fresh start) and in a primed mode (adding data to previously scored pathways).
164 2 Methods
165 2.1 Data used to train and validate model
166 The ground truth pathways considered in this experiment were pathways extracted by the pivotal trials from the FDA Snapshot and manually identified pathways (hand-curated). For more details on the datasets composition and other methods considered here see Supplementary Methods.
111   
112   
113   
114   
115   
116   
117   
118   
119   
120   
121   
122   
123   
124
173 2.2 Neural Network architectures
174 Three architectures were compared in the current research, schematized in Supplementary Figure 1: pure Siamese Neural Network architecture (SNN) where only Siamese branches were present, a hybrid Siamese and Deep Neural Network (SiD NN) consisting of Siamese character-based branches and an additional input branch, and an enhanced version of the SiD NN, having a fully connected layer before the prediction layer (EnSidNet). Supplementary Methods contain the detailed description of the 3 architectures.
185 2.3 Inputs of the model
186 The input features of the networks were: the drugs   
187 used in the clinical trial (intervention), the disease   
188 considered (condition), the phase of the trial   
189 (phase), the countries where the clinical trial was   
190 conducted (country), the sponsors of the trial   
191 (sponsor), the start and end date of the trial   
192 (expressed in days compared to an arbitrary   
193 reference date, January  $1^{\mathrm{st}}$  2000). Details of the   
194 preprocessing of the inputs can be found on   
Supplementary Methods.
196 2.4 Prediction Algorithm
197 Algorithm 1 contains the pipeline to apply the 198 Neural Network to group trials into pathways.
Input: trials to group in pathways and previously scored pathways
Output: pathways containing development trials
1: divide trials in therapeutic areas
2: for every therapeutic area do
3: for every existing pathway do
4: predict similarity between 2 trials of a present pathway and a new trial
5: if probability  $>0.8$  for both couples do
6: add trial to present pathway
7: sort trials (common lead sponsor or condition)
8: divide trials into batches
9: for every trial in batch do
10: match all versus all and predict similarity
11: if probability  $>0.8$  do
12: group the trials in a pathway
13: group pathways with common trial
14: select 1 trial per pathway and repeat steps 9-13
15: return pathways
199 The details of the pipeline are reported in Supplementary Methods. For schematic example of the matching pipeline see Supplementary Figure 2.
203 3 Experiments
In Supplementary Table 2 we report the number of parameters of the networks and training time. The three neural models have different number of parameters to train, and the complexity of SNN compared to the hybrid models made the training time per epoch longer. In terms of time per epoch the other two hybrid models had comparable time per epoch, despite the slightly higher complexity of EnSidNet compared to SiD NN.
213 3.1 Balanced datasets
Accuracy was tested on a balanced validation dataset (see dataset splitting for details on balanced dataset creation). It can be seen from Table 1 that the best performing algorithm was EnSidNet.
Balanced dataset
Accuracy
SNN	0.763393
SiD NN	0.907738
EnSidNet	0.91369
3.2 32-way 1-shot evaluation performances
220 One-shot evaluation was used to predict whether a new trial belongs to established pathways.
The score expected from a random classifier is 3.125, due to the unbalanced 1:32 ratio of positive couples versus negative. It can be seen in Table 2 that all neural models scored significantly higher than a random classifier in a 32-way 1-shot evaluation assay.
Table 1: Accuracy of the best model on a balanced dataset
32-way 1-shot evaluation assay
Neural Network	1-Nearest Neighbor	Random Classifier
SNN	66.67	81.82	6.06
SiD NN	93.94	69.70	0
EnSidNet	96.97	69.70	3.03
Table 2: Results of 1-shot evaluation assay
EnSidNet was the model with the highest performance in the test set. On the contrary, the SNN had the lowest performance between the neural models. Surprisingly the input format of SNN tested on the heuristic 1-Nearest Neighbor gave a relatively high performance.
To understand the contribution of the different features on the final EnSidNet prediction a SHAP analysis was performed. As Supplementary Figure 3 shows the most important feature to distinguish between couples from the same or different pathway is the number of common sponsors. It is interesting to note that the most contributing features belong to the additional inputs branch of the NN, features that increased the performance of the 32-way 1-shot learning metric of almost  $30\%$  (see Table 2).
245 3.3 Metrics on imbalanced dataset
Table 3 shows the other metrics considered in this research, calculated on the 1:32 unbalanced dataset.
Unbalanced dataset
F1	P	R	ROC AUC	PR AUC
SNN	0.16	0.09	0.76	0.85	0.61
Sid NN	0.90	0.86	0.94	0.97	0.89
EnSidNet	0.90	0.86	0.94	0.99	0.92
SNN had the worst performance on all metrics. Despite Sid NN had performances comparable to EnSidNet on precision and recall, ROC AUC and PR AUC showed the higher performance of the Enhanced model.
Figure 1 shows the probabilities associated to couples belonging or not to the same drug-development pathway for EnSidNet. The figure shows that the algorithm can distinguish with great certainty whether the trials belong to the same pathway or not, and the higher recall than precision.
261 3.4 Trials grouping in pathways
262 Algorithm 1 for grouping the trials in possible pathways was applied to clinical trials present in the DrugBank database. The clinical trials included were those in phases 1, 2 and 3, with industry lead sponsors and 'treatment' as the
267 purpose of the trial. Trials to match into drugdevelopment pathways were 34188.The algorithm took less than 4 hours to run.
Figure 1: Predictions probability distribution. Blue bars represent couple of trials from different pathways, orange trials from the same pathway
The therapeutic areas included in these pathways were 27.
As presented in Table 4 the statistics of the possible pathways obtained from Algorithm 1 is overlapping with the statistics of the datasets used to train the neural networks (Supplementary Table 3).
Table 3: Metrics of the neural models.  $\mathrm{P} =$  Precision,  $\mathrm{R} =$  Recall, ROC AUC = area under Receiver Operating Curve, PR AUC = area under Precision-Recall curve
# pathways per therapeutic area	# trial per pathway
min	0	2
25 percentile	2.5	2
50 percentile	7	2
75 percentile	9.5	3
max	26	49
total	191	629 (583 unique)
Table 4: Statistics of the possible pathways obtained by running EnSidNet
Despite the input of Algorithm 1 was more than 34,000 trials, less than 600 were matched in pathways. However, the possible pathways obtained were about 1.5 times the number of total pathways in the dataset, suggesting new possible pathways were discovered running Algorithm 1, highlighting the potential of this semi-supervised approach for the grouping of clinical trials in pathways.
A subset of the predicted pathways was given to human labellers for scoring. The 73 predicted pathways (2-49 trials long), for a total of 264 trials, gave rise to 165 different trials (1-11 trials long). The different distribution of the predicted versus confirmed pathways can be seen in Supplementary Table 4. A total of 112 trials  $(42\%)$  were confirmed being assigned by the algorithm to proper pathways. Only two of the trials selected for human scoring were found also on the ground truth datasets. Specifically, both trials belonged to the FDA snapshot dataset and were single-trial pathways. Interestingly, one of these trials was assigned to 2 other trials, and this 3-trial pathway was then confirmed by the human experts scoring. This is a good example of the capability of EnSidNet and the proposed algorithm to find the contributing trials to a drug-development pathway.
305 4 Conclusion
We present a new approach for the grouping of clinical trials into drug-development pathways. To meet this objective, we proposed 3 different neural network architectures. The best performing model was EnSidNet, an enhanced hybrid Siamese-Deep Neural Network.
EnSidNet was used to develop a semi-supervised learning pipeline using 1-shot evaluation and classification to group trials into existing or new pathways. Human scoring would lead to the increase of the training size with ad-hoc positive and negative samples.
318 Acknowledgements
319 We thank Gregory Lever and Joseph Heenan for useful feedback.
321 References
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, Roopak Shah. 1994. Signature verification using a "Siamese" time delay neural network. In Advances in Neural Information Processing Systems, 6:737-744.
327 Sumit Chopra, Raia Hadsell, Yann LeCun. 2005. Learning a similarity metric discriminatively, with application to face verification. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1:539-546.
332 Gregory Koch, Richard Zemel, Ruslan  
333 Salakhutdinov. 2015. Siamese Neural Networks  
334 for One-shot Image Recognition. In Proceedings of
the 32 nd International Conference on Machine Learning, 37.
337 Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, 338 Iasonas Kokkinos, Pascal Fua, Francesc Moreno-339 Noguer. 2015. Discriminative learning of deep 340 convolutional feature point descriptors. In 341 Proceedings of the IEEE International Conference 342 on Computer Vision:118-126.
343 Ejaz Ahmed, Michael Jones, Tim K. Marks. 2015. An improved deep learning architecture for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition:3908-3916.
348 Nam Vo, James Hays. 2016. Localizing and orienting street views using overhead imagery. In European Conference on Computer Vision:494-509.
351 Arulkumar Subramaniam, Moitreya Chatterjee,  
352 Anurag Mittal. 2016. Deep Neural Networks with  
353 Inexact Matching for Person ReIdentification. In  
354 Advances in Neural Information Processing  
355 Systems:pp. 2667-2675.
Jonas Mueller, Aditya Thyagarajan. 2016. Siamese Recurrent Architectures for Learning Sentence Similarity. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence.
360 Aryo P Gema, Suhendro Winton, Theodorus David, 361 Derwin Suhartono, Muhsin Shodiq, Wikaria 362 Gazali. 2017. It Takes Two To Tango: Modification 363 of Siamese Long Short Term Memory Network 364 with Attention Mechanism in Recognizing 365 Argumentative Relations in Persuasive Essay. In 366 Procedia Computer Science, 116:449-459.
367 Peter B Corr, David A Williams. 2009. The Pathway from Idea to Regulatory Approval: Examples for Drug Development. In U.S. National Library of Medicine.
371 Chi H Wong, Kien W Siah, Andrew W Lo. 2019. Estimation of clinical trial success rates and related parameters. In Biostatistics, 20(2):273-286.
374 David S Wishart, Craig Knox, An C Guo, Savita Shrivastava, Murtaza Hassanali, Paul Stothard, Zhan Chang, Jennifer Woolsey. 2006. Drugbank: a comprehensive resource for in silico drug discovery and exploration. In Nucleic Acids Research, 34:D668-72. 16381955.
Lucia Pagani  
Analytics Center of Excellence, IQVIA Via Fabio Filzi 29, Milan, Italy lucia.pagani@iqvia.com
380 A. Supplementary Methods.
381 Training and evaluation of the models was run on a 4 CPU/32 GiB RAM machine, while Algorithm 383 1 was run on an 8 CPU/64 GiB RAM machine.
384 Pathway dataset
Supplementary Table 3 shows the statistic of the pathways in the two ground truth datasets: FDA Snapshot and hand curated.
388 Dataset splitting
389 A 4 folds split was performed in this research:
- Training and validation set: split in  $80\%$  for training and  $20\%$  for validation, it was composed of balanced couples of trials belonging and not belonging to the same pathway
- 32-way 1-shot evaluation validation set: this dataset was composed by 1 couple of trials belonging to the same pathway and 31 randomly coupled trials
- 32-way 1-shot evaluation test set: similar to the previous dataset, this dataset contained only 1 couple belonging to the same pathway over 32 randomly chosen couples of trials
Supplementary Table 5 shows an example of two couples of trials, one belonging to the same pathway and the other not.
The balanced datasets had trials from 124 unique pathways for a total of 2720 couples, while the 32-way 1-shot evaluation validation and test sets consisted of trials from 35 unique pathways
each, resulting in 1056 couples for both datasets. Pathways consisting of only 1 trial were used to build couples not belonging to the same pathways. Negative labelled couples were formed also from other trials from different pathways. A scheme of the datasets' composition and origin can be found in Supplementary Table 6.
418 Trials data
The trial information used for this experiment came from DrugBank. DrugBank contains information parsed from ClinicalTrials.gov. A step of hand curation is performed on the data before entering them to database.
The DrugBank database contains over 142 k trials, out of which only 3277 trials started before 2000. It also contains the information of completed / ongoing trials, and the purpose of the trial.
429 Model inputs and preprocessing
430 The inputs of the model were indication, condition, sponsor, phase, country, start date and end date of the trial.
433 Character-based inputs: character-based  
434 inputs considered were indication, condition,  
435 country, sponsor. Indication and condition were in  
436 the form of lists. The list of text was joined to form  
437 the text input. Data augmentation was performed  
438 in this case in the form of shuffling the order of  
439 the elements of the list.
The preprocessing of the character-based inputs consisted in the removal of stop words. Each input was tokenized at word-level, padded at 1.2 times the maximum length of the training set. For the 1-shot evaluation baselines the input was also 1-hot encoded.
Numerical inputs: the numerical inputs considered in the network were phase, starting date and end date of the trials. These were calculated or inputted and standard scaled.
Additional Inputs: additional inputs were used for the network. These were features preprocessed and concatenated to the absolute difference vector. The inputs were:
- Difference of phases between the two trials
- Days difference between start date of trial 1 and end date of trial 2
- Days difference between start date of trial 2 and end date of trial 1
- Difference between sponsor numbers between trial 1 and trial 2
Number of common sponsors between the trials
- Difference between the number of countries involved in trial 1 and trial 2
Number of common countries
These inputs, after they were calculated, were standard scaled on the training dataset.
468 Neural Network models
469 The Neural Network models consisted of different 470 branches, depending on the input type (see Supplementary Figure 1 for a scheme of the 472 architectures). These branches contained a single 473 module that encoded trial 1 and trial 2 independently.
475 Character-based module: Input went through 3 layers of bidirectional (Bi) Long-Short Term Memory (LSTM) (dimension 128, 64, 32 vector size). At the end of the 3 Bi-LSTM layers there was an attention layer, and a fully connected layer (64 nodes).
Numerical branch: Inputs went through a single fully connected layer (64 nodes) and dropout.
After the Siamese modules there was a concatenation layer, which concatenated all embedded inputs from trial 1 and all embedded inputs from trial 2. These concatenation vectors were passed through a layer that provided the absolute difference between the embedded trial 1 and trial 2 vectors.
Additional inputs module: Inputs went through a fully connected (32 nodes) layer and dropout. The output vector was concatenated to the absolute difference vector of trial 1 and 2.
Pre-prediction module: an additional fully connected (64 nodes) and dropout layer that preceded the sigmoid activated prediction layer.
498 Three models, schematized in Supplementary Figure 1, were used in this experiment:
- A pure Siamese Neural Network model (SNN), consisting of all character-based inputs modules (indication, condition, sponsor, countries) and numerical inputs (phase, start date, end date). No pre-prediction module was added to this architecture
- A hybrid Siamese-Deep Neural Network (SiD NN) which had character-based inputs (indication and condition) and additional inputs (phase difference, difference between start date and end date of the trials, difference between number of sponsors, number of common sponsors, difference between number of countries, number of common countries)
- An Enhanced hybrid Siamese-Deep Neural Network (EnSidNet) with an architecture similar to SiD NN but containing the pre-prediction module
520 1-shot evaluation baseline models
As baseline models for 1-shot evaluation we used:
1-Nearest Neighbor: calculated as the Euclidean distance between the inputs of the trials. The distance between all inputs was calculated by performing the absolute difference of trial 1 and trial 2, and then summed together.