dataesr/scientific-paragraphs-categorization huggingface dataset:
It uses sophisticated mathematical modeling to process data. The key to these sophisticated neural networks is dealing with unlabeled or unstructured data. This approach is particularly suitable for complex tasks, when all aspects of the data to be treated cannot be categorized upstream. The deep learning system identifies the discriminating characteristics of the data without the need for prior categorization. In each layer, it searches for new specific criteria of the object, which is used as a basis to determine the classification retained at the end of the process. The system does not need to be trained by a developer; it assesses the need to modify the classification or create new categories based on the new data. While machine learning works from a controllable database, deep learning needs a much larger quantity of data.
Methodology and model selection NGS analysis. Our dataset is composed of a cohort of 102,011 SNVs, SNPs and indel (insertion deletion) variants from our 27 gene panel sequenced for colorectal cancer, melanoma, lung cancer and 18 genes sequenced for glioma cancer. It represents a total of 7301 samples. Each SNV, SNP and indel were manually inspected and labeled as either pathogenic (51,008 variants) or benign (51,003 variants) as part of the clinical laboratory workflow. Our panel Oncomine Solid Tumor and Oncomine Solid Tumor + (OST/OST+) contains hotspots of target regions of 27 genes of interest in oncogenesis: KRAS, EGFR, BRAF, PIK3CA, AKT1, PTEN, NRAS, HRAS, STK11, MAP2K1, ALK, DDR2, CTNNB1, MET, TP53, SMAD4, FBXW7, FGFR1, FGFR2, FGFR3, NOTCH1, ERBB4, KIT, PDGFRa, POLE, RET, ROS. Our in-house glioma panel was made to define glioma grades and to target the following genes by sequencing full sequences and hotspots of AKT1, ATRX, BRAF, CDKN2A, CIC, EGFR, FGFR1, H3F3A, HIST1H3B, IDH1, IDH2, NOTCH1, PIK3CA, PIK3R1, PTEN, PTPN11, TERT and TP53.
For all clinical samples, after tumoral genomic DNA extraction by the Maxwell RSC DNA FFPE kit or Maxwell RSC Cell DNA (Promega, France), we performed sequencing with Ion Torrent S5XL (Thermo Fisher, France) with a sensitivity of 5% and minimum coverage of 500X. Then, sequencing data were analyzed through 2 pipelines. The first pipeline was developed by ThermoFisher on the IonTorrent Suite + Ion Reporter. IonTorrent Suite generates FASTQ data and ensures BAM (binary alignment mapping) alignment with the hg19 reference genome by using the TMAP (Torrent Mapping Alignment Program). Ion Reporter makes variant caller and variant annotations. The second pipeline was developed in our laboratory and runs open source software such as BWA-MEM for alignment, SAMtools for mpileup, VarScan2 as variant caller and VEP Ensemble for annotations.
All data are stored in our local MySQL database. Biologists manually validated variants if the following statements were true: • Variant allele frequency ≥ 5% • Minor allele frequency is absent • Variant reads ≥ 500 • Amino acid change is different from silent ( = p. (=) ) or ( = p. ? for a noncoding region) in cases where the variant's location is in the exonic region. This statement is not applied, for example, to MET mutations in lung cancer which are in the splice site region and coded as p. ?. This is also the case for the glioma panel where the TERT mutation hotspot is in the promoter region. • Variants are present in several oncosomatic databases, such as COSMIC, VarSome, and TP53 IARC, and are known to have pathogenic or uncertain significance. • Alignment sequences are clear and show no strand bias in the region where the variant is located.
Construction of the training dataset and feature selection. Before starting dataset construction, we need to calculate entropy and information gain. Calculating information and entropy is a useful tool in machine learning and is used as the basis for techniques such as feature selection, building decision trees, and, more generally, fitting classification models. Entropy is defined by this equation: where p i is the probability of value i and n is the number of possible values. Entropy is the measurement of the impurity or randomness in the data points, and it is calculated between 0 and 1  16. Entropy is the lowest (no disorder = 0) at extremes (both end = 1) and maximum (high disorder =  1  2 ) in the middle. The concept of entropy plays an important role in calculating information gain (IG)  17.
It is applied to quantify which feature provides maximal information about the classification based on the notion of entropy  17  , so it answers this question by measuring how much "information" a feature provides us about the class  18. The information gain is the product of class probabilities with logarithm base 2 of that class probability. where H p is the entropy of the parent node, p ci is the probability of an observation being in child i, and H ci is the entropy of child segment i. Construction of the training dataset. Data were extracted from the local MySQL database, and we cleaned and unified the data to make them reliable for any analytic or machine learning process. Some of the recorded data did not follow standard schema or break integrity constraints. Much of the work involved cleaning datasets, such as removing errors and anomalies or replacing observed values with true values, for integration in the training phase.
Biologists review variants at each NGS run and determine whether a variant can be validated. Therefore, each variant is flagged YES or NO (in our dataset: validationVariant = y = {YES = 1; NO = 0} ). Yes denotes the variant is considered pathogenic for the pathology and NO denotes benign. To perform our analysis, we worked on a Linux Centos 7 workstation with 16 GB Memory and an Intel Xeon E5-2603 v3 1. 60 GHz CPU (12 cores). We defined the following data structure for machine learning with 7 parameters: 1. Chromosome (Chr) ⇒ Chromosome number (integer) 2. Position (POS) ⇒ Chromosome position (big integer) 3. Exon ⇒ Integer. Exon is the exon number in each gene (position of exon in a transcript, starting with one for the first exon). Zero is for a not coding region (intronic). 4. Variant allele frequency (VAF) ⇒ Float 5.
If variable y was equal to zero, the variant was benign, and if the variable y was equal to one, the variant was considered pathogenic. The entropy of our dataset H = 1 , as expected, was different from zero. The entropy for the YES and NO class is zero. This indicates that we acquired a complete knowledge of the contents of this set. If we were asked to draw one observation at random and predict the variant, we knew we would draw a pathogenic or a benign variant. The information gain IG for features contained in our dataset was as follows: chromosome (Chr) 0. 06, position (POS) 0. 06, exon 0. 01, variant allele frequency (VAF) 0. 25, minor allele frequency (MAF) 0. 36, coverage 0. 01 and amino acid change (protdesc) 0. 45. We captured some information from each feature (variable), and amino acid change (protdesc) provided the greatest information gain.
To create and estimate the accuracy of some models of the unseen data, we began by setting-up the test harness to use 10-fold cross-validation  3. Then we built 5 different models to predict a potential pathogenic variant (y) from variant features to finally select the best model. This split our dataset into 10 parts, trained in 9 and tested on 1 and released for all combinations of train-test splits  19. We repeated the process 3 times (repeated k-fold) for each algorithm with different splits of the data in 10 groups, to obtain a more accurate estimate  19. This method is one solution to reduce the noise in an estimated model performance by increasing the k-value. Repeated k-fold cross-validation has the benefit of reducing the bias in the estimated model performance. In our case, k = 10 and repeated = 3, so 30 different models were adjusted and evaluated.
We used the accuracy metric to evaluate the models. This is a ratio of the number of correctly predicted instances divided by the total number of instances in the dataset multiplied by 100 to obtain a percentage  19. We tested 5 ML algorithms mixing linear, nonlinear and complex nonlinear methods (linear discriminant analysis (LDA) ⇒ linear, classification and regression trees (CART) ⇒ nonlinear, k-nearest neigbords (kNN) ⇒ nonlinear, support vector machines (SVM) ⇒ complex nonlinear, random forest (RF) ⇒ complex nonlinear). At each algorithm run, we reset the random number seed to ensure that the evaluation of each algorithm was performed with the same data splits.
Results Model selection with 30-fold cross-validation was performed with 70% of the dataset (train set), and 30% of the remaining data were used for validation. The data were randomly split into two mutually exclusive subsets: training and validation sets. The results of 30-fold cross-validation on the 5 models are as reported in (Table  1  and Fig. 1 ). According to the results presented in Fig. 1  and Table  1 , we can see the accuracy of each classifier and metric such as kappa (see Table  3  for kappa details). The plot of the model evaluation results allows us to compare the spread and the mean accuracy of each model. The accuracy measures for each algorithm are populated because each model was evaluated 30 times (10-fold cross-validation with repeated k-fold cross-validation). Based on the obtained data, it is clear that the random forest is the most accurate model with our NGS dataset.
The OOB error is a measure of the random Table  1. Accuracy of ML algorithms after 30-fold cross-validation. The minimum (min), 1st quartile (1st Qu. ), median, mean, 3rd quartile (3rd Qu. ), maximum (max) are calculated for each machine learning algorithm tested on the oncosomatic variant dataset. 1st Qu. is the median (the middle) of the lower half of the data, approximately 25% of the numbers in the dataset lie below 1st Qu and 75% lie above the 1st Qu. The 3rd Qu. is the median (the middle) of the upper half of the data, about 75% of the numbers in the dataset lie below 3rd Qu. and approximately 25% lie above the 3rd Qu. forest prediction error and evaluates its performance. We analyzed the OBB error with default parameters, and then we tuned the model with input parameters such as ntree (number of trees), and mtry (number of variables considered for each tree node) to fit the model accuracy.
Tuning hyperparameters (ntree, mtry) to achieve two goals: increase the predictive power of the model and make it faster. Mtry refers to the number of variables available for splitting at each tree node. For the classification model, the default is the square root of predictor variables (rounded down); in our case, mtry = 3 was the default. The TuneRF package found the optimal mtry to obtain the smallest OOB. When satisfied with the results classification of the training dataset, we applied the algorithm on new unseen data to test RF prediction. Our results were presented on a confusion matrix (also known as an error matrix): a table presenting the observed data predicted by the RF algorithm demonstrates the performance of the RF algorithm to predict the variable y. Each row of the matrix represents the instances in a predicted class, while each column represents the instances in an actual class.
Our strategy to tune our model was to choose the number of trees keeping the default value of mtry and to test several values by evaluating them. Plotting the model helped us choose the best ntree: before 500 trees, all OOB values oscillated excessively, and after 500, they stabilized. Using more than 500 trees, was more machine time consuming, which is why we fixed the value of ntree = 500 (Supplementary Fig. S1 ). The data demonstrated that the best OOB was clearly improved with mtry = 4 (Supplementary Fig. S2 ). To assess the performance of the model, accuracy, recall, and precision were calculated to measure the performance of the classification model. Accuracy is one metric for evaluating classification models. Informally, it is the fraction of predictions that helped us to determine the correct model.
Accuracy is defined as the following equation: The model achieved 99. 77%. This means that our oncosomatic classifier performs well in identifying malignancies of mutation variants. Accuracy alone is not enough when we are working with a class-balanced dataset, such as this one, where there is not a significant disparity between the number of positive and negative labels. We need to evaluate other metrics, such as precision and recall. Precision is defined as follows: Predicted Precision attempts to determine the proportion of positive identifications that were actually correct. In our dataset,  35, 699  35,699+158 = 0. 99. Our model achieved a precision of 0. 99, in other words, when it predicted that a vari- ant was pathogenic, it was correct 99% of the time. Recall (also known as sensitivity  14  ) attempts to determine the proportion of actual positives identified correctly.
3 ), we can see that this is the best possible ROC curve, as it ranks all positives above all negatives. It has an AUC of 0. 99 for classes YES and NO labels. We need now to evaluate the optimal model classifier on the independent validation dataset, which represents the 30% remaining. It was composed of 30,603 SNV variants (15,215 were SNPs and 15,389 were pathogenic oncosomatic variants). Applying the model, gives us: Of 15,215 TN SNP, 100% were predicted real. A total of 15,309 out of 15,389 (99. 5%) were labeled pathogenic. More importantly, none of the TN SNPs were misclassified as pathogenic, while only 0. 52% were labeled benign  and required further manual investigation to correct them. These 80 are misclassified and 15,309 are correctly classified as pathogenic. This would not have an impact on clinical outcome, as the biologist manually inspected all variants, and the proportions of 0.
5% error were very few in comparison with the efficiency of the classifier in predicting the correct class. We created a model that can classify somatic variants with high precision, recall (sensitivity) = 1, specificity = 0. 99 and accuracy = 0. 997. A high value of accuracy observed on the validation dataset can overfit, but this was not the case. Overfitting is when the model fits well to the training dataset but fails to fit the validation dataset. We observed a good prediction score and no large gap between the training dataset and validation dataset predictions. Moreover, to avoid overfitting, we optimized a tuning parameter that governed the number of features that were randomly chosen to grow each tree from the bootstrapped data. This was done via k-fold cross-validation by choosing tuning parameters (mtry, ntree) that minimized test sample prediction error.
Random forest routine implementation and impact on our workflow. From March 2020 until June 2021, we used RF in the NGS analysis routine at each variant imported into our MySQL database. The architecture of our information system was developed in Hypertext Preprocessor (PHP), so it used R script to call machine learning at each run (Fig. 4 ). The NGS database of the first 2 months of the year represented more than 2000 variants. The RF model misclassified 20 variants (which represents an error rate of < 1%). The errors were nomenclature problems or false positives. We found 2 errors about the same MET mutation in the splice site. Testing neural network on our dataset. We tried several machine learning algorithms, but it may be interesting to determine whether neural networks can achieve a good recognition pattern on our dataset.
Discussion Our aim was to use AI to interpret NGS oncosomatic analysis relying on 7 parameters: chromosome, position, exon, variant allele frequency, minor allele frequency, coverage, and protein description. We developed an accurate and efficient method based on random forest machine learning that classifies a set of variants with high precision from NGS data. Our implemented model obtains decisions and probability for each variant call and allows a biologist to accept or reject variants (Supplementary Table  S1 ). The model was developed in the R language (R version 3. 6. 0, RandomForest version 4. 6-14, NeuralNet version 1. 44. 2), which is machine learning open source modules; hence, more users can tune this model on their data by using our training database. ML and, more particularly, the RF model has already been used in several publications as a tool for risk evaluation, quality control or predictions based on sequencing data  2, 4, 9.
Furthermore, the applications we found using RF rely on the analysis of VCF (variant caller) files such as Smurf, which predicts a consensus set of somatic mutation calls, or Octopus, which uses BAM files as a training dataset to classify variant calls  9, 10. Our tool classifies oncosomatic variants when already annotated (see Supplementary Data Fig. S7 ). It is effortless to implement in an information system and user-friendly for "noncomputer" science persons. Moreover, our tool is not intended to replace biologists but is rather an aid for decisions facing the multitude of somatic variants present at each NGS run analysis. To our knowledge, it has never been published for routinely interpreting NGS analysis. Our RF model shows excellent results with an error rate of < 0. 5% since we routinely use it.
Errors are nomenclature errors and false positives absent from our training database. Nomenclature errors are difficult to correct because the ion reporter annotates variants automatically, so our error rate can never be zero, and we have to stay vigilant to spot them. We can manually change them if there is a nucleotide error, for example, after our RF model is correct. Nevertheless, we cannot eliminate read errors from sequencing, and we easily improved false positive errors. At every new false positive mutation, we increment our training database, so that our RF model can be improved at each NGS run. ML is the oldest and simplest technology. It is based on an algorithm that adapts the system, based on feedback from humans. Implementing this technology implies the existence of organized data.
The system is then fed with structured and categorized data, allowing it to understand how to classify new similar data. Based on this classification, the system then performs programmed actions. After the first phase of use, the algorithm is optimized based on a tuned grid search of two parameters (number of trees: ntree and number of variables randomly sampled at each split: mtry), which informs the system of incorrect classifications and indicates the correct categories. ML works from a controllable database, which means that at each moment, we can modify the training database to correct prediction errors. The system must have many more entries to achieve reliable results. We also have an error rate of < 0. 5% for our glioma panel (AmpliSeq glioma custom panel: AKT1, ATRX, BRAF, CDKN2A, CIC, EGFR, FGFR1, H3F3A, HIST3B, IDH1, IDH2, NOTCH1, PIK3CA, PIK3R1, PTEN, PTPN11, TERT, TP53) NGS analysis.
It seems to be a good reproducible model for this type of interpretation  2, 5, 6, 11. Moreover, the random forest algorithm is a machine learning algorithm that runs efficiently on large datasets with high accuracy  15. Random forest was successfully applied in other laboratories to predict and classify genomic data analysis, and it is an appropriate tool for predicting clinical outcomes under high-throughput genomic platforms  23. Random forest in comparison with k-nearest neighbors (KNN), linear discriminant analysis (LDA), classification and regression trees (CART), support vector machine (SVM) performed better than the others algorithms in terms of the error in prediction. Lee et al. 24  showed that RF has the best performance compared to all tree-based methods. A machine learning method based on the random forest for predicting the classification of variants was also successfully applied in medical case diagnosis of breast cancer  25.
They built a computer "help" diagnostic system to determine if a breast tumor is benign from malignant tumors. They obtained 99. 82% classification accuracy in the test phase, and their results can be applied to other breast cancer problems. Their proposed method was based on N-fold cross-validation with a phase one where RF was trained and tested on the training set and validation set from the University of California, Irvine Breast Cancer Wisconsin (Diagnostic) dataset. Lai et al. 26  developed an interesting machine learning method based on the random forest to predict the classification of variants in genes associated with high risk for hereditary cancer. A training base of 14,226 missense variants in genes associated with high risk for hereditary cancer and 5398 missense variants with elevated risk for cardiovascular disorders were used to train the model.
Considering using a different ML algorithm to ANNs-decision trees based on approaches such as random forest can be faster and more effective than deep learning on many problems. However, they also have the capacity to learn subtle differences from large quantities of data where simpler algorithms may reach a limit. Neural networks have provided good accuracy in the prediction of variant mutations. We compared the performances of the neural network and random forest on unseen data, and RF was slightly over in accuracy most of the time in comparison with the neural network. However, ANN shows the ability to adapt to new data where RF failed. For example, the two MET mutations in splice sites missed by RF were well predicted by the neural networks as pathogenic (Supplementary Table  S1 ). After modifying our database, our RF model achieves a correct classification.
Soft-agar colony formation assays A first agar layer was placed in 10 cm plates at 0. 8% (w/v) of low melting temperature agarose (50101; Life) in αMEM-10%FBS. Once solidified, a second layer of 0. 48% agar containing 4 x 10 4 cells was added. The plates were maintained at 4 °C for 5 min and 10mL of fresh culture medium was subsequently deposited as a top layer. The plates were incubated in hypoxia conditions and colonies were isolated after 3 to 4 weeks and further analyzed. Counting of colonies was performed using ImageJ/Fiji software and T-test analysis was applied. Cell proliferation and siRNA assay hMPCs were seeded in 6-well plates (40. 000 or 60. 000 cells/well) and maintained in hypoxic like or standard conditions for proliferation assay. Cell growth was monitored and analyzed by the IncuCyte Live Cell Analysis system (IncuCyte S3, Essen Bioscience) every 24 hours for 4 to 10 days.
Flow cytometry Immunostaining of cells for CD99 marker was performed by incubating cells with FITC Mouse Anti-Human CD99 (BD Pharmingen, 555688) or FITC Mouse IgG2a, κ Isotype Ctrl Antibody (Biolegend, 400208) for 30 minutes at 4 °C prior to flow cytometry analysis based on SS-A/FS-H gating on alive cells (BD FACS Aria II-BD bioscience, and FlowJo software). Cells labelled with unspecific FITC Mouse IgG2a were used as negative cells for CD99 expression.
qRT-PCR RNA was extracted with the RNeasy Plus Mini Kit (Qiagen) and reverse-transcribed using the High-Capacity cDNA Reverse Transcription Kit (Applied Biosystems). qRT-PCRs were performed using PowerSYBR green Mastermix (Applied Biosystems). Oligonucleotides were purchased from MWG Eurofins Genomics (Oligonucleotides and Primers, see Supplementary Materials). Reactions were run on CFX384 Touch Real-Time PCR instrument (Bio-Rad) and analyzed using the CFX Manager Software (Bio-Rad). Relative expression level was assessed with the ddCT method using RPLP0 as a housekeeping gene. Primer sequences are reported in Supplementary Material.