dataesr/scientific-paragraphs-categorization huggingface dataset:
Training, testing, and performance assessment We examined different machine-learning (ML) classification methods to sort deleterious variants, searching for the best-performing one, while all were designed to properly tackle the class imbalance in Dataset 1. Each method was instantiated during a 20-fold cross-validation (CV) repeated five times, where 19 folds of the dataset were iteratively used for the training and tuning of the hyperparameters, and the remaining fold was used for testing. Each method was tuned by attempting several combinations of hyperparameters (Supplementary Table  3 ). This was done using an inner 10-fold Grid-Search (GS) CV. With the highest average test auPRC (0. 716, Figure  2A ), auROC (0. 95), and the best records for nearly all performance metrics (Supplementary Table  4 ), the KNN Bagging balanced through Random Under-Sampling (RUS) and Synthetic Minority Oversampling Technique (SMOTE) method (KNN_RusSmote) stood out as the best and was then chosen as the APOGEE 2 ML reference method.
The best hyperparameters identified by GS were: "3" for the number of KNN's neighbors; the inverse euclidean distance as a metric to weight the neighbors' importance; 1/4 as the pathogenic/neutral variant ratio before the SMOTE step; and 1/4 as the ratio of features used by each base learner over the total number of selected features. It used 17 out of the 22 considered features, where PhastCons 100V, MutationTaster, FatHmmW, CADD, and ΔΔG were, in fact, discarded in the feature selection step (Figure  2B ). On Dataset 1, APOGEE version 1 underwent a second round of testing and received an average auPRC of 0. 573 (Figure  2A ) and auROC of 0. 855. The classification performance of both APOGEE versions was also compared on the test set of our previous work  15  , which included the union of MITOMAP and VariBench variants available at the time of that writing (see Supplementary Table  Compared with other meta-predictors, APOGEE 2 exhibited the best performance metrics, including auPRC (Figure  2D ), but the sensitivity where MtoolBox and Condel excel at the expense of specificity, which ranked worst for both tools (Supplementary Table  4 ).
To evaluate whether expanding APOGEE 2's training set would improve its performance in future releases, we adopted the same criteria used to create Dataset 1 to generate seven training sets starting from the MITOMAP content from 2008 to 2020, which was randomly retrieved once every two years. The 2022 content was used as a test set. We found that the models' performance in sorting pathogenic from neutral variants in the 2022 dataset increased monotonically over time (Pearson's r: 0. 91, p-value: 0. 004), but at an increasingly slower rate (Figure  2E , Supplementary Table  10 ), indicating that expanding the training set in subsequent iterations will not have as much of an impact on APOGEE 2's performance as adding more informative features would.
Categorization. Interpreting the evidence categories given in Tables  3  and 4  of 11 as categorical conditional probabilities or odds of pathogenicity, in line with  17  , and aiming to provide clinical translationality to APOGEE 2 predictions, we adopted the following ranges of probability and set four classes of pathogenicity: benign ≤ 0. 001, 0. 001 < likely benign ≤ 0. 1, 0. 9 ≤ likely pathogenic < 0. 99, pathogenic ≥ 0. 99. When none of the previous criteria are met, i. e. , when the posterior pathogenicity probability of a variant is between 0. 1 and 0. 9, a variant has uncertain significance (VUS) (Supplementary Table  5 ). With this setting, all known pathogenic variants in Dataset 1 were correctly labeled pathogenic and likely-pathogenic, except for seven of them, which were annotated as VUS.
Regarding the harmless variants in Dataset 1, 190 were annotated as VUS, while the remaining 1544 were correctly labeled benign and likelybenign (Supplementary Table  5 ). Nearly all variants in Datasets 2 and 3, which contain only potentially harmless variants and therefore were used to evaluate APOGEE 2's specificity, were classified as benign or likely-benign by APOGEE 2 (Supplementary Table  7 : 𝜒𝜒 2 (1, N=36) = 16. 19, p =5. 73E-05 and 𝜒𝜒 2 (1, N=35) = 24. 10, p =9. 15E-07, respectively for Datasets 2 and 3). None of the variants in Dataset 2 were classified as pathogenic or likely-pathogenic, while one variant from Dataset 3 was classified as likely-pathogenic.
Variant functionality prediction in human variation databases As a further test of the negative variants, we used APOGEE 2 to score every mitochondrial missense variant in the gnomAD v3. 1. 2 and HelixMTdb databases. Heteroplasmic variants were 1767 (AF range min 0. 0017%: max 0. 037%) in gnomAD and 3058 (AF range min 0. 0005%: max 0. 15%) in HelixMTdb. The homoplasmic variants were 2177 (AF range min 0. 0017%: max 99%) and 2894 (AF range min 0. 0005%: max 99%). The predicted pathogenic and likely pathogenic variants for both databases were comparable (Supplementary Figure  2A, B ), even when considering <0. 002% variants, irrespective of their heteroplasmy levels. GnomAD and HelixMTdb showed, in fact, a high and significant positive correlation between the allelic frequencies of their heteroplasmic (Pearson's r: 0.
88, p-value <2. 2E-16) and homoplasmic (Pearson's r: 0. 91, p-value =6. 434E-11) variants. It is important to note that HelixMTdb was not enriched in patients with mitochondrial diseases, but there were no exclusion criteria based on mitochondrial disorders  9. On the other hand, while gnomAD declares that some individuals with severe disease may still be included in the database, albeit likely at a frequency equivalent to or lower than that seen in the general population, they have explicitly removed individuals known to be affected by severe pediatric disease, as well as their first-degree relatives. This might explain the presence of both likely and frankly pathogenic variants in both datasets. The pathogenicity probability values for ClinGen's neutral variants, which were collected in Dataset 4 (Supplementary Table  7 ), ranged from 0 to 0.
73. Dataset 4 (see Methods) contains neutral variants that were evaluated by the ClinGen mitochondrial VCEP team and, therefore, are supposedly more likely to be neutral than those contained in Datasets 2 and 3. None of Dataset 4 variants was predicted as pathogenic or likely-pathogenic by APOGEE 2. We also verified that the APOGEE 2 scores of these variants were lower on average than those of the neutral variants in Dataset 1 (Mann-Whitney U test, U=152940, p-value=2. 80E-09; Supplementary Figure  2C ), thereby confirming the ability of APOGEE 2 to also quantify the level of certainty of being a neutral variant.
A few pathogenic mutations were found in the helix. One of them, m. 14459G>A, hits the residue Ala72, reported in MITOMAP and ClinGen as a pathogenic change in valine and associated with multiple mitochondrial phenotypes. It is interesting to note that the same residue is also hit by a proline variant (Ala72Pro, m. 14460C>G). The proline variant is included in Dataset 1 because it is deemed benign due to its allelic frequency of 0. 011%, but it is categorized as VUS by APOGEE 2 (score = 0. 64, probability = 0. 82). We have compared the long-range impact of p. Phe46Leu on TMH3 with that caused by Ala72Pro and Ala72Val by simulation. After 200 ns of simulation, we observed that all three mutants altered the helix's folding to different extents, with a more evident destructuration caused by Ala72Pro and Ala72Val than Phe46Leu (Figure  4E ).
Given that some variants with literature reports of suspected pathogenicity show frequency levels higher than 0. 002% in gnomAD and HelixMTdb, it is also likely that some variants that make up the neutral subset of Dataset 1 are actually pathogenic. The reverse is not obvious. We have then tested APOGEE 2 and EFIN HD on ClinGen's recently released neutral variants; in general, APOGEE 2 assigned lower pathogenicity scores to ClinGen's neutral variants than EFIN HD did (Wilcoxon signed-rank test preceded by quantile transformation of both scoring systems; the EFIN score has been inverted since it is proportional to the neutrality of the variant; ranks sum = 3236, p-value = 0. 003). We have demonstrated that the APOGEE 2 pathogenic scores exhibit significant spatial autocorrelation, suggesting that some protein structural regions may be less tolerant to amino acid changes than others.
Datasets Four disjoint sets of variants were used in this work. Dataset 1 comprises 1874 nonsynonymous mtDNA variants, divided into 1734 deemed benign and 140 pathogenic variants (Supplementary Table  5 ). The former set was obtained from MITOMAP's "general" variants (accessed on April 20, 2021), which were purged of overlapping MITOMAP's "disease" variants and variants with ClinGen's pathogenic criteria code "PM2. " The pathogenic set was made up of MITOMAP "disease" variants, including 41 confirmed and 99 reported variants. Overlapping Benign variants in the set had allele frequencies >1% ("BA1") in either MITOMAP, gnomAD, or HelixMTdb and without disease reports or negative in-silico predictors; likely benign variants in the set had allele frequencies of 0. 5%-1% ("BS1") in MITOMAP, gnomAD, or HelixMTdb and without disease reports or negative in-silico predictors, and additionally had either a supporting in-silico tool score for benignity or a resulting synonymous amino acid change.
All other variants meeting the allele frequency cutoffs but with possible disease associations are referred to the ClinGen mitochondrial VCEP for individual curation. The current set of benign/likely-benign variants meeting the ClinGen criteria is available at https://mitomap. org/MITOMAP/Benign and was accessed for this study on September 7, 2022, and released while Dataset 1 was already made up. We gathered 135 neutral missense variants in total, dropped 8 VUS variants, and found that 97 of them overlapped Dataset 1. Datasets 2, 3, and 4 were used to evaluate the specificity of the APOGEE 2 predictions (Supplementary Table  7 ). Features APOGEE 2 was trained on three classes of information: evolution, pathogenicity predictions, and protein structural features. Evolution: PhyloP and PhastCons conservation scores on 100 vertebrate species were obtained from the UCSC Table Browser.
Empirical substitution scores were obtained from the MtMam rate matrix, built on 12 proteins, i. e. , ATP6, ATP8, COX1, COX2, COX3, CYBB, ND1, ND2, ND3, ND4, ND4L, and ND5, which are located on the heavy strand of the mtDNA (3331 sites)  25. Data are from 20 species of mammals and three close outgroups, i. e. , Wallaroo, Opossum, and Platypus. The rate matrix was downloaded from https://github. com/abacusgene/paml/blob/master/dat/mtmam. dat. Pathogenicity predictors: Pathogenicity scores were retrieved for the following software packages: PolyPhen2 26 , SIFT  27  , fathmm  28  , PROVEAN  29  , MutationAssessor  30  , EFIN  31  , CADD 32 , PANTHER  33  , PhD-SNP  34  , SNAP  35  , and MutationTaster  36.
Structural features: The 3D structures of the 13 proteins were retrieved from the RCSB Protein Data Bank (PDB) with the following IDs: 5xtc for the Respiratory Complex I (s, i, j, r, k, l, and m chains for MT-ND1, MT-ND2, MT-ND3, MT-ND4, MT-ND4L, MT-ND5, and MT-ND6, respectively); 5z62 for the Cytochrome C Oxidase (chain A for MT-CO1, chain B for MT-CO2, and chain C for MT-CO3); 5xte for the Respiratory Complex III (chains J and V for Cytochrome b). Since MT-ATP6 and MT-ATP8 were not associated with any resolved X-ray structure, their predicted structures were downloaded from the AlphaFold2  37  Protein Structure Database  38. The five PDB files were placed on the bisector of a 3D space and spaced 3𝑑𝑑 apart, where 𝑑𝑑 = 20Å. The total energy variation (ΔΔG, Kcal/mol) eventually caused by any possible nonsynonymous amino acid change was calculated as follows.
Machine learning workflow The overall learning procedure consists of a double cross-validation procedure that nests an optimization procedure of the best parameter combinations (hyperparameters) of each implemented machine-learning (ML) algorithm under a model selection procedure. The workflow was implemented in Python and made available at https://github. com/mazzalab/playgrounds as a Colab notebook. In particular, for each ML algorithm, the whole dataset was primarily split into training and test sets by a stratified 20-fold cross-validation (CV) procedure, implemented in Python's scikitlearn's StratifiedKFold, and repeated five times. Then, for each unique group, we took it as a holdout and used the remaining groups (19) as a training data set. Each group was guaranteed to maintain the original pathogenic/neutral variant ratio.
Imputer selection The whole MitImpact content, made up of 22,316 variants regardless of their label (neutral or pathogenic) and excluding the variants in Dataset 1, was split into 20 folds. Each fold was iteratively chosen as test-and the remaining 19 as training-variants for each of the following five imputation methods. We implemented two simple interpolation methods based on the mean and median values of the features to be imputed, respectively; a k-Nearest Neighbors imputer (sklearn. impute. KNNImputer, k = 5); two iterative imputers (sklearn. impute. IterativeImputer) based respectively on Bayesian ridge linear regression (sklearn. linear_model. BayesianRidge) and Random Forest regression (sklearn. ensemble. RandomForestRegressor). Once fitted any of these imputers, we have generated twelve 𝑁𝑁𝑁𝑁𝑁𝑁 matrices of probability estimates for each test set by sampling from a uniform distribution in the range [0,1), where N and M are, respectively, the numbers of variants (rows) and features (columns), and multiplying its values by one of the twelve values in the range 0.
Grid-Search cross-validation We performed an exhaustive search over the parameters (or hyperparameters) of the tested ML methods using a stratified 10-fold Grid Search CV applied to a pre-computed and methodspecific parameter grid (GridSearchCV module). The parameter grids for all methods are available in Supplementary Table  3. The preprocessed training set was further split into a 9fold training set and a 1-fold test set. The Grid Search procedure was run for each CV iteration, for a total of 𝑔𝑔𝑔𝑔 = 10 × 𝑑𝑑 times, where 𝑑𝑑 is the number of hyperparameter combinations explored. An ML method was trained and then tested on the 1-fold test set for each combination (Figure  5D ). The area under the precision-recall curve (auPRC) and under the receiver operating characteristic (auROC) curve metrics were evaluated to assess performance.
Model training and testing Each classifier was trained on the 19-fold training set using the best hyperparameter combination found by the Grid-Search loop and then tested on the relative 1-fold test set, which was not involved in the training or tuning phases (Figure  5E ). The best classifier exhibited a superior average auROC over the five replicas of the overall 20-fold CV process. Then, the total number of iterations performed for each ML method was 100 × (𝑔𝑔𝑔𝑔 + 1). Finally, we trained a model of the best method on the entire Dataset 1, using the best hyperparameters found through the GridSearch CV procedure, and scored all possible nonsynonymous mtDNA variants.
Both parameters were optimized by Grid-Search. Gaussian Naive-Bayes (GNB) Balanced Bagging. This is a probabilistic classifier that uses a Gaussian Naive-Bayes (GNB) model as the base learner for a Bootstrap Aggregation (Bagging) classifier. This approach is typically used to improve the performance of a "weak-learner" (such as the GNB) and reduce overfitting. To tackle the problem of class imbalance in our training set, we resorted to the Balance Bagging classifier implemented by the imbalanced-learn package of scikit-learn. We have also bootstrapped the features (without replacement) for each baselearner training in an attempt to reduce overfitting. The Grid-Search-based optimization step was applied to i) the ratio of the bootstrapped samples over the total samples; ii) the ratio of the bootstrapped features over the total number of features; and iii) the GNB variance smoothing (more details in the scikit-learn documentation).
Misclassification and specificity assessment In order to calculate the per-class misclassification rate on Dataset 1, we first divided the APOGEE 2 scores into two classes (pathogenic and benign) based on the median value. Then, we calculated the frequency of incorrectly classified test variants, which are the variants that were disregarded from the model learning and tuning during CV. Based on the distribution of APOGEE 2 scores for all MitImpact variants, the median threshold was recalculated for each test. Using MitImpact's non-benign frequency as the expected frequency for the null hypothesis, we performed a Chi-square test on the alternative hypothesis of having a high frequency of benign variants in Datasets 2 and 3. Finally, we compared the average APOGEE 2 scores of the Dataset 4's benign variants with the average APOGEE 2 scores of the Dataset 1's benign variants, using a Mann-Whitney U test; the OOB scores were used for the Dataset 1's benign variants in this test.
The APOGEE 2's time-dependent learning curve From 2008 through 2020, we fetched the MITOMAP content every two years and generated seven datasets by applying the same criteria we used for Dataset 1 (Supplementary Table  5 ). We trained APOGEE 2 with these datasets and used the current MITOMAP version (July 2022) for testing the trained models. In particular, we applied a random stratified 5-fold partitioning strategy to the 2022 dataset, obtaining 5 partitions. For each partition, we trained a model on Positionality and impact of variants on the protein structure Spatial autocorrelation of the APOGEE 2 scores Moran's index was used to measure the spatial autocorrelation of the predicted APOGEE 2 scores. It is defined as 𝐼𝐼 = 𝑁𝑁 ∑ ∑ 𝑤𝑤 𝑖𝑖𝑖𝑖 𝑁𝑁 𝑖𝑖=1 𝑁𝑁 𝑖𝑖=1 ∑ ∑ 𝑤𝑤 𝑖𝑖𝑖𝑖 (𝑦𝑦 𝑖𝑖 -𝑦𝑦 �)(𝑦𝑦 𝑖𝑖 -𝑦𝑦 �) 𝑁𝑁 𝑖𝑖=1 𝑁𝑁 𝑖𝑖=1 ∑ (𝑦𝑦 𝑖𝑖 -𝑦𝑦 �) 2 𝑁𝑁 𝑖𝑖=1 , where 𝑁𝑁 is the number of spatial data points and 𝑤𝑤 𝑖𝑖𝑖𝑖 a weight matrix that measures the closeness of each residue in a pair.
Molecular Dynamics simulation Molecular Dynamics (MD) simulations were conducted on the human respiratory complex I transmembrane arm (PDB ID: 5xtc, described above). The starting system was refined using MODELLER ver. 9. 16  44  and then mutated in-silico using UCSF Chimera ver. 1. 16  45  to introduce S34P, S34Y, S34F, and T35P amino acid variants in MT-ND3 and A72V, A72P, and All generated systems were energy minimized and equilibrated following the CHARMM-GUI's workflow, consisting of i) steepest-descent minimization for 5000 steps; ii) a gradually heating process conducted into a canonical ensemble (NVT) for 250000 steps with a timestep of 1 fs; iii) an isothermal-isobaric ensemble equilibration for 125000 steps with a timestep of 1 fs followed by 750000 steps of an isothermal-isobaric ensemble equilibration every 2 fs.
Analysis of simulated trajectories AmberTools21 was used to calculate the Root-Mean-Square Fluctuation (RMSF), which measures the time deviation of the positions of the atomic coordinates of the alpha carbons of each residue and those of the reference starting structure. The GetContacts (https://getcontacts. github. io) tool was used to compute all the atomic interactions and contacts established in each time frame of the simulated trajectories. Secondary structures were computed using the DSSP module of the MDTraj 49 tool for each simulation frame. The DSSP assignments were calculated using the 8-category schemes: H=α helix, B=residue in isolated beta-bridge, E=extended strand, participating in beta ladder, G=3-helix (3/10 helix), I=5-helix (π helix), T=hydrogen bonded turn, S=bend, -=unclassified.
Acknowledgements This research was funded by the  Italian Ministry of Health  ( Ricerca Corrente   2020-2022 ) and the " 5x1000 " voluntary contributions and supported by  NVIDIA Corporation  and the Amber team. Additional support came from  National Institutes of Health  grants  NS021328 ,  MH108592 ,  OD010944 , and  CA259635  awarded to  D. C. Wallace.
Data Availability The datasets supporting the conclusions of this article are included within the article and its additional files. The MitImpact database, together with the APOGEE 2 probabilities/classes of pathogenicity, can be freely downloaded from http://mitimpact. css-mendel. it. Datasets 2 and 3 are available from https://gnomad. broadinstitute. org/ and https://www. helix. com/pages/mitochondrial-variant-database, respectively. Dataset 4 is available from MITOMAP at https://mitomap. org/MITOMAP/Benign.
MITOMAP's "general" variants were discarded. Neither set contained any synergistic or conflicting variants. Dataset 2 was obtained from gnomAD version 3. 1. 2. It contained 36 missense variants not present in the training set, which exhibited allelic frequencies ≥0. 002% both in heteroplasmy and homoplasmy. Dataset 3 was extracted from HelixMTdb ver. 20200327 and filtered as Dataset 2. Twelve variants were shared with Dataset 2 and discarded, and 35 were unique to this dataset. Dataset 4 fetches variants from the benign/likelybenign variant sets approved by the Mitochondrial Variant Curation Expert Panel (VCEP) of ClinGen (https://clinicalgenome. org/) as having met the following criteria for mtDNA variants 10 :.
APOGEE 2 1. 21 scores are decimal values ranging from 0 (benign) to 1 (pathogenic). Extreme values suggest a high level of confidence in the predictions. Scores were computed using the KNN_RusSmote machine learning algorithm, tuned through GS CV and trained on the whole Dataset Given that the algorithm is essentially a bagging model, we were able to determine the out-of-bag (OOB) APOGEE 2 score for each variant in the training set. We used the OOB score to infer the conditional APOGEE 2 probability distribution given the pathogenic class (𝐶𝐶 = 1); the APOGEE 2 probability distribution given the benign class (𝐶𝐶 = 0), instead, was inferred on the Dataset 4 (ClinGen's benign variants) scores, considering the OOB scores whenever they overlapped with Dataset 1 (training set). The distributions of both classes were approximated by Beta distributions, whose parameters were optimized through the SciPy Python library by minimizing the negative log-likelihood probability.
• Uninstall the door at the end of the circuit. The AGVs circulate between the different manual or automated workstations to realize these tasks. This pedagogical scenario should encourage students to innovate, to show creativity during the preparatory and implementation phases. So, a great liberty is left for the students to make decisions to design the assembly line and to integrate the industry 4. 0 concepts. The only fixed element is the AGVs circuit for practical reasons. The AGVs circuit is shown in the figure  (3). Visual Components Software for production system simulation: this solution is based on Discrete Event Simulation (DES). In DES, a process is represented as a sequential series of events with a focus on what the event does, at what time it occurs, and how does it affect the other events.
• Visual Components Software for production system simulation: this solution is based on Discrete Event Simulation (DES). In DES, a process is represented as a sequential series of events with a focus on what the event does, at what time it occurs, and how does it affect the other events. The key point of this solution is the pre-defined and ready-to-use components, the Visual Components eCatalog contains a robust library of virtual models of robots, machines and equipment from dozens of leading brands in industrial automation. With this software, we also have the possibility to visualize and control the dynamic simulation in a Virtual Reality environment. The.