dataesr/scientific-paragraphs-categorization huggingface dataset:
intensity normalization to the range [0, 1]. Training strategy: Models were trained using the Adam optimizer over a batch size of 32 samples with accumulated gradients (actual batch size 2), stopped after 30 consecutive epochs without validation loss improvement, following either: (i) training from scratch with 1e -3 initial learning rate, or transfer learning with an initial learning rate of 1e -4 fine-tuning over the best glioblastoma model. For the data augmentation strategy, the following transforms were applied to each input sample with a probability of 50%: horizontal and vertical flipping, random rotation in the range [-20 • , 20 • ], and translation up to 10% of the axis dimension. Training protocol: Given the magnitude difference within our four datasets, two different protocols were considered: (i) a three-way split at the hospital level whereby MRI volumes from one hospital constituted the validation fold; MRI volumes from a second hospital constituted the test fold; and the remaining MRI volumes constituted the training fold.
Proposed Software In order to make our models and tumor features easily available to the community, we have developed two software solutions. The first one is a stand-alone software called Raidionics, and the second one is a plugin to 3D Slicer given its predominant and widespread use in the field  (46). Both solutions provide access to a similar back-end including inference and processing code. However, the GUI and intended user interactions differ. The trained models are stored in a separate online location and are downloaded on the user's computer at runtime. Models can be improved over time and a change will be automatically detected, resulting in the replacement of outdated models to the user's machine. Raidionics can be seen as an improved solution to our initial GSI-RADS software, covering not only glioblastomas but all major brain tumor types, offering the option to compute a similar standardized report, and providing a refined graphical user interface enabling the user to visually assess the results.
Stand-Alone Solution: Raidionics The software proposes two modes: (i) single-use where only one patient is to be processed and results can be visually assessed in the 2D viewer, and (ii) batch-mode whereby a collection of patients can be processed sequentially without any viewing possibility. In each mode, the option is left to the user to solely perform tumor segmentation or to compute the whole set of tumor characteristics and generate the standardized report. For each patient, the software expects an MRI scan as input (i. e. , Gd-enhanced T1-weighted or FLAIR sequence) and the tumor type must be manually selected. Additionally, a preexisting tumor segmentation mask can be provided to bypass the automatic segmentation, if collecting the tumor characteristics is the main interest and manual annotations have been performed beforehand.
The total set of processed files saved on disk includes the standardized reports, brain and tumor segmentation masks in both patient and MNI space, cortical and subcortical structures masks in both patient and MNI space, and the registration files to navigate from patient to MNI space. To complement the reporting and give the possibility for follow-up statistical studies, the complete set of computed features is also provided in comma separated value format (i. e. ,. csv). The software has been developed in Python 3. 6. 9, using PySide2 v5. 15. 2 for the graphical user interface, and only uses the Central Processing Unit (CPU) for the various computations. The software has been tested and is compatible with Windows (≥ 10), macOS (≥ Catalina 10. 15), and Ubuntu Linux (≥ 18. 04). An illustration of the software is provided in Figure  2.
3D Slicer Plugin: Raidionics-Slicer The 3D Slicer plugin has been developed using the DeepInfer plugin as baseline (  47 ) and is mostly intended for tumor segmentation purposes. Through a slider, the possibility is provided to manually alter the probability threshold cutoff in order to refine the proposed binary mask. Further manual editing can be performed thereafter using the existing 3D Slicer functionalities. The back-end processing code has been bundled into a Docker image for convenience, and therefore, administrator rights are required for the end-user to perform the installation locally. The same inputs, behavior, and outputs can be expected as for the stand-alone software. The GitHub repository for the 3D Slicer plugin can be found at https://github. com/dbouget/Raidionics-Slicer, and an illustration is provided in Figure  3.
Representative models selection: identification of one final segmentation model for each tumor type, which will be made available for use in our software solutions. 4. Speed study: computation of the pure inference speed and the total elapsed time required to generate predictions for a new patient, obtained with CPU support and reported in seconds. The operations required to prepare the data to be sent through the network, initialize the environment, load the trained model, and reconstruct the probability map in the referential space of the original volume are accounted for. The experiment was repeated ten consecutive times over the same MRI volume for each model, using a representative sample of each dataset in terms of dimension and spacing.
Implementation Details Results were obtained using a computer with the following specifications: Intel Core Processor (Broadwell, no TSX, IBRS) CPU with 16 cores, 64GB of RAM, Tesla V100S (32GB) dedicated GPU and a regular hard-drive. Training and inference processes were implemented in Python 3. 6 using TensorFlow v1. 13. 1, and the data augmentation was performed using the Imgaug Python library  (63). The metrics were for the most part computed manually using the equations described in the Supplementary Material, or alternatively using the sklearn v0. 24. 2 (64) and medpy v0. 4. 0 (65) Python libraries. The source code used for computing the metrics and performing the validation studies is made publicly available at https://github. com/dbouget/validation_metrics_computation.
Overall Performance Study An overall summary of brain tumor segmentation performance for all four tumor subtypes is presented in Table  3. Meningiomas and lower grade gliomas appear more difficult to segment given average Dice scores of 75%, compared to average Dice scores of 85% for glioblastomas and metastases. A similar trend, yet with a slightly smaller gap, can be noted for the Dice-TP scores ranging between 81 and 90% with a standard deviation of approximately 15%, indicating the quality and relative stability of the trained models. From a patient-wise perspective, those results demonstrate the difficulty of achieving good recall while keeping the precision steadily above 95%. Even though a direct comparison to the literature is impossible since different datasets have been used, obtained performance is on-par if not better than previously reported performances where Dice scores have been ranging from 75 to 85%.
Regarding the lower grade glioma tumor subtype, the diffuse nature of the tumors and less pronounced gradients over image intensities are possible explanations for the lower segmentation performance. For the meningioma category, the reason for the lower Dice-score and recall values can be attributed to the larger number of small tumors (< 2 ml) compared to other subtypes. In addition, outliers have been identified in this dataset whereby a small extent of the tumors were either partly enhanced   S1-S4 ). While tumor volumes and outlier MR scans are reasons for the discrepancy in Dice and recall values across the board, precision is rather unaffected and more stable. The nature of the convolutional neural network architecture and training strategy used can explain those results. By leveraging volumes covering the full brain, global relationships can be learned by the trained model hence reducing the confusion between tumor regions and other contrast-enhancing structures such as blood vessels.
top row in Supplementary Figures  S1-S4 ). Given the non-normalized state of the GCE metric, and its absence of any upper bound, an extremely poor agreement between manual ground truth and automatic segmentation will result in score orders of magnitude higher than its average expression over a given dataset. Regarding the two volume-based metrics, featured rightmost in the second table, an antagonistic pattern toward Dice can be observed. The VS metric has the same cross-type trend as Dice with similar yet slightly greater scores. On the other hand, while the RAVD metric scores best over the metastasis group similar to Dice, its worst average value is obtained for the glioblastoma group, hence potentially exhibiting the same frailty toward outliers as for the GCE metric. Next off, voxel-wise performance for information theorybased and probabilistic metrics are regrouped in Table  6.
Representative Models Selection Only one model can be provided in the software solutions for each tumor type, and the best model selection was done empirically according to the following criterion: the size of the validation or test set, average Dice score, and patient-wise F1-score performances. The exhaustive list of chosen models is the following: the model trained for fold 0 was selected for the glioblastomas, the model trained for fold 3 was selected for the lower grade gliomas, for the meningiomas the model trained for fold 2 was selected, and finally for the metastases the model trained for fold 2 was selected.
Speed Study A comparison in processing speed regarding pure tumor segmentation and complete generation of standardized reports is provided in Table  10  when using the Raidionics software with CPU support. The high-end computer is the computer used for training the models, whereas the mid-end computer is a Windows laptop with an Intel Core Processor (i7@1. 9GHz), and 16GB of RAM. For the smallest MRI volumes on average, 17 s are needed to perform tumor segmentation whereas 4. 5 min are required to generate the complete standardized report with the high-end computer. Unsurprisingly, the larger the MRI volume the more time required to perform the different processing operations (cf. Supplementary Section S3). For the largest MRI volumes overall, 54 s are needed to perform tumor segmentation whereas 15.
DISCUSSION In this study, we have investigated the segmentation of a range of common main brain tumor types in 3D preoperative MR scans using a variant of the Attention U-Net architecture. We have conducted experiments to assess the performances of each trained model using close to 30 metrics and developed two software solutions for end-users to freely benefit from our segmentation models and standardized clinical reports. The main contributions are the high performances of the models, on-par with performances reported in the literature for the glioblastomas, with illustrated robustness and ability to generalize due to the multiple and widespread data sources. In addition, the two proposed open-access and open-source software solutions include our best models, together with a RADS for computing tumor characteristics.
This is the first open RADS solution that supports all major brain tumor types. The software is user-friendly, requiring only a few clicks and no The color intensity of each cell represents the strength of the correlation, where blue denotes direct correlation and red denotes inverse correlation. The architecture and training strategy used in this study were identical to our previously published work considering that the intent was not to directly make advances on the segmentation task. Nevertheless, the stability and robustness to train efficient models had been documented, alongside performance comparison to another well-known architecture [e. g. , nnU-Net (20)], thus not precluding its use to train models for other brain tumor types. Aside from evident outliers in the datasets, where either tumors with partial or missing contrast uptake or suboptimal manual annotations were identified, the major pitfall of using the AGU-Net architecture lies in its struggle to segment equally satisfactorily small tumor pieces with a volume below 2 ml.
Overall, the glioblastoma model is expected to be the most robust and able to generalize since patient data from 15 different sources were used. For other models trained on data from much fewer hospitals, with an expected limited variability in MR scan quality, their robustness is likely to be inferior. While larger datasets are often correlated with improved segmentation performance, the metastasis model is the best performing with the lowest amount of patients included. The relative easiness of the task from a clear demarcation of the tumor from surrounding normal tissue in contrast-enhanced T1weighted volumes, and the potentially low variance in tumor characteristics with patient data coming from two hospitals only, can explain the results. Additionally, the metastasis model has been trained by transfer-learning using as input the second best performing glioblastoma model where the most data was used, which may have been the compelling factor.
In general, a careful mix between losses (e. g. , Dice, cross-entropy, and HD95) is challenging to achieve and adaptive strategies might be required to avoid reaching a local minimum where overall segmentation performance may suffer  (68). As a current trend in the community, inference code and trained segmentation models are often at best available on GitHub repositories. As a consequence, only engineers, or people with some extent of knowledge in machine learning and programming, can benefit from such research advances. Besides, the research focus is heavily angled toward gliomas, due to the BraTS challenge influence, whereby segmentation models are expected to yield superior performance than for meningiomas and metastases. By developing and giving free and unrestricted access to our two proposed software solutions, we hope to facilitate more research on all brain tumor types.
Willing research institutes have the opportunity to generate private annotated datasets at a faster pace than through fully manual labor by exploiting our trained models. Having made all source code available on GitHub, as customarily done, we made the effort to further make stand-alone solutions with easyto-use GUIs. Hopefully, clinicians and other non-programming end-users should feel more comfortable manipulating such tools, available across the three major operating systems and necessitating only a computer with average hardware specifications. For the generation of standardized clinical reports, the computation of tumor characteristics relies heavily on the quality of the automatic segmentation, occasional mishaps are expected as models are not perfect and can omit the tumor. Therefore, manual inputs will be required sporadically to correct the tumor segmentation.
Over time, new and better models will be generated and made available seamlessly into the two software through regular updates. For the time being, support for AGU-Net models only is provided due to its lighter codebase compared to nnU-Net, for similar overall performances. From a software bundling and deployment perspective, integrating a heavier inference framework and mixing backend engines (i. e. , TensorFlow and Torch) will make it more challenging to create stable executables for Raidionics on Mac, Windows, and Ubuntu. Support for other architectures will be considered if new models clearly outperform the current models. In the future, an approach incorporating a set of metrics and converting them into one final score would be highly desirable (e. g. , Roza). Not only would it help to automatically select the best model from a k-fold validation study from one unique score, but a proper assessment and ranking across multiple methods would be enabled.
With all preoperative brain tumor types available for segmentation and reporting in our software, a key missing component is the automatic tumor type classification to supplement manual user input. Concurrently, the variety and amount of tumor characteristics to compute should be extended, considering more type-specific features similar to the resection index for glioblastomas. Alternatively, bringing a similar focus on post-operative segmentation of residual tumors is of great interest to both assess the quality of the surgery and refine the estimated patient outcome. The generation of a complete post-operative standardized clinical report would also be permitted with new features such as the extent of resection. Otherwise, intensifying the gathering of patient data from more widespread hospital centers and a larger array of MRI scanners is always of importance.
CONCLUSION Efficient and robust segmentation models have been trained on pre-operative MR scans for the four main brain tumor types: glioblastoma, lower grade glioma, meningioma, and metastasis. In-depth performance assessment allowed to identify the most relevant metrics from a large panel, computed either voxel-wise, patient-wise, or instance-wise. Trained models and standardized reporting have been made publicly available and packaged into a stand-alone software and a 3D Slicer plugin to enable effortless widespread use.
× [114; 560] × [19; 561] voxels, and the voxel size ranges [0. 43; 1. 33] × [0. 43; 1. 80] × [0. 45; 7. 0] mm 3. An average MRI volume is [301 × 370 × 289] pixels with a spacing of [0. 85 × 0. 76 × 1. 08] mm 3. , showcasing patients either followed at the outpatient clinic or recommended for surgery at the St. Olavs Hospital, Trondheim University Hospital, Norway. Overall, MRI volume dimensions are covering [192; 512] × [224; 512] × [11; 290] voxels, and the voxel size ranges [0. 41; 1. 05] × [0. 41; 1. 05] × [0. 60; 7. 00] mm 3. An average MRI volume is [343 × 350 × 147] pixels with a spacing of [0. 78 × 0. 78 × 1. 67] mm 3. 2. 4. Metastases The metastasis dataset is made of a total of 396 Gd-enhanced T1- weighted MRI volumes, collected from two different hospitals: 329 patients from the St.
Next Generation Sequencing Next-Generation Sequencing (NGS) of the whole exome (the entire coding region of the genome and intron-exon junction regions, corresponding to approximately 23,000 nuclear genes) was performed on a DNB-SEQ400 next-generation mass sequencer using an MGI-V5 exome library. All sequenced data were quality assessed (base quality distribution, nucleotide distribution, and presence of adapters, chimeras, and other contaminants) to identify/remove low-quality data/samples from further analysis.
A more conservative filter would keep based on 3, 4, or 5 damaging predictions. Many variants do not have five algorithms with nonmissing values  [21] [22] [23] [24]. Subsequently, variants identified in genes with the known clinical association and related to monogenic and mitochondrial diseases were analyzed in contrast to our patient's phenotype and clinical manifestations. Only genes with an average coverage of more than 98% and a minimum depth of 20× were analyzed for accomplishing quality standards. As a note of caution, it is essential to mention that this study may not cover all possible pathogenic variants in each gene, and for technical and scientific reasons, whole-exome sequencing may not completely cover the entire coding region of the human genome. The sequencing results were bioinformatically analyzed using the DreamGenics-GenomeOne software, and the aligned and filtered sequences (fulfilling specific quality criteria) were compared against the GRCh37/hg19 reference genome for annotation and variant calling.
Bioinformatics analyses aimed to identify variants included in exonic regions or splicing regions (at least 20 bp), small insertions and deletions, and, in particular cases, genes related to the patient's clinical phenotype. Parameters such as the population frequency of the variant (i. e. , in ExAC, gnomAD, and 1000 Genomes databases), its allelic influence, the associated OMIM phenotype, its classification in ClinVar, and the inheritance model were used to filter the variants to be reported. This sequential, comprehensive, and sometimes targeted analysis allowed for the identification of exonic deletions and duplications (also known as Copy Number Variants, or CNVs) and variants involving large gene regions. If CNVs were identified, a secondary confirmation method was either applied or recommended, as NGS is an indirect method for their identification  [21] [22] [23] [24].
Variants outside of coding regions or harbored out of the intron-exon junctions of the gene cannot be identified (e. g. , variants in promoter regions, enhancers, regulatory regions far from the exonic region, unique variants including dynamic mutations, complex recombination structures, structural variants of genes (e. g. , inversion-type rearrangements), and epigenetic effects). Identified variants have been evaluated against HGMD, ClinVar, LOVD, dbSNP, and gnomAD databases. For missense and splicing site variants of uncertain significance, in silico prediction tools were used, and their analysis is reported in the interpretation of the results. Finally, the association of the identified variants with the syndromes described in OMIM and the clinical association with the phenotype described in the patient are evaluated  [21] [22] [23] [24].
Energetic and Stereochemical Characterization of the PURA Molecular Models Models obtained by molecular modeling were energetically and stereochemically characterized. For the energetic characterization, we used SAVES v6. 0 coupled to the ERRAT parameter. Additionally, we used the SWISS_MODEL server to characterize the QMEAN6 parameter to define a normalized value (linear combination of six terms in favor of its stability). We also estimated the Z-score parameter to indicate whether the model is in trend or not in comparison with experimentally solved structures (either with X-ray diffraction [XRD], cryogenic imaging microscopy [CryoEM], or solid nuclear magnetic resonance [NMR] techniques)  [36, 37]. The stereochemical characterization was performed with the PROCHECK software, which allows one to measure the stereochemical viability of the model by considering the dihedral angles Phi (ϕ) and Psi (ψ) of the planes generated by the peptide bonds of the amide functional groups that link the amino acids in the proteins.
Calculation of Hydropathic Indexes and Polarity Surfaces of PURAwt and Mutated-Phe233del Because a deletion with potential effects was found on the conformation of the secondary structure, polarity evaluation was performed through the hydropathicity index, and the Molecular Lipophilicity Potential (MLP) maps for proteins. The polarity changes in the surroundings of the mutation were calculated with the Protscale software of the Swiss ExPASY suite, which compares the primary sequence of the PURAwt and the mutated-Phe233del variant using the Kyte and Dolittle coefficients. Depending on the score, and its sign, a hydrophilic environment (negative) or a hydrophobic environment (positive) can be favored, with zero being the threshold that separates them  [39]. The molecular surface lipophilicity potential was obtained with the Chimera software, and was categorized in a range between (-20, 20) defined with a color code; negative hydrophilic values are represented in green, while the positive hydrophobic are represented in gold  [40].
3D Alignment Structural Viewers This 3D alignment is intended to obtain the coincidence structurally in a significant part of the spatial distribution of both models. The spatial co-ordinates and the selection of the molecular regions were made with the Deepview/Swiss-PdbViewer software version 3. 7  [41]. The Chimera UCSF v 1. 1. 1 software was used for visualization, and the structural alignments were made with the Needleman-Wunsch global algorithm and the BLOSUM62 matrix. The quantitative alignment to find the RMSD value in the structural comparison was made using PyMol  [40, 42, 43].