dataesr/scientific-paragraphs-categorization huggingface dataset:
Neural Network Architectures For predicting model parameters from the formatted phylogenies, we considered four different neural network architectures: DNN, CNN RNN and GNN (see also Fig. 1 ). All architectures were built within the torch framework  (Falbel and  Luraschi 2019) in R, except GNNs which were built with Torch in Python because the PyTorch Geometric library  (Fey and Lenssen, 2019)  was needed for GNNs. Hyperparameter tuning for each architecture was performed by hand (see Appendix Tables  S0-S5 ).
ACKNOWLEDGEMENTS We thank  Maximilian Pichler  and  Sonia Kéfi  for helpful discussions and comments on the manuscript. The design of our study profited from the results of a  Ismaël Lajaaiti ,  Sophia Lambert ,  Jakub Voznica ,  Hélène Morlon ,  Florian Hartig MSc  Thesis by  Felix Gottschlich  at the  University of Regensburg  that examined the possibility to use GNNs for phylogenetic inference.
Extent of resection after surgery is one of the main prognostic factors for patients diagnosed with glioblastoma. To achieve this, accurate segmentation and classification of residual tumor from post-operative MR images is essential. The current standard method for estimating it is subject to high inter-and intra-rater variability, and an automated method for segmentation of residual tumor in early post-operative MRI could lead to a more accurate estimation of extent of resection. In this study, two state-of-the-art neural network architectures for pre-operative segmentation were trained for the task. The models were extensively validated on a multicenter dataset with nearly 1000 patients, from 12 hospitals in Europe and the United States. The best performance achieved was a 61% Dice score, and the best classification performance was about 80% balanced accuracy, with a demonstrated ability to generalize across hospitals.
In the absence of a publicly available dataset for residual tumor segmentation from EPMR, the literature on this problem is sparse when compared to the pre-operative segmentation task. Semi-automatic methods, combining of one or several voxel-or shapebased image segmentation algorithm, have been proposed from intensity thresholding (e. g. , Otsu and relative entropy)  [17] [18] [19]  , fuzzy algorithms  18  , Gaussian mixture model  20  , morphological operations  19  , region-based active contours  21, 22  , the level set approach  [21] [22] [23]  , and CNNs  24. Unfortunately, these methods relied on user inputs, either by manual initialisation, or by interactive refinement of the resulting segmentation. They are therefore challenging to use in clinical practice, and in large datasets. In addition, all validation studies were solely performed on single-center local datasets, consisting of 15 to 37 patients, making if difficult to demonstrate the generalizability of the proposed methods.
Regarding fully automated approaches, Meier et al. 25  presented an automated method based on decision forests for residual tumor segmentation using EPMR from 19 patients. A more recent work by Ghaffari et al. 26  proposed to fine-tune a 3D densely connected U-Net, pre-trained on the BraTS20 dataset, on a local dataset of 15 post-operative glioblastomas. However, the MR scans were all acquired for radiation therapy planning and not within the recommended time frame to acquire EPMR scans, within 72 hours after surgical resection  6. Deep learning approaches have recently shown to outperform more traditional algorithms on most image segmentation tasks, including segmentation of pre-operative glioblastomas  15, 16. The utmost requirement is the number of included patients and the quality of the MR images comprising a study dataset.
Preferably, the data should originate from different locations, to evaluate the ability of the trained models to generalize across different hospitals, scanners, or clinical practice. In this work, we determine the performance of two CNN architectures to segment residual enhancing glioblastoma on early post-operative scans. The selected architectures are the nnU-Net, state-of-the-art for pre-operative glioblastoma segmentation, and AGU-Net, an architecture developed for pre-operative segmentation of brain tumors. These architectures have both demonstrated excellent performance on pre-operative segmentation in previous studies on pre-operative brain tumor segmentation  [27] [28] [29]  , and they exhibit different strengths and weaknesses. The automatic results are compared with manual segmentations, using different combinations of MRI scans in a large dataset consisting of paired pre-and early post-operative MRI scans from 956 patients in 12 medical centers in Europe and the United States.
Ethics and Informed Consent Statement The study was conducted in accordance with the Declaration of Helsinki. The study protocol was approved by the Medical Ethics Review Committee of VU University Medical Center (IRB00002991, 2014. 336), and the Norwegian regional ethics committee (REK ref. 2013/1348 and 2019/510). Written informed consent was obtained from patients as required for each participating hospital. The cohorts are subsets of a broader dataset, thoroughly described previously for their pre-operative content  30  , for patients with available EPMR data. All EPMR scans were acquired within 72 hours after surgery, with the exception of the UTR center where the limit used was up to one week post-surgery. The recommended time frame for acquiring the EPMR scans has been stated in the National Comprehensive Cancer Network (NCCN) recommendations  31  , in order to maximize differences between residual enhancing tumor and enhancement due to post-surgical changes in the tissue  32, 33.
Under this paradigm, 352 patients (35%) in our dataset had a GTR, whereas the remaining 604 patients had residual tumor. An overview of the data from the 12 hospitals is shown in Table  1 , and some examples are illustrated in Figure  1. In addition, 20 patients out of the 73 in the AMS cohort have been annotated eight times in total, by four novices raters and four experts raters. This cohort has been used in a previous study to evaluate the inter-rater variability of tumor annotation from annotators with different levels of experience 7 , and will be referred to as the inter-rater variability dataset in the remainder of the document.
Pre-processing For proper anatomical consistency across the different inputs sequences, an initial image-to-image registration procedure was performed. The EPMR T1w-CE scan was elected as the reference space and all subsequent volumes were registered to it using the SyN diffeomorphic method  35  from the Advanced Normalization Tools (ANTs) framework  36. Skull-stripping was subsequently performed on all input MR scans, based on the brain mask from the EPMR T1w-CE scan. All brain masks were automatically generated using a pre-trained slab-wise AGU-Net model with input shape 256 × 192 × 32 voxels. For the nnU-Net architecture, the pre-processing was automatically decided by the framework based on the dataset, and all inputs were resampled to 0. 5 × 0. 5 × 1. 0 mm 3 spacing and zero-mean normalized.
Training specifications for the nnU-Net architecture Architecture design From the nnU-Net framework analysis of the dataset, the 3D full-resolution U-Net with the following parameters was recommended, using 192 × 128 × 80 voxels as input patch size. The network used five levels, downsampling using strided convolution layers, and upsampling using transposed convolution layers. Kernel size of 1 × 3 × 3 voxels for the first level, 3 × 3 × 3 for the remaining four levels, and filter sizes of {32, 64, 128, 256, 320} were used for each level, respectively. The loss function was a combination of the Dice score and cross-entropy. A stride of one was used for the convolution layers. Network training All models were trained from scratch for 1000 epochs using a stochastic gradient descent with Nesterov momentum optimizer (momentum=0.
Protocols The validation studies presented in this work were conducted following a five-fold cross-validation, summarized in Table  2. First, all patients from 11 out of the 12 the hospitals in our dataset, excluding the AMS cohort, were split into five hospital-stratified folds, with an approximately balanced number of patients in each fold. The remaining 73 patients from the AMS hospital were kept as an hold-out test set. For each iteration of the cross-validation, four folds were used for training, the remaining fifth fold was used for validation, and the hold-out set was used for test. This approach presents similar benefits to the leave-one-hospital-out strategy used in previous work  27  , with the advantage of a reduced training time. Finally, predictions over the test set were generated by ensembling over the predictions obtained by each of the five trained models.
A residual tumor volume below the clinical volume threshold was thus counted as a negative (i. e. , GTR) and as positive otherwise (i. e. , RT). Following this consideration, a patient was considered a true positive (TP) if both the ground truth annotation residual tumor volume and detected residual tumor volume were ≥ 0. 175 ml, for any given Dice score (i. e. , ≥ 0. 01). Conversely, if both volumes were < 0. 175 ml, the patient was labelled as a true negative (TN). Patients where the ground truth volume was above the threshold volume and the prediction was below were marked as false negatives (FN), and false positive (FP) vice versa. In the case of inter-rater variability, the Jaccard score, closely related to the Dice score by J = D 2-D , was used to compare the models' performance. The Jaccard was chosen for easy comparison with a previously published work on the same dataset 7.
Experiments The following three experiments were conducted in this study: (i) Residual tumor segmentation performance study: using the 5-fold cross-validation protocol and segmentation metrics, both nnU-Net and AGU-Net architectures' segmentation performances were compared for the five combinations of input sequences. (ii) Gross total resection classification performance study: using the 5-fold cross-validation protocol, classification metrics, and best input combination identified in the first experiment, both architectures were compared in terms of ability to classify between gross total resection and residual tumor patients. (iii) Inter-rater variability study: the best model from each architecture was benchmarked in terms of segmentation performance against the performance of novice and expert annotators, using the inter-rater variability dataset.
Results The studies were performed using multiple machines with the two following specifications: (i) Intel Core Processor (Broadwell, no TSX, IBRS) central processing unit (CPU) with 16 cores, 64GB of RAM, Tesla V100S (32GB) dedicated GPU, and a regular hard-drive and (ii) a GPU server with a total of 256 CPU cores, 2TB of RAM, and six NVIDIA A100-SXM4 (80GB) cards. The AGU-Net architecture was implemented in Python 3. 6 with the TensorFlow v1. 13. 1 library  38. For the nnU-Net architecture, Python 3. 8, PyTorch v1. 13. 1  39  , and the nnU-Net framework v1. 7. 0 14 were used. Segmentation performances across both architectures, for all input sequences combinations, and only for patients with residual tumor are summarized in Table  3. For both architectures, the lowest average Dice score over the external test set was obtained with configuration A, indicating that solely using T1w-CE MR scans is insufficient for identifying post-operative residual tumor.
Inter-rater variability study For the 20 patients constituting the inter-rater variability dataset, a comparison of the Jaccard scores, obtained between each rater and the best model from each architecture, are reported in Figure  4. As all configurations B-E yielded very similar segmentation performance scores, the selected best configuration for each architecture were the models that produced the best trade-off between sensitivity and specificity for the classification task. For the nnU-Net, configuration D was selected as the best configuration as this model achieved the highest specificity out of all the trained nnU-Net models, and for the AGU-Net, configuration E was selected as this model yielded the highest sensitivity while maintaining a reasonable specificity. Using the ground truth annotation from the test dataset as a reference segmentation, both architectures achieved Jaccard scores within a variability range very similar to that of the novice and expert annotators.
The multimodal and multicentric dataset in this study is the largest cohort used for the task of early post-operative glioblastoma segmentation, with a total of 956 patients. Regarding the dataset curation, our strict inclusion criteria required availability of all four MR scans as input (i. e. , post-operative T1w-CE, T1w, FLAIR, and pre-operative T1w-CE) for each patient. Whereas this decision was motivated by a simpler method design, approximately 150 patients were excluded as one or more MR scans were missing. A relaxation of the inclusion criteria would increase the size of the dataset, and open the possibility to generate a more diverse set of input MR scans, including for example T2-weighted images. Ideally, the trained methods should be able to deal with a sparse set of inputs, where one or more MR scans are missing.
Acknowledgements Data were processed in digital labs at HUNT Cloud,  Norwegian University of Science and Technology, Trondheim, Norway. FB is supported by the  National Institute for Health Research (NIHR)  biomedical research centre at  UCLH. The PICTURE project is sponsored by an unrestricted grant of  Stichting Hanarth fonds , " Machine learning for better neurosurgical decisions in patients with glioblastoma "; a grant for public-private partnerships ( Amsterdam UMC PPP-grant ) sponsored by the  Dutch government (Ministry of Economic Affairs)  through the  Rijksdienst voor Ondernemend Nederland (RVO)  and  Topsector Life Sciences and Health (LSH) , " Picturing predictions for patients with brain tumors "; a grant from the  Innovative Medical Devices Initiative program , project number  10-10400-96-14003 ; The  Netherlands Organisation for Scientific Research (NWO) ,  2020.
027 ; a grant from the  Dutch Cancer Society ,  VU2014-7113  and the  Anita Veldman foundation ,  CCA2018-2-17. R. H. H. is supported by a grant from  The Research Council of Norway , grant number  323339. D. B. , I. R. , and O. S. are partly funded by the  Norwegian National Research Center for Minimally Invasive and Image-Guided Diagnostics and Therapy.
Our pipeline can also be used in combination with molecular docking, either by simply prioritizing compounds or by mixing the predicted score with the docking score. We show that our new model widely outperforms previous ones. We also introduce a virtual screening test case for RNA and evaluate the performance of our tool in a realistic drug discovery setting. Our affinity prediction model shows native ligand recovery rates competitive with full docking experiments with runtimes in seconds. When the affinity prediction model is used to prioritize docking candidates, we show a 1200% acceleration of native ligand recovery with no performance loss. Finally, we show that using our model in combination with docking enhances virtual screening results with almost no overhead. Our source code and data, as well as a Google Colab notebook for inference, are available on GitHub.
These methods offer higher confidence screening results but come at the cost of high computational demands : time needed to evaluate a single pocket-ligand pair with a docking software is on the order of minutes. Attempts at accelerating this process use statistical or heuristic physics-based scoring approaches such as LigandRNA  [29]  and RLDOCK  [38]. On the implicit side, methods try to further increase throughput by learning complex statistical associations between RNA and small molecules using machine learning methods. Here, tools such as InfoRNA  [10]  match a query secondary structure RNA with a library of structure motifs associated with known binders in an experimental assay. RNAmigos  [24]  made the first attempt at explicitly modeling the tertiary structure of the pocket to screen compound libraries and showed the potential of graph representations based on the Leontis-Westhof nomenclature of base pairs in 3D  [21].
Dataset Construction The first step of our pipeline is to create a relevant dataset for training and evaluating our models. Training data has four main components: 1) a supervised dataset of binding site-native ligand pairs from the PDB, 2) synthetic affinity data from large-scale docking experiments, 3) local substructures sampled from a non-redundant set of RNA structures for unsupervised learning, and 4) a large library of chemical compounds. These datasets are summarized in Figure  1.
Constructing a Dataset of Binding Sites To construct our dataset of binding sites, we downloaded all RNA structures published in the RCSB Protein Data Bank (PDB)  [32]  and we filtered these structures into a set of RNA-small molecule complexes containing RNA and at least one ligand. We then extract RNA binding pockets as RNA residues with an atom at most 10Å from the ligand, and filter out resulting binding sites with less than five RNA residues or involving a majority of protein residues. Pocket-ligand pairs (systems) are filtered based on their ligands, following Hariboss rules  [26]  (e. g. removing ions, crystallization artefacts, etc. ). The final dataset consists of 1740 binding sites associated with 264 unique ligands. We then used RNAGlib  [23]  to obtain a representation of the structure of our binding sites in the form of ABPN graphs.
Building a Library of Docking Scores We build three ligands sets to screen against our pockets. The first set of small molecules is the set of 264 co-crystallized binders present in the PDB. For each of those compounds, we obtain non-binders using DecoyFinder  [6]  to simulate a virtual screening without docking (following RNAmigos), resulting in a second set. The last set of compounds is used to simulate an actual virtual screening. We used the ChEMBL  [13]  database to select ligands with physico-chemical characteristics plausible in a drug design setting. We picked a diverse set of ChEMBL compounds using the MaxMin algorithm (we used the RDKit implementation  [19] ) based on Tanimoto distance between fingerprints. We complement this procedure with a drug-like filter and keep compounds with a molecular weight below 400, at least one ring, less than five rotatable bonds and H-bond donors, less than ten H-bond acceptors and logp below five.
We follow that procedure to retrieve a fixed set of 500 compounds. Then, for each binding site, we ensure that up to 100 compounds have a Tanimoto Coefficient greater than 0. 6 with the native ligand by sampling additional ligands if necessary. We assessed two candidate docking tools, rDock  [33]  and AutoDock Vina  [39]  which were identified in a recent review of nucleic-acid docking tools  [22]. In addition, we used AnnapuRNA  [37] , a knowledge-based scoring function, as an independent referee to compare these candidate software in terms of native pose identification. We performed self-docking and cross-docking experiments with this software to compare their capabilities for RNA molecules and present the results in the Appendix (section A. 2. 1). Eventually, we chose to use rDock to train our models as it showed the best performance.
We perform docking for two reasons: to use the docking results as a data augmentation, and to obtain virtual screening results. We thus dock the 264 native binders and the 500 ChemBL compounds in our 1740 binding sites, resulting in over 1. 2 million values (runtime details are presented in A. 2. 2). This enriches our original scarce dataset of experimental binding sites. We additionally rely on all RNA residue neighborhoods and a curated dataset of ZINC  [18]  compounds to pretrain our RNA and molecule encoders, as will be detailed below. The whole of the data we will be using is presented in Figure  1.
Models Equipped with this dataset, we are now ready to train models to complement molecular docking with an increased throughput, building over RNAmigos  [24]. We use the ABPN graph representation, and make it more powerful by allowing the graph to have directed edges, resolving an ambiguity for asymmetric edges such as Sugar-Hoogsteen (SH). Given such a directed graph G = (V, E), we then use a neural network encoder f ✓ based on Relational Graph Convolutional Network (RGCN)  [34]  layers to produce node embeddings of dimension d 2 N and a pooling step reduces those node embeddings to a single graph-level vector representation G = f ✓ (G). We then propose three strategies to model binding sites and ligands interactions, depicted in Figure  2. In the fp setting, we input the binding site to a graph encoder, and decode the resulting embedding into a predicted fingerprint for this binding site.
In the other settings, we additionally encode candidate ligands structures and use our decoder to predict a compatibility score. For the native setting, the score is supervised with a binary classification into native or decoy, whereas in the dock setting, this score tries to approximate the docking score for the given binding site and ligand pair. In our first setting (fp), this graph representation G is fed through a decoder g ✓ to predict a fingerprint of dimension d, following RNAmigos  [24] , so that our model can be written as M 1 = g ✓ f ✓ with M 1 (G) 2 [0, 1] d. We used a Multi-Layer Perceptron (MLP) for the decoder and train our networks using the norm two distance between our prediction and the fingerprint of the native ligand F, so that the loss can be written as L 2 (G, F) = kM 1 (G) Fk 2.
Model pretraining In low data regimes which are common for tasks on RNA 3D structures, it has been shown that pre-training models on larger yet unlabeled datasets can benefit the performance on specialized downstream models  [24]. We therefore pretrain our RNA encoder using the strategies proposed in  [25]  which are based on the metric learning framework  [42]. The intuition behind such methods is to train the model to recognize geometrically similar structures according to a similarity function. It was shown to help convergence for downstream tasks where structure is relevant. Our directed graphs open the door to using more powerful similarity functions, and hence enhance pre-training. Following this framework, we pretrain our ligand encoder using rooted subgraphs around all nodes of RNA non redundant data represented as graphs.
Therefore, in the following and for clarity, we will report AuROC scores and denote them as Mean Active Rank (MAR). Because the number of drug-like compounds available in databases such as Enamine  [15]  is in the billions, slow running methods such as molecular docking are not tractable. Thus, we introduce a novel metric, denoted efficiency, to measure the expected performance if we were to stop our screening after a certain time budget is spent. We perform a virtual screening over a randomly ordered list of compounds and put processed compounds at the top of the list. We keep track of the MAR as a function of time and define the efficiency of a search as the area under the time vs. MAR curve. In this manner we can capture the trade-off between time expenditure and the quality of the search results in a single value.