dataesr/scientific-paragraphs-categorization huggingface dataset:
5. Finally, the "non-enhancing (solid) core" structures were defined as the remaining part of the gross tumor core, i. e. , after subtraction of the "enhancing core" and the "necrotic (or fluid-filled) core" structures [Fig. 3(D) ]. Following this protocol, the MRI scans were annotated by a trained team of radiologists and altogether seven radiographers in Bern, Debrecen and Boston. They outlined structures in every third axial slice, interpolated the segmentation using morphological operators (region growing), and visually inspected the results in order to perform further manual corrections, if necessary. All segmentations were performed using the 3D slicer software 5 , taking about 60 min per subject. As mentioned previously, the tumor labels used initially in the BRATS 2012 challenge contained only two classes for both high-and low-grade glioma cases: "edema," which was defined similarly as the edema class above, and "core" representing the three core classes.
F. Online Evaluation Platforms A central element of the BRATS benchmark is its online evaluation tool. We used two different platforms: the Virtual Skeleton Database (VSD), hosted at the University of Bern, and the Multimedia Digital Archiving System (MIDAS), hosted at Kitware  [80]. On both systems participants can download annotated training and "blinded" test data, and upload their segmentations for the test cases. Each system automatically evaluates the performance of the uploaded label maps, and makes detailed-case by case-results available to the participant. Average scores for the different subgroups are also reported online, as well as a ranked comparison with previous results submitted for the same test sets. The VSD 6 provides an online repository system tailored to the needs of the medical research community.
In addition to storing and exchanging medical image datasets, the VSD provides generic tools to process the most common image format types, includes a statistical shape modeling framework and an ontology-based searching capability. The hosted data is accessible to the community and collaborative research efforts. In addition, the VSD can be used to evaluate the submissions of competitors during and after a segmentation challenge. The BRATS data is publicly available at the VSD, allowing any team around the world to develop and test novel brain tumor segmentation algorithms. Ground truth segmentation files for the BRATS test data are hosted on the VSD but their download is protected through appropriate file permissions. The users upload their segmentation results through a webinterface, review the uploaded segmentation and then choose to start an automatic evaluation process.
MIDAS 7 is an open source toolkit that is designed to manage grand challenges. The toolkit contains a collection of server, client, and stand-alone tools for data archiving, analysis, and access. This system was used in parallel with VSD for hosting the BRATS training and test data in 2012, as well as managing submissions from participants and providing final scores using a collection of metrics. It has not been used any more for the 2013 BRATS challenge. The software that generates the comparison metrics between ground truth and user submissions in both VSD and MIDAS is available as the open source COVALIC (Comparison and Validation of Image Computing) toolkit. 8.
IV. Results In a first step we evaluate the variability between the segmentations of our experts in order to quantify the difficulty of the different segmentation tasks. Results of this evaluation also serve as a baseline we can use to compare our algorithms against in a second step. As combining several segmentations may potentially lead to consensus labels that are of higher quality than the individual segmentations, we perform an experiment that applies the hierarchical fusion algorithm to the automatic segmentations as a final step. 6www. virtualskeleton. ch 7www. midasplatform. org 8github. com/InsightSoftwareConsortium/covalic.
A. Inter-Rater Variability of Manual Segmentations Fig. 5  analyzes the inter-rater variability in the four-label manual segmentations of the training scans (30 cases, four different raters), as well as of the final off-site test scans (15 cases, three raters). The results for the training and test datasets are overall very similar, although the inter-rater variability is a bit higher (lower Dice scores) in the test set, indicating that images in our training dataset were slightly easier to segment (Fig. 5 , plots at the top). The scores obtained by comparing individual raters against the consensus segmentation provides an estimate of an upper limit for the performance of any algorithmic segmentation, indicating that segmenting the whole tumor region for both low-and highgrade and the tumor core region for high-grade is comparatively easy, while identifying the "core" in low-grade glioma and delineating the enhancing structures for high-grade cases is considerably more difficult (Fig.
5 , table at the bottom). The comparison between an individual rater and the consensus segmentation, however, may be somewhat overly optimistic with respect to the upper limit of accuracy that can be obtained on the given datasets, as the consensus label is generated using the rater's segmentation it is compared against. So we use the inter-rater variation as an unbiased proxy that we compare with the algorithmic segmentations in the remainder. This sets the bar that has to be passed by an algorithm to Dice scores in the high 80% for the whole tumor region (median 87%), to scores in the high 80% for "core" region (median 94% for high-grade, median 82% for lowgrade), and to average scores in the high 70% for "active" tumor region (median 77%) (Fig.
A. Overall Segmentation Performance The synthetic data was segmented very well by most algorithms, reaching Dice scores on the synthetic data that were much higher than those for similar real cases (Fig. 6 , top left), even surpassing the inter-rater accuracies. As the synthetic datasets have a high variability in tumor shape and location, but are less variable in intensity and less artifact-loaded than the real images, these results suggest that the algorithms used are capable of dealing well with variability in shape and location of the tumor segments, provided intensities can be calibrated in a reproducible fashion. As intensity-calibration of magnetic resonance images remains a challenging problem, a more explicit use of tumor shape information may still help to improve the performance, for example from simulated tumor shapes  [81]  or simulations that are adapted to the geometry of the given patients  [31].
These choices can certainly be criticized; however, we believe the major challenge for the segmentation algorithms was ultimately not interpolation or label fusion details, but rather the large spatial and structural variability of the tumors in the BRATS dataset, as well as the variability in image intensities arising from differences in imaging equipment and acquisition protocols. Although we were able to identify several overall "winning" algorithmic properties (discussed in Section V-C), one general limitation of image analysis benchmarks is that it is often difficult to explain why a particular algorithm does well or-even more difficult-why it does not do well. This is because even the best algorithmic pipeline will fail if just one element is badly parameterized or implemented. Detecting such failures would require a meticulous study of each element of every processing pipeline-for a learning-based approach, for example, of the intensity normalization, the feature extraction, the classification algorithm, and the spatial regularization.
F. Lessons Learned There are lessons that we learned from organizing BRATS 2012 and 2013 that may also be relevant for future benchmark organizers confronted with complex and expensive annotation tasks. First, it may be recommended to generate multiple annotations for the test datarather than for the training set as we did here-as this is where the comparisons between experts and algorithms take place. Many algorithms will be able to overcome slight inconsistencies or errors in the training data that are present when only a single rater labels each case. At the same time, most algorithms will benefit from having larger training datasets and, hence, can be improved by annotating larger amounts of data even if this comes at the price of fewer annotations per image volume. Second, while it may be useful to make unprocessed data available as well, we strongly recommend providing participants with maximally homogenized datasets-i.
G. Future Work Given that many of the algorithms that participated in this study offered good glioma segmentation quality, it would seem valuable to have their software implementations more easily accessible. Right now, only an implementation of Bauer and Meier's method is freely available, 11 and Tustison's code 12 The online MIDAS and VSD platforms that we used for BRATS may be extended to not only host and distribute data, but also to host and distribute such algorithms. Making the top algorithms available through appropriate infrastructures and interfaces-for example as developed for the VISCERAL benchmark 13  [86] , or as used in the commercial NITRC Amazon cloud service 14 -may help to make thoroughly benchmarked algorithms available to the wider clinical research community. Since our results indicate that current automated glioma segmentation methods only reach the level of consensus-rater variation in the "whole" tumor case (Fig.
8 ), continued algorithmic development seems warranted. Other tumor substructures may also be relevant with respect to diagnosis and prognosis, and a more refined tumor model-with more than the four classes used in this study-may be helpful, in particular when additional image modalities are integrated into the evaluation. Finally, in clinical routine the change of tumor structures over time is often of primary relevance, something the current BRATS study did not address. Evaluating the accuracy of automated routines in longitudinal settings including 10www. virtualskeleton. ch/→BRATS 2013→"BRATS 2013 Individual Observer Ground-truth Data" 11www. nitrc. org/projects/bratumia [85]  12github. com/ntustison/BRATS2013 13www. visceral. eu 14www. nitrc. org/ce-marketplace both pre-and postoperative images, are important directions for future work along with further algorithmic developments.
VI. Summary and Conclusion In this paper we presented the BRATS brain tumor segmentation benchmark. We generated the largest public dataset available for this task and evaluated a large number of state-of-theart brain tumor segmentation methods. Our results indicate that, while brain tumor segmentation is difficult even for human raters, currently available algorithms can reach Dice scores of over 80% for whole tumor segmentation. Segmenting the tumor core region, and especially the active core region in high-grade gliomas, proved more challenging, with Dice scores reaching 70% and 60%, respectively. Of the algorithms tested, no single method performed best for all tumor regions considered. However, the errors of the best algorithms for each individual region fell within human inter-rater variability.
Training and Testing The classifier was trained using 5-fold cross-validation on the training dataset, with separate training for high-and low-grade as well as synthetic and patient data. The parameters of the algorithm were chosen empirically. We also compared the proposed approach to our previous method  [43]  which used SVMs as a classifier instead of decision forests and which had a less sophisticated regularization. With the new method, the computation time could be reduced by more than a factor of two and the accuracy was significantly improved. However, we still discovered difficulties with datasets that were very different from the training data, which hints at some problems of the supervised algorithm with generalization.
Segmentation of Brain Tissues Algorithm and Data We describe a fully automated approach inspired by the human brain labelling method described in  [90] , and similar to multi-atlas label propagation methods  [91] -  [93]. A database of multi-channel local patches is first built from a set of training pathological cases. Then, given a test case, similar multi-channel patches are retrieved in the patch database, along with the corresponding labels. Finally, a classification map for the test case is inferred as a combination of the retrieved labels during a label fusion step  [94] -  [97]. To decrease the computation time, images are sub-sampled to 2-mm isotropic resolution  [98]. A candidate tumor mask is defined by thresholding (50% percentile) a denoised  [99]  T 2 -weighted FLAIR image. Since the patch retrieval is driven by a sum-of-squareddifferences (SSD), a global intensity alignment  [98]  is applied to the mean image intensity restricted to the candidate tumor mask.
Training images are cropped along the Z-axis to focus on training patches surrounding the tumor. Image features are the concatenation of 3 × 3 × 3 intensity patches extracted from four MR channels. Given a multi-channel patch query, the five nearest-neighbor patches are retrieved within each training patch database, each of which contributes to a weighted voting. We use exponential weights, based on patch similarity  [90] ,  [95] ,  [97] ; the decay parameter σ 2 depends on the test case, and is set to the maximum of SSD between every voxel in the test case and every first-neighbor training patch. For each label, the weighted voting results in a probability-like map. Since the label regions are interlocked, label maps are hierarchically computed: first, complete tumor is distinguished from healthy tissues; then tumor core from edema; finally enhancing tumor from the rest of the core.
Training and Testing The most important parameters are manually set and consist of the patch size and the number of training cases. A range of values for the number of retrieved nearest-neighbor patches were tested, and the segmentation results were almost not affected. For the training data, the labelling is performed in a leave-one-out scheme, while for the test data, every relevant training case is used. Real cases are processed separately from simulated cases. For real low-grade test cases, the training dataset includes both high-and low-grade cases, while for real high-grade test cases, the training dataset only includes high-grade cases. The algorithm shows a few shortcomings which would require the following steps to be refined. • The necrotic core is sometimes partially missed by the candidate tumor mask.
Training and Testing The algorithm was tested on real-patient data from the BRATS 2012 and 2013 dataset. No training was performed; the initial labeling was random, and all model parameters were estimated iteratively. The proposed algorithm is fully automated and uses all available MRI sequences. Three preprocessing steps were performed. The first aims for the bias field correction, with N4ITK method  [103]. The second normalizes the intensity scale of each sequence to a chosen reference, by histogram matching using ITK  [104]. Finally, since some FLAIR images were already limited to the volume of interest, all sequences from each subject were cropped to have the same brain volume. A random decision forest is used to classify each brain voxel, based on several features extracted from the training data.
Training and Testing Three datasets were available: "Training" (with corresponding ground truth), "LeaderBoard" and "Challenge. " The training step was done using all real data from the Training dataset, from both grades to increase the representation of some labels (like non-enhancing). The testing step was performed with all datasets. Leave-one-out cross-validation was used for the Training dataset. The features set, as well as the hyperparameters for the decision forest, were found using leave-one-out cross-validation of the Training dataset. To segment highgrade tumors, all images (used in training and testing stages) were normalized to a highgrade reference. Similarly, images were normalized to a low-grade reference when segmenting these tumors. The critical part of the proposed algorithm is the normalization, which influences the whole pipeline, especially with intensity related features used in a supervised classifier.
Segmentation in Multi-Channel MR Images Medical imaging protocols produce large amounts of multi-modal volumetric images. The large size of the produced datasets contributes to the success of machine learning methods. These methods automatically learn from the data how to perform challenging task such as, for instance, semantic annotation. Although being a tremendous asset in theory, very large datasets are down-sampled to ensure tractability of the learning process. Moreover, the informative data is often submerged in overwhelming amounts of redundant data. Thus, most state of the art methods need to parse large amounts of uninformative data before reaching valuable data. We present the "Spatially Adaptive Random Forests" (SARFs)  [42]  to overcome these issues in the context of volumetric medical images.
Guo, Schwartz and Zhao (2013): Semi-automatic Segmentation of Multimodal Brain Tumor Using Active Contours In this paper, we present a semi-automatic segmentation method for multimodal brain tumors. It requires only that a user manually draw a region of interest (ROI) roughly surrounding the tumor on a single image. The algorithm combines the image analysis techniques of region and edge-based active con-tours and level set approach, and has the advantages of easy initialization, quick segmentation, and efficient modification. The typical run-time for each case in the training dataset can be within 1 min.
For the normal classes (that are not available from the manual annotation of the challenge dataset) we infer the maximum a posterior estimates of the generative model and use them as label during training. Random forests learn many decision trees from bootstrapped samples of the training data, and at each split in the tree they only evaluate a random subspaces to find the best split. To minimize correlation in the training data, and also to speed up training, we draw no more 2000 samples from each of the ≈ 10 6 voxels in each of the 25 dataset. We train an ensemble with 300 randomized decision trees, and choose a subspace dimensionality of 10. We use the random forest implementation from Breiman and Cutler. To improve segmentation, we use a Markov random field (MRF) imposing a smoothness constraint on the class labels.
Reza and Iftekharuddin (2013): Multi-Class Abnormal Brain Tissue Segmentation Using Texture Features Algorithm and Data In this work, we propose fully automated multi-class abnormal brain tissue segmentation in multimodality brain MRI. Fig. 11  shows a generic flow diagram for our algorithm pipeline. Since BRATS-2013 dataset is already skull stripped and co-registered; the first step involves preprocessing of 2D MRI slices extracted from 3D volume for each patient. Intensity normalization and inhomogeneity correction are used as pre-processing steps. Then two primary sets of features are extracted from each preprocessed image. The first set includes non-local features such as pixel intensities (I T1 , I T2 , I FL , I T1c ) and differences of intensities (d 1 = I T1 -I T2 , d 2 = I T2 -I FL , d 3 = I FL -I T1c ) that represents global characteristics of brain tissues.
Shin (2012): Hybrid Clustering and Logistic Regression for Multi-Modal Brain Tumor Segmentation Unsupervised learning approaches have potential for applications in medical image processing, as previously discussed  [111] -  [113]. Additionally, the approach can be extended readily to a previously unseen dataset avoiding the issues of overfitting that can occur in supervised learning methods, where overfitting has a larger influence in tumor segmentation when tumors have very heterogeneous characteristics. Unsupervised learning approaches were applied 1) in  [112]  for the previous two-class segmentation challenge, and 2) in  [113]  to detect multiple organs from a dataset where a few roughly labeled samples were available. These methods however were not directly applicable when the format of this challenge was changed to classify four-class labels.
The four-class segmentation problem was therefore approached with a supervised learning algorithm, used previously in  [112] , to segment the tumor-cores, trained with logistic regression. Four-dimensional patches (3 × 3 × 3 volume-patch ×4 channels) were used with second-order polynomial features as described in  [112] , as opposed to the three-dimensional patches (mps l × mps l 2-D image-patch ×T temporal-dimension) used previously in  [113]  to identify organs (but not for segmentation). This was because the dataset for this challenge was carefully registered with little motion, compared to the abdominal scans in  [113]  where the registration over the 40 volumes along the time-course was difficult as the region is usually affected by breathing motion. Deep neural networks with up to six layers were tried as well, pre-training the hidden-layers with stacked-autoencoder feature learning and subsequently fine-tuning them with the labeled samples in the training dataset.
Neural network model was not used for the challenge however, because the improvement of classification accuracy was small (< ~0. 05) relatively to the higher complexity compared to the logistic regression model. Each channel of volumes was normalized separately, to try to learn the relation between the multi-channel intensity values, and to avoid any biases in the image intensities in different scans. The same type of classifier was used to classify all labels including the not-of-interest label (label:0), where they were trained only on the patient-dataset which has four-class labels, and applied to synthetic data which has only two labels. Two cross-validations were performed for the parameter adaptation, and no additional post-processing steps were applied to the patch-wise classification.
The second inference stage is shown to resolve local inhomogeneities and impose a smoothing constraint, while maintaining appropriate boundaries as supported by local intensity difference observations. The method was trained and tested on the updated MICCAI 2012 BRATS Database, which included four tumor subclasses: necrotic core, edema, solid tumor, and enhanced tumor. The algorithm was trained and tested on clinical volumes, including low-grade and high-grade tumors. Classifiers were built separately for all categories. No other datasets were used for training or tuning. Online segmentation statistics (e. g. , Dice overlap metrics) were provided. For training cases, the method was tested in a leave-one-out fashion. After training, the algorithm was tested on all test cases. On I7 Dell Optiplex machines, the training took a day, due to both convolution and simulated annealing algorithms used.
Segmentation and Results Segmenting with the HMM (Fig. 3 ) involves extracting the feature vector for each voxel in the target case in the same manner as HMM training. Voxels from FLAIR, T1, T1C, and T2 in a neighborhood around the voxel of interest are organized into the feature vector and provided to the trained HMM model. The HMM model produces a predicted label for the feature vector. Postprocessing involved filtering out small objects and applying dilation and erosion operations on each segmented class. Our method has been evaluated on the BRATS2013 challenge dataset for high-grade glioma cases. We achieve an mean accuracy (Dice score) of [59. 5]% for edema and [65. 6]% for tumor in the real cases. The Map-Reduce enabled HMM is able to train on all cases simultaneously, performing 220% faster on an 8node cluster than on a single node.