dataesr/scientific-paragraphs-categorization huggingface dataset:
MRI data acquisition and preprocessing. MRI data were collected on a 3 T Siemens Trio with a 32-channel head coil. Functional data were acquired using gradient-echo echoplanar imaging (EPI) (3 mm isotropic voxels, 64 × 64 matrix, 42 axial slices tilted 30°from the AC-PC plane, TE = 25 ms, flip angle = 75°, TR = 2500 ms). There were 4 runs with 226 images per run. T1-weighted MPRAGE structural images (0. 9375 × 0. 9375 × 1 mm voxels, 192 × 256 matrix, 160 axial slices, TI = 1100 ms, TE = 3. 11 ms, flip angle = 15°, TR = 1630 ms) and matched fieldmap images (TE = 2. 69 and 5. 27 ms, flip angle = 60°, TR = 1000 ms) were also collected. Data were preprocessed with FSL  37, 38  and AFNI  39, 40. Functional data were corrected for slice timing (AFNI's 3dTshift) and head motion (FSL's MCFLIRT), attenuated for outliers (AFNI's 3dDespike), undistorted and warped to MNI space (FSL's FLIRT and FNIRT), smoothed with 6 mm FWHM Gaussian kernel (FSL's fslmaths) and intensity scaled by the grand-mean value per run.
However, new concerns related to the learning process have been raised, such as the need to build large volumes of manually segmented data, time and energy required for learning, and risk of over-fitting. To prevent these problems, researchers are exploring the possibilities offered by multi-agent systems (MASs). A MAS is composed of a population of software entities called agents, which are situated in an environment and capable of making autonomous decisions in this environment, to meet their local objectives and the global purpose of the system  [10]. MASs provide a distributed and collective decision process that allows segmentation of medical images with flexibility, autonomy, and robustness. Thus, they have been used for medical image segmentation  [11]  during the prior decades. However, these approaches still suffer from several drawbacks, such as the excessive number of parameters and the need for user interaction.
They were initially introduced in  [29]. Although they have been very successful, different improvements and combinations with other methods have been proposed  [30, 31]. Machine learning techniques are also used for classifying medical images as part of segmentation using a dataset of manually labeled images. Machine learning approaches are first trained on these manually labeled data, and segmentation is performed using the trained model. Many machine learning-based medical image segmentation approaches have been proposed in the literature, such as decision tree  [32] , support vector machine  [33] , random forest  [34] , and more recently, deep learning methods  [35]. Similar to machine learning approaches, medical image segmentation techniques using manually annotated data offer outstanding results particularly when deep learning methods are used.
However, several challenges hamper their democratization, such as the lack of manually labeled medical images, class imbalance inside training datasets, and overfitting  [35]. Despite the advances made in the last few years within the medical imaging research community, correct identification and segmentation of all tissues present in a medical image remains a complex task, and its automation remains a challenge. Rather than attempting to segment all the tissues present in a medical image, researchers have focused on identifying and isolating pathological tissues (e. g. , detection of Alzheimer lesions or tumors), providing valuable information for medical staff. Tumor segmentation is one of the most active areas in the medical image segmentation field, and many studies have been published on tumor segmentation in MR brain images.
Some of those approaches are reviewed in  [37] [38] [39] [40] [41]. Another type of approach relies on machine learning. Instead of using different algorithms to isolate the tumor from the rest of the image, machine learning techniques use training datasets to generate models based on pattern matching and similarities. Those techniques have generated considerable interest in medical image segmentation and tumor segmentation in brain MRI. Several machine learning approaches have been proposed and can be classified into two main categories: generative models and discriminative models. The latter have emerged from democratization of deep learning and convolutional neuronal networks (CNN)  [42]. Unlike traditional classification methods, where images with manually tagged features are used for training, CNNs automatically learn complex representative features directly from the data themselves.
Research on CNN-based brain tumor segmentation mainly focuses on network architecture design rather than image processing to extract features  [36]. This focus is motivated by the results obtained with these methods using available segmentation benchmarks. As an example, during the last Brain Tumor Segmentation (BraTS) challenge  [43] , a well-known dataset and challenge for MRI brain tumors segmentation, first place was exclusively won using deep learning-based approaches. Despite their effectiveness in dealing with benchmarks, deep learning-based approaches suffer from the following limitations  [44] : • Lack of training datasets: Training deep networks with imbalanced data can produce biased models  [44]. Thus, with the brain tumor segmentation (BraTS) dataset, where all provided images (training, validation, and testing) contain tumors, deep learning-based approaches offer substantial results for tumor segmentation.
Nevertheless, these results do not ensure the accuracy of deep learning approaches when dealing with a healthy brain or tissues affected by a disorder unrelated to a tumor. • Lack of appropriately tagged data: Deep learning approaches are designed to learn from a vast amount of training data, while the set of available manually segmented medical images is limited. • Overfitting: An excessive training phase applied to a highly specific dataset may lead to a model that is suitable to these data but cannot be generalized to other data. Most of the approaches listed above employ a monolithic, sequential, centralized process to perform tumor segmentation  [45]. Furthermore, they use a unidirectional execution, where backtracking is impossible during the segmentation process. Thus, these approaches ignore other segmentation choices that can be more significant than the considered one.
Experimental results on 2D MR phantom images The first series of experiments was performed on brain MRI 2D phantom image slices extracted from 3D brain MR images. Those images were generated from the BrainWeb platform, a phantom database provided by the McConnell Brain Imaging Center at Montreal Neurological Institute (https://brainweb. bic. mni. mcgill. ca/brainweb/) [69]. According to Collins  [69] , simulated phantom images are an effective tool for validating the results of a segmentation algorithm. They also permit a comparison with results obtained from other published methods. Moreover, we chose the simulated images generated by BrainWeb because this tool allows the efficiency and robustness of MAMES to be evaluated despite the presence of image artifacts, such as noise and INU, in a quantitative evaluation.
Experimental results on 3D MR tumoral images One of the main problems associated with the design of image segmentation methods is their evaluation process. This evaluation is generally based on comparing the experimental results with reference results produced manually by a human expert. However, achieving a set of manually segmented images (ideally done by several experts) is not an easy task, and it is very time-consuming (several hours to segment a single 3D MR image). Existing works often use small datasets produced by radiologists to evaluate their approaches. This procedure complicates the evaluation process as no shared data is available to compare segmentation methods. Thus, the BraTS challenge began in conjunction with the International Conference on Medical Image Computing (2012) to address the brain tumor segmentation issue.
The BraTS challenge  [70]  offers a useful environment for evaluating automatic segmentation approaches of tumor extraction in multi-modal 3D MR images. The BraTS dataset (BraTS2018) is composed only of pre-operative images acquired in four modalities (T1, T2, FLAIR, and T1 with contrast enhancement). It contains high-grade glioblastoma (HGG) and lower-grade glioma (LGG) manually segmented by experts. In the BraTS dataset, voxels in a reference image were divided into three different classes (labels), the enhanced tumor (label 4), peritumoral edema (label 2), and necrotic tumor core (label 1). A fourth label (label 3) was initially used to define non-enhancing tumor core, but it was eliminated afterward. This annotation was made by several domain experts and validated by experienced neuro-radiologists.
Therefore, the MR images used in BraTS were acquired in different institutions with different equipment and protocols, making this dataset heterogeneous in terms of MRI rendering and quality (same tissue type can have different gray level intensities from one acquisition device to another). The BraTS dataset takes the form of a set of folders, one folder for each human subject. Inside a folder, a multi-modal image is stored in five different files (four files for the MR image modalities and a file for the labeled reference). To conduct our experiments, we used eight 3D images from the BraTS dataset. Considering that MAMES is designed for single modality images, we focus on the FLAIR modality as it offers desirable contrast between the tumoral area and the healthy tissues. The obtained tumor segmentation results are detailed in Table  2  and Fig.
8. In the latter and in addition to the Dice coefficient, we include specificity (TNR), accuracy (ACC), precision (PPV), and sensitivity (TPR) as common metrics used for evaluating tumor segmentation. TNR = TN TN + FP (7) The quantitative evaluation presented in Table  2  and Fig. 8  illustrates the effectiveness of our approach for tumor segmentation in 3D MRI. The results of tumor segmentation exceed 80% for all eight volumes and an average of 86% Dice coefficient. These results are similar to those obtained by a segmentation expert when compared to the consensus obtained during development of the BraTS dataset  [70]. Tables  3  and 4  show results obtained by MAMES for segmentation of LGG and HGG tumors, respectively, in the axial, coronal, and sagittal planes. The first line shows different slices for the T1 modality, while the second line shows different slices for the FLAIR modality.
The tumors are composed of tissues (i. e. , edema, necrotic core, and active tumor) that can have different intensities inside an MR image. Differentiation of those textures requires the use of multiple modalities (T1, T2, FLAIR, and T1e). Because our approach only uses the FLAIR modality to segment a tumor, resulting in these small undetected areas, this issue can be overcome by using some mathematical morphology operations. A comparative evaluation of these results is presented in Table  5  and Fig. 9. From the latter, we observe that our results are satisfactory when compared to some state-of-the-art approaches using the BraTS dataset  [42]. We also highlight the fact that our approach uses only one modality (FLAIR) when most of the other presented methods use the four modalities provided in the BraTS dataset.
Our current implementation uses pseudo-parallelism during execution, which explains the Table  4  HGG segmentation results in the axial, coronal, and sagittal planes. segmentation duration. Therefore, one direction of future research is implementation of a physically distributed version of MAMES, allowing the processing time to be decreased. Experimental results on the BraTS database have shown the efficiency of MAMES for accurate segmentation of a brain tumor into its main parts, namely the entire tumor and the edema. The Dice coefficient results show that the proposed method provides segmentation accuracy that is close to the accuracy found with other cited methods, despite the fact that most of those methods are learning-based, and in some cases, require a massive training dataset, such as for deep learning-based methods.
MAMES was used to segment several brain MR images. The segmentation experiments validated its implementation and targeted characteristics performance, such as the segmentation quality. With only one modality, our approach offers similar and, in some cases, more accurate results than other approaches that use the four modalities provided by the BraTS dataset. MAMES is a promising approach owing to the use of well-known segmentation techniques and a multi-agent architecture. One of our future research directions is to modify MAMES to consider several modalities in the same image simultaneously. We believe that those modalities would provide higher segmentation accuracy by producing a clearer distinction between different tissues composing the tumor. We aim to focus on using our approach to segment other anatomical structures using other types of medical images (e.
We use the multilingual OSCAR corpus, extracted from Common Crawl via language classification, filtering and cleaning, to train monolingual contextualized word embeddings (ELMo) for five mid-resource languages. We then compare the performance of OSCARbased and Wikipedia-based ELMo embeddings for these languages on the part-ofspeech tagging and parsing tasks. We show that, despite the noise in the Common-Crawlbased OSCAR data, embeddings trained on OSCAR perform much better than monolingual embeddings trained on Wikipedia. They actually equal or improve the current state of the art in tagging and parsing for all five languages. In particular, they also improve over multilingual Wikipedia-based contextual embeddings (multilingual BERT), which almost always constitutes the previous state of the art, thereby showing that the benefit of a larger, more diverse corpus surpasses the crosslingual benefit of multilingual embedding architectures.
, 2019). All of them have repeatedly improved the state-of-the art in many downstream NLP tasks over the last year. In general, the main advantage of using language models is that they are mostly built in an unsupervised manner and they can be trained with raw, unannotated plain text. Their main drawback is that enormous quantities of data seem to be required to properly train them especially in the case of contextualized models, for which larger corpora are thought to be needed to properly address polysemy and cover the wide range of uses that com-monly exist within languages. For gathering data in a wide range of languages, Wikipedia is a commonly used option. It has been used to train fixed embeddings  (Al-Rfou et al. , 2013; Bojanowski et al. , 2017)  and more recently the multilingual BERT  (Devlin et al.
, 2018) , hereafter mBERT. However, for some languages, Wikipedia might not be large enough to train good quality contextualized word embeddings. Moreover, Wikipedia data all belong to the same specific genre and style. To address this problem, one can resort to crawled text from the internet; the largest and most widespread dataset of crawled text being Common Crawl. 1 Such an approach generally solves the quantity and genre/style coverage problems but might introduce noise in the data, an issue which has earned the corpus some criticism, most notably by Trinh and Le (2018) and  Radford et al. (2019). Using Common Crawl also leads to data management challenges as the corpus is distributed in the form of a large set of plain text each containing a large quantity of unclassified multilingual documents from different websites.
In this paper we study the trade-off between quantity and quality of data for training contextualized representations. To this end, we use the OSCAR corpus  (Ortiz Suárez et al. , 2019) , a freely available 2 multilingual dataset obtained by performing language classification, filtering and cleaning of the whole Common Crawl corpus. 3  OSCAR was created following the approach of  Grave et al. (2018)  but proposing a simple improvement on their filtering method. We then train OSCAR-based and Wikipedia-based ELMo contextualized word embeddings  (Peters et al. , 2018)  for 5 languages: Bulgarian, Catalan, Danish, Finnish and Indonesian. We evaluate the models by attaching them to the to UDPipe 2. 0 architecture  (Straka, 2018; Straka et al. , 2019)  for dependency parsing and part-of-speech (POS) tagging.
Corpora We train ELMo contextualized word embeddings for 5 languages: Bulgarian, Catalan, Danish, Finnish and Indonesian. We train one set of embeddings using only Wikipedia data, and another set using only Common-Crawl-based OS-CAR data. We chose these languages primarily because they are morphologically and typologically different from one another, but also because all of the OSCAR datasets for these languages were of a sufficiently manageable size such that the ELMo pre-training was doable in less than one month. Contrary to HIT-SCIR team  (Che et al. , 2018) , we do not impose any cap on the amount of data, and instead use the entirety of Wikipedia or OSCAR for each of our 5 chosen languages. (Wu and Weld, 2010; Mihalcea, 2007; Al-Rfou et al. , 2013; Bojanowski et al. , 2017). We downloaded the XML Wikipedia dumps 7  and extracted the plain-text from them using the wikiextractor.
OSCAR Common Crawl is a non-profit organization that produces and maintains an open, freely available repository of crawled data from the web. Common Crawl's complete archive consists of petabytes of monthly snapshots collected since 2011. Common Crawl snapshots are not classified by language, and contain a certain level of noise (e. g. one-word "sentences" such as "OK" and "Cancel" are unsurprisingly very frequent). This is what motivated the creation of the freely available multilingual OSCAR corpus  (Ortiz Suárez et al. , 2019) , extracted from the November 2018 snapshot, which amounts to more than 20 terabytes of plain-text. In order to create OSCAR from this Common Crawl snapshot, Ortiz Suárez et al. (  2019 ) reproduced the pipeline proposed by  (Grave et al. , 2018)    language classification was performed using the fastText linear classifier  (Joulin et al.
Contextualized word embeddings Embeddings from Language Models (ELMo)  (Peters et al. , 2018 ) is an LSTM-based language model. More precisely, it uses a bidirectional language model, which combines a forward and a backward LSTM-based language model. ELMo also computes a context-independent token representation via a CNN over characters. We train ELMo models for Bulgarian, Catalan, Danish, Finnish and Indonesian using the OS-CAR corpora on the one hand and the Wikipedia corpora on the other. We train each model for 10 epochs, as was done for the original English ELMo  (Peters et al. , 2018). We save checkpoints at 1 st , 3 rd and 5 th epoch in order to investigate some concerns about possible overfitting for smaller corpora (Wikipedia in this case) raised by the original ELMo authors. 12 4.
Parsing and POS tagging results We use UDPipe 2. 0 without contextualized embeddings as our baseline for POS tagging and dependency parsing. However, we did not train the model without contextualized word embedding ourselves. We instead take the scores as they are reported in  (Kondratyuk and Straka, 2019). We also compare our UDPipe 2. 0 + ELMo models against the state-of-the-art results (assuming gold tokenization) for these languages, which are either UDify  (Kondratyuk and Straka, 2019)  or UDPipe 2. 0 + mBERT  (Straka et al. , 2019). Results for UPOS, UAS and LAS are shown in Table  5. We obtain the state of the art for the three metrics in each of the languages with the UDPipe Table  5 : Scores from UDPipe 2. 0 (from  Kondratyuk and Straka, 2019) , the previous stateof-the-art models UDPipe 2.
0 + ELMo Wikipedia results considerably improve the UDPipe 2. 0 baseline. This is the case for all of our ELMo Wikipedia models as we never see any evidence of a negative impact when we add them to the baseline model. In fact, the results of UDPipe 2. 0 + ELMo Wikipedia give better than previous state-of-the-art results in all metrics for the Finnish-FTB and in UPOS for the Finnish-TDT. The results for Finnish are actually quite interesting, as mBERT was pre-trained on Wikipedia and here we see that the multilingual setting in which UDify was fine-tuned exhibits sub-baseline results for all metrics, and that the UDPipe + mBERT scores are often lower than those of our UDPipe 2. 0 + ELMo Wikipedia. This actually suggests that even though the multilingual approach of mBERT (in pre-training) or UDify (in pre-training and fine-tuning) leads to better performance for high-resource languages or languages that are closely related to high-resource languages, it might also significantly degrade the representations for more isolated or even simply more morphologically rich languages like Finnish.
Here we actually see an unexpected behavior where the model performance caps around the 3 rd to 5 th epoch. This is surprising because the Finnish OS-CAR corpus is more than 20 times bigger than our smallest Wikipedia corpus, the Danish Wikipedia, that did not exhibit this behavior. As previously mentioned Finnish is morphologically richer than the other languages in which we trained ELMo, we hypothesize that the representation space given by the ELMo embeddings might not be sufficiently big to extract more features from the Finnish OS-CAR corpus beyond the 5 th epoch mark, however in order to test this we would need to train a larger language model like BERT which is sadly beyond our computing infrastructure limits (cf. Subsection 5. 3). However we do note that pre-training our current language model architectures in a morphologically rich language like Finnish might actually better expose the limits of our existing approaches to language modeling.
However we believe that ELMo contextualized word embeddings remain a useful model that still provide an extremely good trade-off between performance 20 https://www. rte-france. com/fr/eco2mix/eco2mix-co2 21 https://github. com/google-research/bert to training cost, even setting new state-of-the-art scores in parsing and POS tagging for our five chosen languages, performing even better than the multilingual mBERT model.
Acknowledgments We want to thank  Ganesh Jawahar  for his insightful comments and suggestions during the early stages of this project. This work was partly funded by the  French national ANR  grant  BASNUM  ( ANR-18-CE38-0003 ), as well as by the last author's chair in the PRAIRIE institute, 22 funded by the  French national ANR as part of the "Investissements d'avenir" programme  under the reference  ANR-19-P3IA-0001. The authors are grateful.
Participants Twenty-seven Caucasian individuals, mainly undergraduate students from the Universit e Catholique de Louvain, participated in the experiment in total. Sixteen participants completed a first version of the experiment, and 11 participants completed a slightly modified version of the experiment (see below, section 'Stimuli') a few months later. Among these 11 participants, one of them also participated in the first version of the experiment. We removed data of one individual from the first version of the experiment after testing because of too much noise/muscular artefacts in the EEG data. There was no age difference between the two groups of participants, t (24) ¼. 75, p >. 1. The final sample consisted of 26 participants (10 males; mean age, 22 ± 2. 13 years). All participants were right-handed by self-report and had normal or corrected-tonormal vision.