dataesr/scientific-paragraphs-categorization huggingface dataset:
3% of those 3,000 selected reads (i. e. , when setting a minimum reference mapping depth of 100 reads in LORCAN, which corresponds to the minimum number of reads recommended for reliable consensus creation [see Fig. S7] ). In most cases, however, even when a sample may consist of amplicons derived from a unique species, not all reads are assigned to the target species (e. g. , due FIG  3  Influence of reference database completeness on consensus sequence accuracy. Each consensus sequence was compared to a consensus sequence produced with a perfectly matching reference sequence. Additionally, each consensus sequence was identified by BLAST similarity search against the full reference database. The uneven spacing of the data points reflects the database composition after subsetting. Missing values are a result of insufficient numbers of reads mapping to the reference database.
(A) Filled circles indicate correct taxonomic identification of the ATCC strains. The low identities and unsuccessful identification of Eggerthella lenta are a result of a low-level contamination in combination with unsuccessful mapping of the Eggerthella reads. (B) The diameter of the circles is proportional to the number of reads mapped and further used in the consensus generation step (obtained from the LORCAN output). Additional detail is provided in Table  S3  and Fig. S10  in the supplemental material. to read errors and/or the presence of highly similar sequences associated with other species in the reference database). Furthermore, demultiplexing and size selection could result in significant reduction of available reads. For illustrative purposes, during our last 11 sequencing runs consisting of 89 samples (including place-holder samples; see Library preparation in Materials and Methods), an average of 639,944 Ϯ 267,704 basecalled reads were produced, while multiplexing on average 8 Ϯ 3 barcoded samples per sequencing run.
With these parameters being set this way, accurate consensus sequences (Ն99% identity to Sanger sequences produced from the same DNA) were reliably produced with as few as 100 size-filtered reads per sample (see Fig. S7  in the supplemental material), confirming previous findings  (33). Applicability to samples consisting of mixed amplicons was a key requirement during development of LORCAN, as contaminations are not rare in bacterial cultures derived from clinical samples. To exclude sources of variation due to fluctuations in wet laboratory processes, we analyzed artificially mixed amplicons based on pure reads generated from pure amplicons. LORCAN showed high robustness against such mixture events and was capable of quantitatively representing read compositions in mixed samples as long as the analyzed gene region and the used database provide the required taxonomic resolution.
Most computational workflows either include reference-based consensus generation or de novo assembly in combination with additional error correction steps. They were reported to perform similarly in terms of the accuracy of the produced sequences  (12, 13, 15, 17, 33). De novo approaches are preferable when reference sequences are missing; however, so far, the only studies demonstrating "reference-free" consensus generation from complex samples (e. g. , mock communities) relied on rather laborious wet lab procedures, such as rolling cycle amplification or unique tagging of the individual amplicons before sequencing  (15, 16). Unlike previous studies, we specifically designed our workflow for clinical routine applications. Compatibility with mixed samples and time/cost efficacy were therefore key requirements, and comprehensive reference databases were readily available.
Finally, even when the reference sequence database is incomplete or lacks closely related reference sequences, we showed that the approach is robust and provides correct taxonomic identification of the bacterial species. Our approach has several limitations. (i) The taxonomic resolution is inherently limited by the choice of a single-gene-based approach. Commonly used 16S rRNA gene regions, for example, have been reported to allow for genus identification in Ͼ90% of cases, for species identification in 65% to 83% of cases and to result in unsuccessful identification in 1% to 14% of all analyzed isolates  (8, 34, 35). Other approaches, such as matrix-assisted laser desorption ionization-time of flight (MALDI-TOF) mass spectrometry may complementarily provide fast and reliable identification of clinically relevant microorganisms  (36).
Yet, MALDI-TOF may also suffer from suboptimal identification due to limitations, including insufficient representation of reference species profiles in available commercial databases, absence of newly discovered species, and the existence of several commercial systems  (37) (38) (39). (ii) The dependency on database quality and completeness in the LORCAN reference-based approach for consensus building was explored extensively by using modified databases, which lacked reference sequences closely related to the analyzed strains. Not surprisingly, consensus accuracy was strongly affected, and LORCAN required reference databases of high quality and completeness to reliably reach sequence qualities on par with the quality obtained by the Sanger method. Even if databases contained sequences with up to 99% identity to the analyzed species, further improvements could often be made by adding closer reference sequences (Fig.
3 ). Importantly though for clinical diagnostics, taxonomic identification based on the produced consensus sequence was far less affected by database completeness. Even consensus sequences produced with distant reference sequences (Յ90% identity to the query sequence using an incomplete database) allowed for reliable bacterial species identification when the generated consensus was compared to a complete database. This finding indicates a high reliability of the taxonomic identification despite the database dependency of the approach. This was confirmed by extensive validation in our diagnostics department, which was based on the parallel sequencing and analysis of clinical samples using both Sanger and nanopore sequencing over several months, which overall showed average sequence identities of 99.
A literature review was performed for retrieving from PubMed papers indexed with keywords and free terms related to BTI. Independent review allowed the section editors to select a list of 15 candidate best papers which were subsequently peer-reviewed. A final consensus meeting gathering the whole Yearbook editorial committee was organized to finally decide on the selection of the best papers. Results: Among the 931 retrieved papers covering the various subareas of BTI, the review process selected four best papers. The first paper presents a logical modeling of cancer pathways. Using their tools, the authors are able to identify two known behaviours of tumors. The second paper describes a deep-learning approach to predicting resistance to antibiotics in Mycobacterium tuberculosis. The authors of the third paper introduce a Genomic Global Positioning System (GPS) enabling comparison of genomic data with other individuals or genomics databases while preserving privacy.
Introduction Within the 2020 International Medical Informatics Association (IMIA) Yearbook of Medical Informatics, the goal of the Bioinformatics and Translational Informatics (BTI) section is to provide an overview of research trends from 2019 publications that demonstrated excellent research about various aspects of bioinformatics methods and techniques to advance clinical care. In 2008, the American Medical Informatics Association (AMIA) has defined translational bioinformatics as "… the development of storage, analytic, and interpretive methods to optimize the transformation of increasingly voluminous biomedical data into proactive, predictive, preventative, and participatory health"  [1]. The first priorities addressed storage, retrieval, and focused analytics of high-throughput data that motivated numerous research and development studies in the last decade.
Paper Selection Method Following the method described in  [2] , a comprehensive review of articles published in 2019 and addressing various subtopics for BTI was conducted. The selection was performed by querying MEDLINE via PubMed (from NCBI, National Center for Biotechnology Information, NLM, NIH) with a set of predefined MeSH descriptors along with free terms: "Translational informatics", "Translational bioinformatics", "Bioinformatics", "Computational molecular biology", "Computing Methodologies", "Information storage and retrieval", "Pattern recognition", "automated", "Medical informatics, Algorithms", "Translational medical Research", "Genetics, Medical", "Precision Medicine", "Personalized medicine", "Molecular Medicine", "Genomic medicine", "Medical genetics", "Medical genomics", "Clinical genomics", "Genetics", "Genomics", "Next-generation sequencing", "High throughput sequencing", "Transcriptome", "Transcriptomics", "Proteome", "Proteomics", "Proteogenomics", "Epigenomics", "Metabolomics", "Metagenomics", "Largescale datasets", "Big data", "Omics", and "Multi-omics".
Bibliographic databases were searched on January 20th, 2020 for papers published in 2019, considering the electronic publication date. The original set of 931 references was reviewed jointly by the two section editors to select a consensual list. Hence, 41 and 34 references were selected by MS, respectively BR, based on the title and the abstract of the papers. Six articles were in common to both selections, and 11 were accepted in one selection and pending in the other. Section editors agreed on 15 papers out of the 17 pre-selected references which were subsequently peer-reviewed by the IMIA Yearbook editors and external reviewers (at least four reviews per paper). Four papers were finally selected as best papers (Table  1 ). A content summary of these best papers can be found in the appendix of this synopsis.
Graim et al. ,  [10]  proposed PLATYPUS a machine learning approach to identify drug sensitivity signatures from cancer cell-lines databases. To overcome the problem of missing data and data sparsity both at the learning and prediction times, the authors developed an approach relying on a concept called multiple view learning, a semi-supervised method, and leveraged multi-omics data (expression, CNV, mutation, and also Sample-and Patient-specific information). Liu et al. ,  [14] , as mentioned before, also relied on cell line data but in combination with features extracted from SMILES representation of drugs. Esteban-Medina et al,. [8]  used a machine learning approach, combined with data from KEGG, Orphanet, GEO, GTEx and DrugBank to identify drugs that could have an effect on signaling the circuits that cause the treatment of the Fanconi anemia.
Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available models have either been trained on English data or on the concatenation of data in multiple languages. This makes practical use of such models-in all languages except English-very limited. In this paper, we investigate the feasibility of training monolingual Transformer-based language models for other languages, taking French as an example and evaluating our language models on part-of-speech tagging, dependency parsing, named entity recognition and natural language inference tasks. We show that the use of web crawled data is preferable to the use of Wikipedia data. More surprisingly, we show that a relatively small web crawled dataset (4GB) leads to results that are as good as those obtained using larger datasets (130+GB).
Although multilingual models give remarkable results, they are often larger, and their results, as we will observe for French, can lag behind their monolingual counterparts for high-resource languages. In order to reproduce and validate results that have so far only been obtained for English, we take advantage of the newly available multilingual corpora OSCAR  (Ortiz Suárez et al. , 2019)  to train a monolingual language model for French, dubbed CamemBERT. We also train alternative versions of CamemBERT on different smaller corpora with different levels of homogeneity in genre and style in order to assess the impact of these parameters on downstream task performance. CamemBERT uses the RoBERTa architecture  (Liu et al. , 2019) , an improved variant of the high-performing and widely used BERT architecture  (Devlin et al.
, 2019) , BERT  (Devlin et al. , 2019) , RoBERTa  (Liu et al. , 2019)  and more 1 Released at: https://camembert-model. fr recently ALBERT  (Lan et al. , 2019)  and T5  (Raffel et al. , 2019). Non-English contextualized models Following the success of large pretrained language models, they were extended to the multilingual setting with multilingual BERT (hereafter mBERT)  (Devlin et al. , 2018) , a single multilingual model for 104 different languages trained on Wikipedia data, and later XLM  (Lample and Conneau, 2019) , which significantly improved unsupervized machine translation. More recently XLM-R  (Conneau et al. , 2019) , extended XLM by training on 2. 5TB of data and outperformed previous scores on multilingual benchmarks. They show that multilingual models can obtain results competitive with monolingual models by leveraging higher quality data from other languages on specific downstream tasks.
A few non-English monolingual models have been released: ELMo models for Japanese, Portuguese, German and Basque 2  and BERT for Simplified and Traditional Chinese  (Devlin et al. , 2018)  and German  (Chan et al. , 2019). However, to the best of our knowledge, no particular effort has been made toward training models for languages other than English at a scale similar to the latest English models (e. g. RoBERTa trained on more than 100GB of data). BERT and RoBERTa Our approach is based on RoBERTa  (Liu et al. , 2019)  which itself is based on BERT  (Devlin et al. , 2019). BERT is a multi-layer bidirectional Transformer encoder trained with a masked language modeling (MLM) objective, inspired by the Cloze task  (Taylor, 1953). It comes in two sizes: the BERT BASE architecture and the BERT LARGE architecture.
(2012). The FTB contains more than 11 thousand entity mentions distributed among 7 different entity types. A brief overview of the FTB can also be found in Table  1. •••••••••••••••••••• FTB 350, Finally, we evaluate our model on NLI, using the French part of the XNLI dataset  (Conneau et al. , 2018). NLI consists in predicting whether a hypothesis sentence is entailed, neutral or contradicts a premise sentence. The XNLI dataset is the extension of the Multi-Genre NLI (MultiNLI) corpus  (Williams et al. , 2018)  to 15 languages by translating the validation and test sets manually into each of those languages. The English training set is machine translated for all languages other than English. The dataset is composed of 122k train, 2490 development and 5010 test examples for each language. As usual, NLI performance is evaluated using accuracy.
Training data Pretrained language models benefits from being trained on large datasets  (Devlin et al. , 2018; Liu et al. , 2019; Raffel et al. , 2019). We therefore use the French part of the OSCAR corpus  (Ortiz Suárez et al. , 2019) , a pre-filtered and pre-classified version of Common Crawl. 7 OSCAR is a set of monolingual corpora extracted from Common Crawl snapshots. It follows the same approach as  (Grave et al. , 2018)  by using a language classification model based on the fastText linear classifier  (Grave et al. , 2017; Joulin et al. , 2016)  pretrained on Wikipedia, Tatoeba and SETimes, which supports 176 languages. No other filtering is done. We use a non-shuffled version of the French data, which amounts to 138GB of raw text and 32. 7B tokens after subword tokenization.
Pre-processing We segment the input text data into subword units using SentencePiece  (Kudo and Richardson, 2018). SentencePiece is an extension of Byte-Pair encoding (BPE)  (Sennrich et al. , 2016)  and WordPiece  (Kudo, 2018)  that does not require pre-tokenization (at the word or token level), thus removing the need for language-specific tokenisers. We use a vocabulary size of 32k subword tokens. These subwords are learned on 10 7 sentences sampled randomly from the pretraining dataset. We do not use subword regularisation (i. e. sampling from multiple possible segmentations) for the sake of simplicity.
Language Modeling Transformer Similar to RoBERTa and BERT, CamemBERT is a multi-layer bidirectional Transformer  (Vaswani et al. , 2017). Given the widespread usage of Transformers, we do not describe them here and refer the reader to  (Vaswani et al. , 2017). CamemBERT uses the original architectures of BERT BASE (12 layers, 768 hidden dimensions, 12 attention heads, 110M parameters) and BERT LARGE (24 layers, 1024 hidden dimensions, 12 attention heads, 110M parameters). CamemBERT is very similar to RoBERTa, the main difference being the use of whole-word masking and the usage of SentencePiece tokenization  (Kudo and Richardson, 2018)  instead of WordPiece  (Schuster and Nakajima, 2012). Pretraining Objective We train our model on the Masked Language Modeling (MLM) task. Given an input text sequence composed of N tokens x 1 ,.
, x N , we select 15% of tokens for possible replacement. Among those selected tokens, 80% are replaced with the special <MASK> token, 10% are left unchanged and 10% are replaced by a random token. The model is then trained to predict the initial masked tokens using cross-entropy loss. Following the RoBERTa approach, we dynamically mask tokens instead of fixing them statically for the whole dataset during preprocessing. This improves variability and makes the model more robust when training for multiple epochs. Since we use SentencePiece to tokenize our corpus, the input tokens to the model are a mix of whole words and subwords. An upgraded version of BERT 8  and  Joshi et al. (2019)  have shown that masking whole words instead of individual subwords leads to improved performance. Whole-word Masking (WWM) makes the training task more difficult because the model has to predict a whole word rather than predicting only part of the word given the rest.
98) for 100k steps with large batch sizes of 8192 sequences, each sequence containing at most 512 tokens. We enforce each sequence to only contain complete paragraphs (which correspond to lines in the our pretraining dataset). Pretraining We use the RoBERTa implementation in the fairseq library  (Ott et al. , 2019). Our learning rate is warmed up for 10k steps up to a peak value of 0. 0007 instead of the original 0. 0001 given our large batch size, and then fades to zero with polynomial decay. Unless otherwise specified, our models use the BASE architecture, and are pretrained for 100k backpropagation steps on 256 Nvidia V100 GPUs (32GB each) for a day. We do not train our models for longer due to practical considerations, even though the performance still seemed to be increasing.
For dependency parsing, we plug a bi-affine graph predictor head as inspired by  Dozat and Manning (2017). We refer the reader to this article for more details on this module. We fine-tune on XNLI by adding a classification head composed of one hidden layer with a non-linearity and one linear projection layer, with input dropout for both. We fine-tune CamemBERT independently for each task and each dataset. We optimize the model using the Adam optimiser (Kingma and Ba, 2014) with a fixed learning rate. We run a grid search on a combination of learning rates and batch sizes. We select the best model on the validation set out of the 30 first epochs. For NLI we use the default hyperparameters provided by the authors of RoBERTa on the MNLI task. 9 Although this might have pushed the performances even further, we do not apply any regularisation techniques such as weight decay, learning rate warm-up or discriminative finetuning, except for NLI.
Impact of corpus origin and size In this section we investigate the influence of the homogeneity and size of the pretraining corpus on downstream task performance. With this aim, we train alternative version of CamemBERT by varying the pretraining datasets. For this experiment, we fix the number of pretraining steps to 100k, and allow the number of epochs to vary accordingly (more epochs for smaller dataset sizes). All models use the BASE architecture. In order to investigate the need for homogeneous clean data versus more diverse and possibly noisier data, we use alternative sources of pretraining data in addition to OSCAR: • Wikipedia, which is homogeneous in terms of genre and style. We use the official 2019 French Wikipedia dumps 13. We remove HTML tags and tables using Giuseppe Attardi's WikiExtractor.
14 • CCNet  (Wenzek et al. , 2019) , a dataset extracted from Common Crawl with a different filtering process than for OSCAR. It was built using a language model trained on Wikipedia, in order to filter out bad quality texts such as code or tables. 15 As this filtering step biases the noisy data from Common Crawl to more Wikipedia-like text, we expect CCNet to act as a middle ground between the unfiltered "noisy" OSCAR dataset, and the "clean" Wikipedia dataset. As a result of the different filtering processes, CCNet contains longer documents on average compared to OSCAR with smaller-and often noisier-documents weeded out. Table  6  summarizes statistics of these different corpora. In order to make the comparison between these three sources of pretraining data, we randomly sample 4GB of text (at the document level) from OS-CAR and CCNet, thereby creating samples of both Common-Crawl-based corpora of the same size as the French Wikipedia.
17 This calls into question the need to use a very large corpus such as OSCAR or CCNet when training a monolingual Transformerbased language model such as BERT or RoBERTa. Not only does this mean that the computational (and therefore environmental) cost of training a state-of-the-art language model can be reduced, but it also means that CamemBERT-like models can be trained for all languages for which a Common-Crawl-based corpus of 4GB or more can be created. OSCAR is available in 166 languages, and provides such a corpus for 38 languages. Moreover, it is possible that slightly smaller corpora (e. g. down to 1GB) could also prove sufficient to train highperforming language models. We obtained our results with BASE architectures. Further research is needed to confirm the validity of our findings on larger architectures and other more complex natural language understanding tasks.
Conclusion In this work, we investigated the feasibility of training a Transformer-based language model for lan-guages other than English. Using French as an example, we trained CamemBERT, a language model based on RoBERTa. We evaluated Camem-BERT on four downstream tasks (part-of-speech tagging, dependency parsing, named entity recognition and natural language inference) in which our best model reached or improved the state of the art in all tasks considered, even when compared to strong multilingual models such as mBERT, XLM and XLM-R, while also having fewer parameters. Our experiments demonstrate that using web crawled data with high variability is preferable to using Wikipedia-based data. In addition we showed that our models could reach surprisingly high performances with as low as 4GB of pretraining data, questioning thus the need for large scale pretraining corpora.
Acknowledgments We want to thank  Clémentine Fourrier  for her proofreading and insightful comments, and  Alix Chagué  for her great logo. This work was partly funded by three  French National  funded projects granted to Inria and other partners by the  Agence Nationale de la Recherche , namely projects  PARSITI  ( ANR-16-CE33-0021 ),  SoSweet  ( ANR-15-CE38-0011 ) and  BASNUM  ( ANR-18-CE38-0003 ), as well as by the last author's chair in the PRAIRIE institute funded by the  French national agency ANR  as part of the  "Investissements d'avenir" programme  under the reference  ANR-19-P3IA-0001.