dataesr/scientific-paragraphs-categorization huggingface dataset:
Assessment of risk of bias (quality assessment) Two authors (JL, SG) independently assessed study method quality by using the RoB 2. 0 tool released through the Cochrane, to assess risk of bias in randomized studies 9,10. In nonrandomized trials, we assessed the risk of bias using the ROBINS-I tool released through the Cochrane 11. In case of disagreement, a third researcher (AMD) adjudicated the assessment of risk of bias. The risk of bias was assessed according to preliminary considerations, the randomization process, effect of assignment to intervention, effect of starting and adhering to intervention, missing outcome data, and selection of the reported result. Data were presented using the robvis R package 12.
14. We used a random effect model to assess the population average mean difference and 95 % IC of intervention. We did exploratory subgroup analyses of the type of trial included (RCT or RCS) and a sensitivity analysis focusing on ACS. Treatment effects in subgroups were compared by a test of interaction (using Cochran's Q test and Higgins' I2). To evaluate an outcome, we needed at least three studies that analyzed it with complete data. Individual study effects and pooled effects were visualized through forest plots. Publication biases were assessed graphically through traffic-light plot. Data were pooled using Review manager with a 2-sided significance of 5%.
Selection of studies We identified 2,577 references from our searches (Figure  1 ). After removing 1,112 duplicates, we screened 1,465 titles of whom 1,437 were excluded. We then screened 28 abstracts, among which we identified nine potentially eligible articles. Among the 19 studies excluded by abstract, there were seven systematic reviews not eligible for analysis because they did not provide original data and/or did not address the question of systemic corticosteroids therapy in the field of VOC and/or ACS. Full texts of nine studies were reviewed and three were excluded because of the absence of control group (n=2)  16, 17  or absence of comparative analysis between corticosteroids and control group (n=1).
Proposed Methodology This section provides a full description of the proposed methodology to infer fault models that will help in designing hardware and software countermeasures at an optimal cost. It deals with three different levels of understanding in order to provide a cross-layer fault analysis. Fig. 7  depicts the proposed methodology. It is centered around a comparison between the obtained results that are stored in three databases (hardware, RTL and software databases) in order to make decisions about the consistency and applicability of RTL and software fault models. In other words, starting from the observations obtained at the lowest level of abstraction (i. e. , hardware level), it will be possible to optimize fault models at the RTL level, for example, by removing RTL faults that do no correspond to observable faulty outputs.
These parameters as well as the target board layout must be taken into account when describing the fault model. • Get a snapshot of the state of the target: for example, the registers and memory states will be read at the beginning and at the end of the program execution (using a serial communication link with the host PC or a debugger for example). The richer the information that can be accessed, the more precise the model will be: for instance, hidden performance counters could be used to get a more detailed view of the internal state, in particular when advanced microarchitectural features are implemented. Then, the snapshot will be compared with the configuration of a golden run. The faulty behaviours will be stored in a database (Hardware faults database in Fig. 7 ). This step will allow us to observe the relation between the observed faulty behaviours and the instructions in the target part.
As in the previous step, the resulting faulty behaviours will be stored in a dedicated database and then be compared with those obtained from the physical injections. To ease the comparison and the fault characterization at the RTL level, a divide-and-conquer approach is used to reduce the complexity: the fault simulation is applied into specific RTL module or specific microarchitectural component at once. This comparison will help in two aspects, as shown visually in Fig. 8. On the one hand, this aims at explaining at the hardware level the faulty behaviours obtained from physical injections, and hence, making the fault effect characterization easier. The explanation is done by revealing the origin of the fault at the RTL level and determining the responsible microarchitectural component, the register, or even the single flip flop behind obtaining the faulty behaviour resulting from injecting the fault(s) (step 3 in Fig.
In this case, dedicated techniques shall be employed to model the advanced architectural characteristics and the related faults at the ISA level. This can be achieved by modifying the source code before the compilation or mutating the compiled code itself after applying different combinations of compiler optimization levels. The expected faulty outputs will again be stored in a corresponding database. Then, a comparison process similar to the one mentioned earlier will take place between the RTL and the software faulty results. In step 5, an RTL model validates the consistency of a software model, whereas a software model will be usable to describe the occurrence and explain an RTL model at the application level, which makes the fault effect characterization at this level easier.
Preliminary Validation To validate our approach, a preliminary analysis study has been conducted by performing RTL fault simulation experiments on the same target part of Program 2 shown in Listing 2. The RTL description is for the ARM Cortex-M3, in particular, the DesignStart evaluation version  [30]. Using typical RTL fault models, we were able to observe various faulty behaviours. For example, by a single bit-set or a single bit-reset to specific signals that are located in the path of the instructions fetch between the Flash memory and the core as shown in Fig. 9 , in particular in the interface, we were able to observe the following faults: • Quad-skip or 128-bit skip. • Double-skip of the second and the third instructions. • Skip the second and repeat the first. • 128-bit skip and repeat the previous 128-bit.
Acknowledgments We thank  Leah Morgan  and  Bruce Fischl ,  Massachusetts General Hospital, A. A. Martinos Center for Biomedical Imaging, Department of Radiology, USA  for providing the human brain specimen 2 analyzed in this study. We express gratitude to the two donors involved in the body donation programs of the  Association des dons du corps du Centre Ouest, Tours , and of the  Massachusetts General Hospital. They made this study possible through their generous donation to science. This project has received funding from the  European Union 's  Horizon 2020 research and innovation Framework Programme  under grant agreement No. 654148  ( Laserlab-Europe ), from the  European Union 's  Horizon 2020 Framework Programme for Research and Innovation  under the  Specific Grant  Agreement No. 785907  ( Human Brain  Project  SGA2 ) and No.
Methods: A cross-sectional survey was conducted using a web-based questionnaire. Patient sociodemographic data were collected, including age, sex, ethnicity, educational level, health status, geographical location, motivation to self-manage, and digital health literacy (measured using the eHealth Literacy Scale tool). Patients with experience of using the Care Information Exchange (CIE) portal, who specified both age and sex, were included in these analyses. The patients' responses to 4 relevant survey items (closed-ended questions, some with space for free-text comments) were examined to understand their willingness and ability to identify and respond to errors in their PHRs. Multinomial logistic regression was used to identify patients' characteristics that predict the ability to understand information in the CIE and willingness to respond to errors in their records.
Study Design, Participants, and Data Collection A cross-sectional study, using a web-based survey, explored patients' views and experiences of using the Care Information Exchange (CIE), the largest shared PHR program in the United Kingdom. In 2018, the CIE was rolled out to the diverse 2. 3 million patients treated in North West London. At the time of this survey, the CIE held patient information from both hospitals and general practitioner practices in North West London, and records from 15 hospitals outside London-in Birmingham, Bristol, Liverpool, Manchester, Scotland, and Wales  [23]. The questionnaire was open for completion between July 1, 2018, and July 1, 2019. All patients registered with the CIE during the study period (N=27,411) were eligible to participate. The survey was administered via Qualtrics (a web-based survey platform).
Acknowledgments This research was funded by the  National Institute for Health Research (NIHR) Imperial Patient Safety Translational Research Centre  and supported by the  NIHR Imperial Biomedical Research Centre. This research was enabled by the  Imperial Clinical Analytics Research and Evaluation environment  and used the  Imperial Clinical Analytics Research and Evaluation team  and data resources. The views expressed are those of the authors and not necessarily those of the  National Health Service, NIHR or Department of Health and Social Care.
Metagenomics is a culture-independent method for studying the microbes inhabiting a particular environment. Comparing the composition of samples (functionally/taxonomically), either from a longitudinal study or cross-sectional studies can provide clues into how the microbiota has adapted to the environment. However, a recurring challenge, especially when comparing results between independent studies, is that key metadata about the sample and molecular methods used to extract and sequence the genetic material are often missing from sequence records, making it difficult to account for confounding factors. Nevertheless, this missing metadata may be found in the narrative of publications describing the research. Here, we describe a machine learning framework that automatically extracts essential metadata for a wide range of metagenomics studies from the literature contained in Europe PMC.
Introduction Whole genome shotgun DNA sequencing has emerged as a powerful tool for assessing the microbial content in a wide range of environmental samples. The advent of metagenomics analyses has enabled the phylogenetic and functional profiling of microbial communities found in a particular biome without the need of culturing. Meta-analysis between samples can reveal specific adaptations to environmental conditions, or candidate members of the community that may be responsible for health or disease  [1] [2] [3]. However, to establish causal relationships between observed phenotypes and compositional changes, potential confounding factors such as individual alcohol consumption or therapeutic interventions need to be taken into account  4, 5. Thus, to undertake meaningful analyses spanning multiple metagenomics datasets, accurate contextual metadata about the environmental conditions from which the sample was taken and the experimental methods is required  6.
Despite the efforts to deposit, organise and analyse metagenomics data in public databases  7, 8  , major obstacles to their usability thereof remain, due to incomplete, missing and/or inaccurate contextual metadata. Sequence databases, such as Sequence Read Archive (SRA)  9  and European Nucleotide Archive (ENA)  10  , provide permanent storage for metagenome sequences, which offer the potential for knowledge databases, such as MGnify  11  , MG-RAST  12  and Integrated Microbial Genomes with Microbiome Samples (IMG/M)  13  , to explore the taxonomic diversity and metabolic pathways of biome microbial communities. Although significant advances have been made in the computation and indexing of metagenomics data, the lack of detailed and structured microbiome metadata has dramatically hampered metagenomics cross-study comparisons.
To address this issue, metagenomics databases, such as Terrestrial Metagenome DB  14  , Human Microbiome Project  15  , Human Metagenome DB  16  , Genomes OnLine Database  17  (GOLD) and MGnify, have used controlled vocabularies from ontologies, such as ENVO (Environmental ontology)  18  , to manually annotate a broad range of metagenomics samples. However, many of these studies have incomplete detailed metadata about sample origins, due to the lack of metadata provided when depositing sequences into a database  19  and the inflexibility of hierarchical ontology relationships for describing diverse and specific environments, for example ENVO lacks vocabularies that could describe samples from many engineered environments. A scalable way to overcome these shortcomings is the development of automated methods to extract sample and experiment metadata from research articles describing metagenomics studies and link them to metagenomics datasets.
Overview A machine learning framework was developed and integrated for enriching a wide variety of metagenomics studies with essential and accurate metadata from open access research articles. This framework: (1) trained machine learning models on cross-referenced data to classify a broad range of biome publications (literature classification and triage); (2) defined and curated 16 novel bio-entities that describe vital metadata for diverse metagenomics studies (biocuration; Table  1 ); (3) trained and validated BioBERT-based NER models on curated datasets (BioBERT fine-tuning and internal validation); (4) deployed 16 high performance trained NER models in retrieving accurate metagenomics metadata from thousands of publications in Europe PMC (NER); (  5 ) standardised BioBERT-modelpredicted annotations with multiple domain-specific ontologies (normalisation); (6) validated the transfer of models accuracy and reliability to uncurated datasets in Europe PMC and ENA (external validation); and lastly (7) developed an integrated and fully automated metagenomics annotations pipeline that regularly enriches up-to-date metagenomics studies (19,209) with model-predicted metadata from research articles (114,099) (databases enrichments).
Literature classification Predicting a wide variety of microbiome environments was an essential requirement to triage publications for NER tasks. This requirement was addressed by constructing supervised training datasets, where the GOLD annotations assigned to metagenomics studies were mapped to the corresponding MGnify cross-referenced publications. Subsequently, random forest models were trained on the hierarchical levels of GOLD ontology yielding diverse biome prediction models. Comparing models performance metrics revealed higher F1-scores of 91-97% for the top GOLD hierarchical levels (Engineered, Environmental and Hostassociated). However, Doc2Vec models were outperformed by their TF-IDF alternatives (Table  2 , Figure  1 ). Hence, TF-IDF models were applied for the prediction of microbiome environments in new metagenomics publications (selected based on the publication being referenced from an ENA study record) and subsequently literature triage.
Named Entity Recognition (NER) To identify metagenomics data in research articles, 16 novel metagenomics entities (Table  1 ) were defined, curated, and trained using NER models. Training and test datasets were constructed from the curated 140 ENA cross-referenced publications, which were not contained in the MGnify database to ensure unbiased and broader coverage of biome. Triaged publications (see supplementary Table  2 ) were randomly selected and equally categorised using highest performance random forest classifiers (Table  2 , Figure  1 ). The manual curation of metagenomics entities in literature triage yielded 9567 annotations for 2496 sentences. As entities can be nested within each other, 16 individual datasets were constructed separately for fine-tuning 16 BioBERT models. Each entity dataset, comprising 2496 BERT-tokenized sentences with BIO-tagged entities, was partitioned for training (90%; 2246 sentences) and testing (10%; 250 sentences).
During training, grid search over 5 learning rates and 7 epochs showed best performance models with diverse combinations of hyperparameters per entity. These varieties of combinations were expected to give the best performance, given the overlapping nature of the entities. Typically, the greater the contexts overlap, the greater the training the model required to achieve the best performance. Table  3  shows the best tokenwise precision (P), recall (R) and F1-score (F1) macro-averages for trained NER models per entity. Models achieved a precision of 80-100% and a F1-score between 71-98%. To validate the accuracy of NER models predictions, 16 new NER models were trained from scratch on uncurated predicted annotations from 1500 new articles. The trained models were then tested in recognizing entities in the curated datasets (140 articles that were not included in the 1500 training articles).
Database enrichment To enrich metagenomics studies with data from research articles, 114,099 publications in Europe PMC were processed and annotated using the highest NER performance models (Table  3 ). Those publications encompassed 19,900 publications linked to studies in the ENA and MGnify databases. A new pipeline was developed to continuously provide metagenomics annotations for new open access publications. Users are now allowed to search, explore and retrieve metagenomics annotations programmatically from the Europe PMC website, using search queries (e. g. ANNOTATION_PROVIDER:"Metagenomics"), a SciLite application (Figure  2  : article view) and the annotations API (https://europepmc. org/AnnotationsApi ), respectively. To date, MGnify has enriched the metadata of 1600 studies with the metagenomics annotations from corresponding articles using the Europe PMC annotations API.
g. , forests, oceans, rivers, lakes) and engineered environments (e. g. , bioreactors, wastewater treatment plants, food, and microbial fuel cells) (see supplementary Table  2 ). Subsequently, introducing 16 novel entities that flexibly accommodate the essential metadata of these environmental categories and training BioBERT NER models on their miscellaneous contexts have shown their utility for addressing the limitations of current databases and ontologies in describing extensive ranges of novel or hybrid microbiome environments. Although metagenomics NER models demonstrated precisions of 80-100% (Table  3 ) for most of the entities, relatively lower F1-scores were observed with the categories engineered, site, state and treatment. Qualitative assessment of these entities revealed that their metadata can be represented by single (unigrams) or multiple tokens (ngrams).
Curators seemed to be more comprehensive in annotating entity-related tokens when compared to models, which tend to generate few, yet precise annotations; hence, the relatively low token-wise recall (see supplementary Table  9 ). Accordingly, low F1-scores seem not necessarily reflecting poor model performance, but rather suggest that BERT-based models can perform better than curators in learning the contexts and patterns of weakly imprecise data  32   We also demonstrated that the high accuracy of metagenomics NER models can extend to uncurated datasets. Training BioBERT-based NER models from scratch on large uncurated predicted datasets, generated from metagenomics NER models annotations, revealed the reversible transfer of their reliability and outperformance in predicting curated datasets with precision of 76-99% (see supplementary Table  6 ).
Moreover, evaluating the accuracy of model-derived annotations against ENA author-submitted metadata showed large overlaps between them in numerous studies regarding MIxS biome environmental packages, sequencing library strategy, material, location, host, sequencing methods, and collection dates (Figure  4 ). Conversely, although model-derived annotations showed greater MIxS coverage for more studies, limited overlaps were noticed due to incorrect, inaccurate, or inconsistent author-submitted data formats (see supplementary Table  8 ). The analysis is not without limitations, as it covered metadata only from ENA fields used in more than 50 studies. However, it still offers some interesting insights. For instance, metagenomics annotations provided metadata for nearly 2810 studies that had no sample or experimental metadata in ENA; hence the enrichment of ambiguous sequence datasets to a more meaningful and exploitable dataset.
As demonstrated in Figure  4 , this framework captures metadata terms that have a high degree of overlap to the corresponding metadata found in ENA, even when considering only exact terms. Moreover, the additional metadata identified through this framework will facilitate the more extensive use of existing datasets, which have been collected and sequenced at great expense and cannot simply be recreated. The expanded set of metadata will also simplify new meta-analyses, for example the analysis of amplicon sequence variants (ASVs)  39  , a method that can compare taxonomic profiles between different studies. However, ASV analysis requires that only equivalent regions are compared. Thus, the expansion of metadata related to amplicon datasets, namely "target gene", "target subfragment" and "PCR primer", means that datasets can now be rapidly selected using a single API call to Europe PMC, without the labouring reading of papers that would be otherwise be required.
Note, that for very common terms like "rRNA", it may be necessary to pair this with other terms, such as "library strategy" to ensure that the queries are restricted to amplicon-based studies. Furthermore, with the enriched metadata for shotgun metagenomics datasets, it will become easier to control for confounding factors by either producing closely paired datasets or statistically evaluating the significance of metadata differences. The wide range of metadata fields and biomes covered by the Europe PMC Submission System for Annotations means that these results are applicable to a broad range of researchers. This system is already linked to MGnify to enable the dynamic retrieval and display of additional metadata terms alongside users submitted annotations. Given that MGnify provides a uniform analysis across databases, this eliminates confounding factors from bioinformatics analysis.
Literature classification Training dataset. Due to the lack of public datasets for training metagenomics-specific literature classifiers, we constructed our own training dataset based on MGnify, which provides access to a broad range of metagenomics studies and associated analysis results, but importantly curates microbiome studies with standardised biome annotations and provides direct links to their relevant publications. Accordingly, a supervised training dataset was constructed by mapping the curated GOLD biome annotations assigned to the metagenomics studies in MGnify (https://www. ebi. ac. uk/metagenomics/) to the corresponding publications in Europe PMC describing these studies (674 articles accessed on 17 February 2020). First, metagenomics studies were collected from MGnify with both PMC identifiers and GOLD annotations.