dataesr/scientific-paragraphs-categorization huggingface dataset:
The study sample was initially composed of 2,121 adults (18-79y) and after exclusion of underreporters the final sample contained 1,774 adults (749 men and 1,025 women). The NutriNet-Santé study is an ongoing French web-based cohort which started in 2009 and has previously been fully described elsewhere  (21). The NutriNet-Santé study is conducted in accordance with the Declaration of Helsinki guidelines and was approved by the Institutional Review Board of the French Institute for Health and Medical Research (IRB Inserm no. 0000388FWA00005831) and the 'Commission Nationale de l'Informatique et des Libertés' (CNIL no. 908450 and 909216). Electronic informed consent is obtained from each participant (EudraCT no. 2013-000929-31). All questionnaires are completed online via a web-based interface.
* Because of insufficient data to define an intake at which clinical signs of deficiency may appear for these nutrients, the thresholds were defined as the intake covering the requirements of 2. 5% of the population, defined as EAR(1-2CV) (see Supplemental Table  1 ). † These thresholds parameters were then further corrected to calibrate the estimated average risk using figures for the national prevalence of goitre for iodine and of anaemia for iron. For iodine, the recalibrated DT was set at 100 µg and the aDT at 71. 3 µg. For iron, the recalibrated DT was set at 0. 70mg and the aDT at 0. 38 (see Supplemental Method 1). ‡ This value applies to men and non-menstruating women. Requirements for menstruating women were estimated by considering menstrual losses using a Monte-Carlo simulation (see Supplemental Method 1).
Acknowledgements The authors sincerely thank all the volunteers of the NutriNet-Santé cohort for their continuous participation in the study. We also thank  Younes Esseddik ,  Thi Hong Van Duong ,  Régis Gatibelza ,  Jagatjit Mohinder , and  Aladi Timera  (computer scientists);  Fabien Szabo de Elenyi ,  Nathalie Arnault ,  Julien Allegre , and  Laurent Bourhis  (datamanager/statisticians);  Cédric Agaesse  (dietitian);  Sandrine Kamdem  (physicians); and  Nathalie Druesne-Pecollo  (operational coordinator) for their technical contribution to the  NutriNet-Santé study. Financial support  M Salomé's PhD  fellowship is currently being funded by a research contract with  Terres Univia , the  French Interbranch organization  for plant oils and proteins. F Mariotti  is the scientific leader of this contract.
Open research data are increasingly recognized as a quality indicator and an important resource to increase transparency, robustness and collaboration in science. However, no standardized way of reporting Open Data in publications exists, making it difficult to find shared datasets and assess the prevalence of Open Data in an automated fashion. We developed ODDPub (Open Data Detection in Publications), a text-mining algorithm that screens biomedical publications and detects cases of Open Data. Using English-language original research publications from a single biomedical research institution (n = 8689) and randomly selected from PubMed (n = 1500) we iteratively developed a set of derived keyword categories. ODDPub can detect data sharing through field-specific repositories, general-purpose repositories or the supplement.
Additionally, it can detect shared analysis code (Open Code). To validate ODDPub, we manually screened 792 publications randomly selected from PubMed. On this validation dataset, our algorithm detected Open Data publications with a sensitivity of 0. 73 and specificity of 0. 97. Open Data was detected for 11. 5% (n = 91) of publications. Open Code was detected for 1. 4% (n = 11) of publications with a sensitivity of 0. 73 and specificity of 1. 00. We compared our results to the linked datasets found in the databases PubMed and Web of Science. Our algorithm can automatically screen large numbers of publications for Open Data. It can thus be used to assess Open Data sharing rates on the level of subject areas, journals, or institutions. It can also identify individual Open Data publications in a larger publication corpus.
Introduction The benefits of openly shared research data (Open Data) for science and society as a whole are manifold, including increases in reproducibility, resource efficiency, economic growth, and public trust in research, as stated e. g. by the Concordat on Open Research Data, drafted by Universities UK, Research Councils UK, HEFCE and Wellcome Trust (The Concordat Working Group 2016). Open Data are increasingly recognized as both an indicator of quality and an important resource that can be reused by other scientists and that can increase transparency, robustness, and collaboration in science  (Fecher, Friesike, and Hebing 2015). Several large funders and institutions advocate for Open Data, e. g. the European Commission, NIH, EUA, RDA Europe  (Guedj and Ramjoué 2015; NIH 2003; EUA 2017).
Definition for Open Data & Open Code used in this study Apart from openness as defined by the Open Definition, see (Open Knowledge Foundation 2017), there are other criteria concerning the reusability of datasets, known as the FAIR criteria  (Wilkinson et al. 2016 ). However, few publications would currently comply with the full set of FAIR criteria. Therefore, we used a definition of Open Data that covers only some aspects of the FAIR criteria in addition to the openness criterion. We only considered research data shared together with a publication. These research data could be raw or preprocessed, but not analyzed or aggregated. Thus, they would allow the analytical replication of the results (or at least a part thereof). However, an imprecision with respect to when data are considered "raw" necessarily remains due to differences in disciplinary practices.
Primary data or secondary data (data that is not generated by own experiments but rather by the combination of available data sets, e. g. as part of a meta-analysis) were both eligible, as long as they fulfilled above requirement and the datasets shared were generated by the authors. We considered image data or other audiovisual data only if they encompass a substantial amount of the research data used in the study. In case of doubt, data deposited in repositories were considered Open Data due to their findability and potential reusability, while conversely data in supplementary materials were only considered Open Data if these were clearly raw data. We excluded audiovisual data that are primarily shared for illustrative purposes. The amount of data was not a criterion for inclusion per se, such that data from a study with a handful of subjects were still considered Open Data.
We used similar criteria for Open Code, and counted a publication as Open Data/Code publication if: (i) The publication explicitly mentioned where the shared data/code can be found. This can be a DOI, URL, database ID/accession code or a reference to a specific file in the supplement (a reference to supplementary materials without further details was not sufficient) (ii) The data/code was freely accessible to everyone. Data/code was not only shared upon request and no application or registration process was required. (iii) The data/code in principle allowed an analytical replication of at least parts of the results. Sharing summary results was not sufficient. However, we did not check the actual analytical replicability of the publications, as this is outside the scope of this study. (iv) The data/code was in a format that is machine-readable or that can easily be transferred into one.
We allowed tabular and textual data shared in Excel-or Word files but excluded data shared in PDFs (including the main publication manuscript) or as pictures of tables, as those formats make it difficult to reuse the shared data. For image data,. pdf would have been an acceptable format, but we did not encounter cases where PDF files contained raw image data by our definition, i. e. , beyond illustrative purposes. (v) The reuse of data/code previously published by other researchers was not considered. (vi) We did not presuppose an explicitly stated license (e. g. CC BY, MIT) for the shared data/code. We specifically did not address all possible kinds of data sharing statements. We explicitly excluded cases of data sharing where a registration or application is necessary to access the data.
While cases of restricted access to data can have good reasons, for example for clinical datasets, this is out of the scope of the current work, which only considers Open Data. We also excluded cases of data being available upon request. Making data available upon request is considered a suboptimal data sharing strategy and several studies  (Savage and Vickers 2009; Rowhani-Farid and Barnett 2016; Naudet et al. 2018 ) have shown that in many of those cases data cannot be accessed, especially after longer periods of time  (Vines et al. 2014).
Creation of training and validation datasets We created two training sets and a validation dataset for the development of the algorithm. The first training dataset consisted of the institutional publications record of the Charité -Universitätsmedizin Berlin from the years 2015 to 2017. To retrieve the publication record, we searched the publication databases PubMed and Embase for publications with affiliations that matched a list of keywords with variations of the institutional name (see Supplementary File S1). Additionally, we filtered the results for the publication years 2015-17 and for the publication type 'journal article' (PubMed) or ' article' (Embase). We obtained 9896 results from PubMed and 9256 results from Embase. The resulting publication lists from both databases were then combined and duplicates were removed by matching the DOI, PubMed ID (PMID), or the title and the last name of the first author.
The combined publication list consisted of 11192 entries. To retrieve the publication PDFs, we loaded the combined publication list into EndNote TM (Version: X8. 2, https://www. endnote. com) and used the 'Find full text' function. That way, 8689 (78%) of the publications could be retrieved. Some full texts could not be retrieved from journals for which our institution has no access (see Figure  1 ). The second training dataset consisted of publications randomly selected from all publications listed in PubMed in the year 2018. For this, we downloaded a list of all PMIDs from 2018 by exporting the search results using the following search filter:   This dataset was originally planned as the validation dataset, but due to low sensitivity of the algorithm developed on the institutional publication record, we decided to use this second dataset to improve the algorithm further.
Manual check of test & validation samples To establish a gold standard for the Open Data detection in the training and validation samples, we manually screened the publications for Open Data. The screening followed a standardized procedure, involving the following steps: 1. We checked if the article is a research article. Editorials, reviews, case reports or other article types (see above) were excluded from the sample. As the automatic exclusion via the metadata did not detect all those cases, this manual check was necessary. 2. We screened the article for a distinct "data availability" section, which is primarily found either at the beginning of the publication or at the end of the main text, before the references. 3. We used the search function to search for several keywords: "availab", "access", "raw", "deposit", "supp", "data", "code", "script", "software".
Those keywords were chosen as very general terms that might find any sentence relevant for Open Data while still keeping the manual search feasible in a reasonable time. The sentences of the detected keywords were then manually checked for relevant Open Data or Open Code statements. If some of the keywords appeared more than ten times in the manuscript, they were replaced by more specific keywords: "access" was replaced by "accession" and "accessible", "supp" was replaced by "supporting" and "supplementary", and "data" was replaced by "dataset", "data set", "database" and "data base". We used this approach instead of reading the entire full text for time reasons but also to find the most relevant passages more reliably. 4. If a relevant statement was found, we manually verified that the data or analysis code was indeed present and that they fulfilled the Open Data definition as explained above.
If the found data were stored in a very large ZIP file (>200MB), we did not extract the data but assumed that they contained data that agree with the Open Data definition. A more detailed description of the manual search strategy can be found in supplemental file S2. To estimate the interrater reliability of the manual screening, a second rater checked 100 publications from the validation dataset independently. For this, we chose a random sample of 25% of all positively identified Open Data papers (n = 21) and added a random sample of n = 79 negatively identified papers to partially even out the strong class imbalance in this dataset. Both raters used the manual screening procedure as defined above to determine the Open Data status (either "true" or "false") for each of the 100 publications independently.
We then calculated the percentage of cases where the raters agreed in their assessment of the Open Data status of the paper. If the agreement between raters had been too low, the interrater reliability check would have been extended to a larger sample, including all positively identified cases. However, no prespecified threshold was defined for this. For the 100 publications of the validation dataset that were checked by two independent raters, an agreement regarding Open Data status was reached in 95% of the cases. Due to the high degree of agreement, we decided that no further extension of the double check was necessary. For details on the interrater sample see supplemental file S6.
Algorithm development procedure For the development of ODDPub, we used an iterative procedure to manually derive and refine a set of keywords and key phrases to detect Open Data statements, focusing on first increasing the sensitivity and subsequently increasing the specificity in each step. As we could not check all publications in our first training sample manually for Open Data and as the prevalence of Open Data was rather low, we focused on analyzing all positively detected cases as well as a random subset of the negatively detected cases during the first iterations. We give a brief summary of the development procedure here. A more detailed description can be found in supplemental file S3. For details on the different algorithm stages, see supplemental files S4 and S5 as well as the GitHub repository (https://doi.
org/10. 5281/zenodo. 4071699). We started with a small number of confirmed Open Data publications. We extracted all keywords or key phrases relevant for the Open Data status and wrote an R script that uses regular expression matching to search for the keywords in the sentences of the preprocessed publication texts. If any of the keywords was detected, the publication was counted as hit. We applied this first version of the keyword search to the whole Charité training dataset (N = 8689 publications). We manually checked samples from the positively detected cases as well as from the negative cases. Using the information from the false positive and false negative cases, we improved the algorithm by adding new keywords and forming keyword groups of which several have to match in the same sentence.
This was repeated for three rounds, where the algorithm was again applied to the whole training dataset after each improvement and samples of positive and negative cases were manually assessed to further improve the keyword groups and to decrease the false positive and false negative rate of the algorithm. Among the keyword groups were groups that comprised names of repositories as well as their accession numbers. In those keyword groups we entered all the repositories that we found at least once among the detected Open Data cases in the publications of the training dataset. Additionally, we added repositories that were in the following two repository lists: 1) a list of the 20 most common repositories from a large analysis of data availability statements of PLoS One publications from  Federer et al.
2018  and 2) a list of repositories recommended by the data journal Scientific Data (Scientific Data 2020), from which we only included the biomedically relevant journals. We then applied the improved algorithm to the first random sample of 992 PubMed publications. First, we manually searched all publications in the sample for Open Data, where we detected 98 (9. 9%) Open Data publications. Then we ran the algorithm on the same publication sample and compared the results to the gold standard of manual search. We found a sensitivity of 45% and a specificity of 98%. As we considered a sensitivity of 45% insufficient, we decided to use this first PubMed sample to improve the algorithm further and to draw a second PubMed sample as the final validation sample. In a final round of improvement, we used the manually detected Open Data statements from the first PubMed sample that were missed by the algorithm to add new keywords and keyword categories for improved coverage of the algorithm.
Measuring performance of the algorithm We applied the final version of our algorithm to all publications of the validation dataset and compared the results with the manual screening results using the sensitivity, specificity and F 1 score as performance measures. As an additional benchmark, we also extracted the associated metadata on the datasets recorded in the databases PubMed and Web of Science for all publications in the validation sample. For PubMed we used the metadata field 'Secondary Source ID', which contains database names and dataset identifiers of datasets associated with the publications. Those databases comprise many genetic databases as well as the generalpurpose databases Dryad and Figshare. For Web of Science we used the associated datasets that were linked to the publications of the validation sample.
Structure of the resulting algorithm The resulting text-mining algorithm has a hierarchical keyword structure and searches for several categories of similar keywords in each sentence (see Figure  4 ). We grouped similar keywords identified in the statements indicating open data or code from the training dataset into categories (see supplemental file S4). Examples are the categories FIELD_SPECIFIC_DB, containing database names like "Gene Expression Omnibus", "Sequence Read Archive", or "Protein Data Bank" and AVAILABLE, containing words to denote that the data have been made available, like "deposited", "released", or "submitted". The explanations in this section hold true for the detection of both Open Data and Open Code. As only few cases of Open Code were found, the estimates for the performance metrics for the Open Code detection are unreliable.
Comparison to PubMed and Web of Science Next, we compared our results to the linked datasets found in the databases PubMed and Web of Science (see Figure  5 ). For the comparison with PubMed we retrieved the entries in the 'Secondary Source ID' field in PubMed for the 792 publications in the validation sample. We found that 19 of those publications have an entry in the secondary ID field. 11 of the entries referred to clinical trial registration IDs (8 on ClinicalTrials. gov, 3 on ISRCTN). Four entries linked to a Figshare dataset, two linked to a Dryad repository, and one entry linked to the Genbank and the Protein Data Bank, respectively. All Figshare and Dryad datasets were also detected by ODDPub. The two entries for Genbank and the Protein Data Bank were both cases of reuse of previously published data.
Yes No Human rater Yes  For the comparison with Web of Science, we retrieved the datasets linked in Web of Science for the 792 publications in the validation samples. We found that 11 publications have reported associated data. Those fall into the following categories: • 3 of them were previously published R packages on CRAN (no dataset; also did not fall under our definition of Open Code) • 4 had links to datasets in the 'Cambridge Crystallographic Data Centre' (CCDC) database • 2 had links to datasets in the 'Protein Data Bank' (PDB) database • 1 had a link to a Dryad repository • 1 had a link to a dataset in the PRIDE database Of those 8 reported datasets, 6 were found by both ODDPub and manual search. The PDB dataset was only found by manual search and one of the CCDC datasets was found by neither ODDPub nor the manual search.
Categories of Open Data detected To determine which categories of Open Data were most frequently discovered, we categorized all 91 manually detected Open Data publications of the validation dataset in the following categories: supplemental data, field-specific repository, general-purpose repository (including GitHub), institutional repository, data journal, and personal/project-specific website. One publication can be assigned to multiple categories if applicable. Most of the detected Open Data publications used either the supplement or a field-specific repository (Table  4 ). Of the 24 manually detected Open Data cases that were not found by ODDPub, there were 17 cases with supplemental data and 7 cases with field-specific repositories. Sharing data via the supplement was thus less likely to be detected by the algorithm.
Data available upon request 1 Linked OSF repository was empty 1 Case of Open Data missed by manual search 1 Open Data was not fulfilled in some way. In three cases, the algorithm made a major mistake and detected a statement that was not related to data sharing at all. In one case, an Open Data publication that was missed by the manual search was detected. A list of all detected Open Data statements for the training and validation datasets as well as the results of the manual screening can be found in a separate OSF repository (https://doi. org/10. 17605/OSF. IO/YV5RX).
The studies of  (Federer et al. 2018; Colavizza et al. 2020 ) both could use a less complex classification algorithm on the data availability statements, which is not possible when searching the full text. Compared to the two currently available publication databases PubMed and Web of Science that do link information on Open Data to publications we could show a much better coverage of our algorithm on the validation dataset. This suggests that currently our approach is superior to the other available approaches for the automated detection of Open Data publications from a larger publication corpus.
For an assessment of datasets published independently of a publication, a different approach would be necessary. Additionally, the scope of our algorithm was the biomedical literature and many of the Open Data statements found by the algorithm refer to repositories specific to the biomedical domain. The performance on other research areas has not been tested and one can expect a decrease in performance of the algorithm when applied to other research areas. The accessibility of journals is another limitation. As not all journals were accessible to us, we could not include those into our training dataset. Due to this, we could have potentially introduced a bias in our training dataset and missed additional relevant repositories or keywords. Importantly, our definition of Open Data did not require data to be FAIR  (Wilkinson et al.