dataesr/scientific-paragraphs-categorization huggingface dataset:
Rationale for developing one checklist versus four different checklists for trials conducted using cohorts, registries, EHRs and administrative databases Team members discussed the advantages and disadvantages of creating individual checklists for each of the four types of data versus a single checklist for all four. It was determined that, although there are some differences in the implementation of trials across the different types of data sources, the methodological principles are similar, and there is substantial overlap in the design, conduct and factors that may influence interpretability. Thus, the steering committee reached consensus to develop a single statement, addressing any differences by including 'if applicable' to items in the checklist that may not apply to all trial designs and to clarify differences in the Explanation & Elaboration publication as deemed necessary.
22 oject phase 2: scoping review A preliminary 'long list' of possible reporting items was formulated by LK and KAM based on review of the CONSORT 2010 statement items, the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE)  27  and the RECORD statements,  20  as well as discussions with steering committee members. The STROBE and RECORD statements were considered the most relevant to this project because of their focus on reporting of observational studies and non-interventional studies using routinely collected data. A scoping review was conducted to identify: (1) articles on the methodology or reporting of RCTs conducted using cohorts or routinely collected data that could Open access inform the development of new items or modification of existing CONSORT items; and (2) trial reports to identify aspects of reporting that need improvement and examples of good reporting of potential checklist items that could be used to support CONSORT-ROUTINE.
28 We searched for relevant articles on trials conducted using cohorts, registries, EHRs and administrative databases from 2007 to 2018. After screening articles for inclusion and exclusion at the abstract and full-text level, 10 people from the team independently reviewed the included papers and provided suggestions for modifications or additional reporting guideline items until no new ideas emerged (saturation). Suggestions were added in a standardised, shared spreadsheet. At the same time, team members provided examples of good reporting for each proposed item or item modification. Additionally, the review helped us to create a list of authors with experience in these trial designs as potential participants for the Delphi study. Search terms used in the scoping review are shown in online supplemental file 1.
For all items (proposed modifications and CONSORT 2010 items), we provided participants with the opportunity to give open-ended feedback, using freetext boxes provided at the bottom of each survey page and at the end of the survey. At the end of the survey, participants were asked to provide any additional items that they believed would be important for reporting in trials conducted using cohorts and routinely collected data but that had not been included in the proposed set of new and modified items. We launched round 1 of the survey on 4 February 2019 with 2 weeks to provide responses. Round 2 was launched on 4 March 2019, and round 3 was launched on 1 April 2019. After each round, the Qualtrics built-in analysis software was used to generate a distribution of scores and to aggregate group results for each item (mean score, maximum and minimum score, SD, variance Open access and percentage ratings of 1-5 ranking for items) and summary statistics were circulated among all participants.
Session chairs then separately presented items from the preliminary checklist, results of the Delphi study and feedback from stakeholders, after which the group discussed in an open forum. Decisions were made on items to be modified or added based on the following criteria: (1) whether they addressed elements unique to trials conducted using cohorts or routinely collected data versus elements applicable to any trial and (2) whether they reflected information that should be included in a minimum reporting set of items. Notes were taken, and the discussions were audio-recorded to ensure that the content was accurately captured. Following the consensus meeting, refinement of the content and wording of the items was continued through online group discussions with CONSORT-ROUTINE project team members.
Delphi study results Of 125 people invited to take part in the Delphi study, 115 people registered via an online survey, and 92 (74%) provided responses on the items in round 1. Figures  3  and 4  present the types of stakeholder groups that completed round 1 of the Delphi study and the type of trials conducted using cohorts or routinely collected databases with which they had familiarity. Participants belonging to more than one category had the option of checking multiple options in the survey.
Round 3 Of the 77 people who completed round 2, 62 (81%) completed round 3. Of the 11 items in round 3, five items reached consensus in round 3. The remaining six items did not reach consensus after the three rounds. There were several new items suggested via the Delphi process but not added to the potential item list. The main reasons why some items were suggested but not incorporated were: 1. The suggestion was encapsulated in CONSORT 2010 items, was already captured by proposed new or modified items or could be captured by further modifying new or modified items. 2. The suggestion was not specific to trials conducted using cohorts and routinely collected data and, thus, was recommending a change to the CONSORT 2010 checklist, which was not the task of the CONSORT-ROUTINE group. Summary results of the three rounds can be accessed at: https:// osf.
io/ 4zh6f/ In-person consensus meeting Table  1  summarises the CONSORT-ROUTINE group's discussions and advisory decisions for each of the items that was discussed during the in-person meeting. If there were differing opinions on the inclusion or exclusion of items and consensus could not be reached, voting was implemented by the session chair, with an 80% threshold for inclusion in the checklist as part of the minimal set of recommended reporting items. The key recommendations that emerged were as follows: ► Proposed modification to CONSORT 2010 items: it was recommended to retain proposed modifications to seven CONSORT 2010 items. These modifications pertained to differences in mechanisms used to conduct trials using cohorts or routinely collected databases. As in previous CONSORT extensions, some of the recommended changes end with 'if applicable' to show that some information which authors are being asked to report might not be relevant or applicable for their particular RCT or the particular type of data that was used in the RCT.
► Proposed additional items: consensus was reached to include six additional items and to add a new subheading, 'Cohort or routinely collected database', to the checklist. A recurrent discussion point was the need to minimise adding new items to the abstract unless they are essential due to word limits imposed by journals. A suggestion was made to expand the explanatory text of the Explanation & Elaboration document for nine unchanged CONSORT 2010 items to clarify additional requirements for reporting aspects of the trial without modifying the item: item 1a (identification as a randomised trial in the title), item 4b (settings and location where the data were collected), item 5 (intervention), item 13b (losses and exclusions after randomisation), item 14a (dates of recruitment/follow-up), item 15 (baseline data), item 20 (limitations), item 21 (generalisability) and item 24 (study protocol).
CONSORT-ROUTINE pilot test The preliminary version of the checklist was pilot-tested by 17 people who had been previously involved in conducting trials using cohorts and routinely collected data. Based on feedback received from the pilot test, there were minor  Completely defined prespecified primary and secondary outcome measures, including how and when they were ascertained and the cohort or routinely collected database(s) used to ascertain each outcome. Open access modifications made to the wording of two items for clarity (item 1b and item 9) in the final checklist. 24.
23 We engaged with key stakeholders in trials research and potential end-users of the resultant CONSORT-ROUTINE reporting guideline throughout the development process. The process involved participants from a wide range of scientific disciplines and with diverse experience in conducting trials using different cohorts and routinely collected databases. As with other CONSORT-related guidelines, the inclusion of CONSORT Group members (IB, DM and PR) was intended to ensure consistency in the use of recommended methods in the development, dissemination and implementation of the extension. We recorded high response rates of 74% (92 respondents), 84% (77 respondents) and 81% (62 respondents) in Delphi rounds 1, 2 and 3, respectively. In addition, the number of registered participants and responders is larger than in most Delphi surveys used to develop healthcare reporting guidelines.
8 35 36 Finally, we achieved a high degree of consensus that was consistent across Delphi survey rounds for the majority of the items. There are also limitations to consider. One is that most participants were academic researchers with primary roles in trials research, and despite our broad engagement efforts, the number of participants from some stakeholder groups was small. One patient was included as a member of the reporting guideline development team, but no patients participated in the Delphi exercise. It is possible that perceptions about the importance of items might have differed across different stakeholder groups that might have favoured the inclusion or exclusion of certain items. Nonetheless, our project group included people from diverse backgrounds with expertise in using different types of data sources, who oversaw the development process to ensure that the checklist was equally applicable to, and representative of, all four types of data sources.
Decision: do not include the modification and retain the CONSORT 2010 item; expand the E&E text for clarification. No suggested modification. Decision: retain the CONSORT 2010 item. Note: additional item 1c was later merged with this item (see further). Reached for Noted the importance of stating the cohort inclusion. or routinely collected database(s) used to conduct the trial in the abstract, if not in the title. Decision: include the suggested new item with revisions. Note: the item was later merged with item 1b from CONSORT 2010. Not reached. Mixed views on the necessity of reporting the suggested new item in the abstract. Agreed that linkage is important to report in the body of the paper but not necessarily the abstract. Decision: do not include the suggested new item. Reached for Mixed views on the necessity of reporting inclusion.
Textbox 1. Description of critical appraisal tools The ROBIS contains 21 signaling questions divided into the following three phases: i) the assessment of relevance (optional); ii) the identification of concerns with the review process, in which bias can be introduced in four domains: study eligibility criteria; identification and selection of studies; data collection and study appraisal; and synthesis and findings; and iii) overall judgment about risk. The AMSTAR is an 11-item tool used to assess the methodological quality of quantitative reviews using mainly randomized controlled trials (RCT) designs. The AMSTAR 2 is a 16-item instrument that provides detailed and comprehensive assessment of SRs that include randomised or non-randomised studies of healthcare interventions. The MMSR tool has been developed to "categorise and critically appraise systematic reviews that incorporate quantitative and qualitative evidence to answer different questions about the effectiveness of development interventions  (Jimenez et al.
Research infrastructures (RIs) offer researchers a multitude of research opportunities and services and play a key role in the performance, innovative strength, and international competitiveness of science. As an important part of the generation and use of new knowledge and technologies, they are essential for research policies. Because of their strategic importance and their need for significant funding, there is a growing demand for the assessment of their scientific output and impact. Current research information systems (CRIS) have contributed for many years now to the evaluation of universities and research organizations. Based on studies on the application of CRIS to infrastructures and on a recent French report on the scientometric assessment of RI, this paper analyzes the potential of CRIS and their data models and standards (in particular the international CERIF format and the German RDC model) for the monitoring and evaluation of RIs.
These infrastructures are used by researchers from all disciplines, in astronomy, biology, physics, chemistry, human and social sciences, earth • Second, we provide a summary of the French study on RI assessment, in particular of the specific demands and expectations and of the recommendations for further action in the field of RI metrics. • Third, we discuss how the Common European Research Information Format (CERIF) and the German Research Core Dataset (KDSF/RCD) meet the requirements of present standard information and metrics for RI. The conclusion will summarize the main findings and make recommendations for further research and development on the evaluation of RI.
Florio, Forte, and Sirtori (2016)  published a case study to show how a social cost-benefit analysis (CBA) probabilistic model can be applied to evaluate a large-scale RI project, the CERN Large Hadron Collider. Based on empirical methods, they estimated that there is around a 90% probability that benefits exceed costs, with an expected net present value of about A2. 9 billion. Their approach combines several categories of data sources, including (a) accounting data and expert analysis of capital and operating expenditures, including in-kind contributions; (b) scientometric data to estimate trajectories of publications and their impact in a specific domain; (c) firms' survey data on technological spillovers expressed in terms of increased sales and cost savings, or increased profits; expert analysis of the technological content of procurement; company accounting data for industries involved in procurement; and expert analysis of the cost savings or other quantifiable effects of open source software or other technological spillovers; (d) survey data and other statistical evidence of the expected or ex-post effects on salaries of former students and early career scientists; (e) statistics about on-site visitors, web access, use of social media, exposure to traditional media, and data on travel costs, opportunity costs of time, and other information related to cultural effects; (f ) contingent valuation data through survey of samples of potential taxpayers about their WTP for potential discoveries related to a specific project"  (Florio et al.
A quite different use case of the evaluation of RI is the European VRE4EIC project  (Ivanovic, Theodoridou, & Remy, 2018; Theodoridou, Patkos, & Doerr, 2016). Coordinated by the European Research Consortium for Informatics and Mathematics (ERCIM), this project builds on existing e-RIs providing services, software, data, and resources to develop an enhanced virtual research environment (VRE). The ingestion of information on research, data, and computing infrastructures requires interoperable (standard) metadata on RI. Again, the CERIF is selected as the target format, because of its flexibility, quality maintenance, and political support. (Bailo et al. , 2017). A last and more recent example is the implementation of the European Open Science Cloud (EOSC). One part of their activity is the inventory, description, and assessment of existing data, computational, networking, and thematic infrastructures  (Vancauwenbergh, 2019 ).
Yet, so far, (August 2020), EOSC does not provide information on descriptive elements or formats for this assessment, except for general information on national open science and FAIR data policies. However, in spite of the need for evaluation, most projects dealing with RI and evaluation assess the RI content (i. e. , data and documents), and do not consider the RI performance as an object of evaluation on itself. For instance, the initiative for collaborative research information management by the UK Science and Technology Facilities Council (STFC) which "operates large scientific facilities to support experimental research in […] chemistry, materials science, and biochemistry (including) the ISIS neutron source, the Central Laser Facility and the Diamond synchrotron light source (with) large volumes of data and […] used each year by many thousands of experimental scientists from around the world"  (Crompton, Matthews et al.
). A recent survey provides some empirical elements on the place of RI evaluation in German public research  (Schöpfel, Azeroual, & Saake, 2019). Universities and other research organizations are regularly evaluated and must report on their research activities. To improve the quality of this reporting, many of them have implemented some kind of CRIS, as a central database for the collection, presentation, and evaluation of data related to research. Yet, following a survey with 51 German institutions, only a small percentage (about 10%) make use of their CRIS to evaluate the performance of their own RI in terms of output and input, with appropriate metadata 7. This last survey raises two other issues: Insofar as RIs serve different purposes from different communities and institutions, should their performance be assessed differently, according to and for each community and institution, as is done by those German institutions with their institutional CRIS? In this case, assessing the global performance of an RI would require the aggregation of all performance metrics produced by relevant institutions and clearly identified as related to this specific RI-a tedious method whose success would require a high degree of standardization between the institutions involved.
We will come back to these characteristics in more detail in the following section, as part of the French study. In summary, there is a general consensus that RIs are part of research evaluation and that they must be described and assessed. Also, because of the large variety of RIs, a standard and interoperable data model seems appropriate, in particular the only international standard format recommended by the European Commission for research information management system (CERIF). Section 4 will provide more information about the CERIF model and its potential for the evaluation of RI.
THE FRENCH STUDY ON RESEARCH INFRASTRUCTURE METRICS For the reasons mentioned above, the existing procedures and metrics should take into account the particular characteristics and functioning of RI to provide appropriate assessment of RI. Data models and systems made for universities and research organizations are useful but need adjustment for the specific needs of RI. Regarding CRIS in France and compared to other European countries, there is a relatively low degree of standardization among research structures, with few CERIF-compliant systems. In France, large RIs, known as Very Large Research Infrastructures (TGIRs), are mainly defined by their scientific potential of a national or international nature. The distinction in France between RIs and TGIRs is currently being called into question: It stems, for the most part, from agreements of administrative or financial scope, and it has been recently observed by the Court of Audit that this distinction compromises the readability of national policy (Cour des Comptes, 2019).
• Networking of means and results: Technical assistance is frequently associated with networking. In terms of resources, this is carried out by internationalization of similar resources (e. g. , LIGO and EGO VIRGO) or additional resources (neutron lines and X-ray lines coupling the experiments in a mixed program between SOLEIL and the LLB); there are also many opportunities for mutualization of instruments, software, and other resources in astronomy and climate sciences. In addition, pooling is also frequent and currently developing in the sharing of results (standardization of the presentation of acknowledgments, data, affiliations, databases, scientific publications; standardization of the presentation of platforms, European key performance indicators, ERC nomenclature for indexing disciplines, etc.
Develop open science metrics for publications and data This first group of recommendations is particularly sensitive for TGIRs, which, unlike university or research institutions, lack visibility in the large bibliometric databases referencing scientific production. The actors concerned by the recommendations are the persons in charge of the TGIR who must define, in an operational way, the contours of their scientific production: Indeed, this is most often not restricted to that of their teams but must also include that of their users, or even consider more broadly the production of knowledge that has directly benefited from the existence of the infrastructure. The publishers of large databases are also concerned by these recommendations, who may seek to include specific metadata for instruments and infrastructures, and to develop the nomenclatures of research fields.
In fact, as the French study shows, many infrastructures already do some kind of assessment, often without appropriate tools or models, specific and not standard, and not interoperable. The published projects in the field of research information management show that CRISs, with their standard data models, may be an option for the assessment of RI. Yet, as mentioned above, CRISs are generally designed for the evaluation of research institutions and organizations, not of infrastructures, which are usually considered and assessed by such systems as part of institutional resources, similar to other facilities, services, and equipment. Therefore, the following section analyzes how the main standard CRIS format (i. e. , the Common European Research Information Format [CERIF]), and the new German Research Core Dataset (KDSF/RCD) meet the requirements of the present standard information and metrics for RI.
CERIF Developed with the support of the European Commission and recommended for use by the EU member states, CERIF 13 is a generic and standard model for organizing and exchanging research information, the research domain and their relationships to each other on conceptual, logical, and physical levels. CERIF is intended to serve as a model for homogeneous access to heterogeneous data systems and as a definition of a data exchange format. The aim of CERIF is to serve as an interoperability level between the digital infrastructure and the research data, and to promote integration and exchange through standardization. The CERIF data model includes persons, organizations, their projects, funding, and generally everything that arises from or is connected to the research process. At the very heart of the CERIF model are three interconnected core elements: persons, organizations, and projects; all the other elements-outputs, activities, metrics etc.