Monitoring of Open Science practices in major medical journals - a case study
Creators: name: DeVito, Nicholas J
name: Staš, Jan
name: Kertész, István
name: Folly, Brenda Bley
name: Colombo, Cinzia
name: Stegeman, Inge
name: Onghena, Patrick
name: Florian, NAUDET
name: Baranyai, László
name: Van Den Eynden, Veerle
name: Kozula, Magdalena
name: VINATIER, Constant
name: Fratelli, Maddalena
name: Banzi, Rita
name: Varga, Monika
name: Gopalakrishna, Gowri
name: Leeflang, Mariska
Publisher: name: OSF Registries
Year: 2024
Subjects: subject: Medicine and Health Sciences

description: BACKGROUND

Monitoring (evaluating and quantifying) the use and application of open science practices is key to fostering open science and to understanding barriers and facilitators of open science practices. Monitoring also aids in measuring the impact of open science policies and can serve as a possible intervention to improve transparency and reproducibility. For instance, the Hong Kong principles (1) propose tracking open science practices as a key metric for researcher assessment. Open science monitoring is a key priority for organizations such as the UNESCO (2) and the G7 (3). The scientific community has begun to develop both principles covering which open science practices should be tracked (4) and how to monitor them. For instance, in biomedicine, a core set of 19 open science practices has been developed (5) covering areas such as preregistration, open data, open access and open code. 

Many biomedical journals have also begun to implement policies that promote open science practices. Most notably, the International Committee of Medical Journal Editors (ICMJE) made prospective registration a requirement to publish clinical trial results in 2005 (6). In 2018, it required a specific data-sharing statement to be included in each newly submitted paper (and prespecified in study registration) (7). The impact of these policies has been studied previously, showing a major impact on trial registration practices (8) and mixed results regarding actual data sharing (9)(10). Beyond the common policies set out by the ICMJE, there is variation across biomedical journals regarding other open science policies. In 2007, the Annals of Internal Medicine encouraged, without mandating, data-sharing practices (11). The BMJ encouraged data-sharing in 25 March 2009 (12), made it mandatory in 01 September 2013 for trials of drugs and devices (13) and in 01 July 2015 for all clinical trials (14). Starting in May 2024, all trials published in The BMJ will be required to share their underlying data (15). PLOS journals enforced clinical trial data-sharing in 03 March 2014 (16). This diversity in journals’ open science policies can be monitored by using the Transparency and Openness Promotion (TOP) factor (17), a metric proposed by the Center for Open Science (18) to assess journals’ open science policies. 

Various automated tools have been developed to track changes in behavior across the biomedical literature (e.g.(19)). These tools can be used to assess and monitor open science practices (20), and aid in developing an observatory of open science practices across scientific journals. If those tools are indeed able produce accurate and reproducible results, such an observatory could allow 1/ to determine the real percentage of open science practices, 2/ to assess whether changes in journal’s policies were followed by changes in published articles, and 3/ to point what needs to be improved by researchers and editors. 
 
OBJECTIVES
 
This aim of this study is three-fold:
 to set up a proof-of-concept observatory of major medical journals using automatic tools that can be used to monitor key transparency practices; 
 to describe open science policies in major medical journals;
 to describe retrospectively changes on open science and transparency practices for articles published in the major medical journals with a longitudinal description of registration and data sharing practices following major changes in their policies
METHODS

This protocol will be pre-registered on the open science Framework (link). 
Selection of journals 

Our study will assess a sample of major medical journals (Table 1). We will not rely on impact factor to select target journals as this metric has several limitations (21). Instead, we will focus on a purposeful sample of ten internationally recognized generalist medical journals (specialists medical journals will be excluded) with a majority of current (or former) members of the International Committee of Medical Journals Editors (ICMJE). Furthermore, two major open access journals in the field of medicine, JAMA Network Open and BMC Medicine (the flagship medical journal of the BMC series) will also be included.

Selected journals Creation date ICMJE Membership (2023) Open access for all research articles Text mining available (22)


Table 1: see attachement 

Table 1: Sample of medical journals
Text mining available means that those journals were already listed in the paper by Menke et al.
Description of journals’ current open science policies

Based on either an existing TOP Factor assessment by the Center for Open Science conducted within the previous year, or an analysis of editorial policies published each journal website, two researchers (CV and APMD) will independently extract information related to the TOP factor criteria for each included journal (23). In case of disagreement, third research (FN will arbitrate). 

Selection of articles 
Key transparency practices
In order to monitor key transparency practices in the target journals, we will include all research articles published in the 2020-2023 period. 
Registration practices for clinical trials following policy changes
We will retrospectively describe changes in registration practices for clinical trials, within our sample of journals by including all randomised controlled trials published in these journals between 2003-2010 (i.e. starting 2 years before the ICMJE policy mandating registration, and up to 5 years after implementation of this policy). 
Registration practices for clinical trials following policy changes
We will retrospectively describe changes in data-sharing practices for clinical trials. This will include extracting all randomised controlled trials published in a given journal from our sample from 2 years before a change in its data-sharing policy, and up to 2 years after implementation of the policy. Because journals have implemented policies at different time points, the selected timeframe will differ across journals (Table 3, see Background and Outcomes sections for more details). 
 

 Table 2: see attachement 
 

Table 2: Timeframe of the article to be analyzed. Depending on the journals and type from 2003 to date.
Orange: exploration of registration practices for RCT
Green: exploration of data sharing practices for RCT
Blue: exploration of all Open science practices for all types of research articles (RCT, Meta analysis or other). 
Search strategies
All eligible articles from the target journals will be identified via searches on PubMed detailed in the following subsection for the articles used for the analysis of key transparency practices. These search strings have been validated according to the PRESS guideline (24) with a specific reviewing by an information specialist (it is the same search string as our study that aims to develop the LLM tool, available on the OSF at https://doi.org/10.17605/OSF.IO/3V6JW ). 
Research articles
We will use the following search string: ((2020:2023[Date - Publication]) AND "hasabstract" NOT ("comment"[Publication Type]) NOT ("letter"[Publication Type]) NOT ("editorial"[Publication Type]) NOT ("published erratum"[Publication Type]) NOT ("news"[Publication Type]) NOT ("introductory journal article"[Publication Type]) NOT ("biography"[Publication Type]) NOT ("portrait"[Publication Type]) NOT ("congress"[Publication Type]) NOT ("interview"[Publication Type]) NOT ("retraction of publication"[Publication Type]) NOT ("personal narrative"[Publication Type]) NOT ("retracted publication"[Publication Type]) NOT ("patient education handout"[Publication Type]) NOT ("lecture"[Publication Type]) NOT ("autobiography"[Publication Type]) NOT ("clinical conference"[Publication Type]) NOT ("classical article"[Publication Type]) NOT ("address"[Publication Type]) NOT ("legal case"[Publication Type]) NOT ("expression of concern"[Publication Type]) NOT ("festschrift"[Publication Type]) NOT ("overall"[Publication Type]) NOT ("bibliography"[Publication Type]) NOT ("corrected and republished article"[Publication Type]) NOT ("interactive tutorial"[Publication Type]) NOT ("duplicate publication"[Publication Type]) NOT ("directory"[Publication Type]) NOT ("newspaper article"[Publication Type]) NOT ("periodical index"[Publication Type]) NOT ("dictionary"[Publication Type]) NOT ("meta analysis"[Publication Type]) NOT ("randomized controlled trial"[Publication Type])) AND ("Annals of Internal Medicine"[Journal] OR "BMC Medicine"[Journal] OR "BMJ"[Journal] OR "CMAJ"[Journal] OR "JAMA"[Journal] OR "JAMA Netw Open"[Journal] OR "Lancet"[Journal] OR "Nat Med"[Journal] OR "N Engl J Med"[Journal] OR "PLOS Med"[Journal]). 
Randomised controlled trials
We will use the following search string: (2020:2023[Date - Publication] AND "hasabstract" AND randomizedcontrolledtrial[Filter] AND ("Annals of Internal Medicine"[Journal] OR "BMC Medicine"[Journal] OR "BMJ"[Journal] OR "CMAJ"[Journal] OR "JAMA"[Journal] OR "JAMA Netw Open"[Journal] OR "Lancet"[Journal] OR "Nat Med"[Journal] OR "N Engl J Med"[Journal] OR "PLOS Med"[Journal])). 

After further development we will also try to use a IA Tool that we are developing (refer to box 1 and to research protocol: https://doi.org/10.17605/OSF.IO/3V6JW ) to identify all RCTs in case it outperforms our search strategy.
Similar search strings adapted to the different journals and timeframe will be used for analyses of registration and data sharing practices following policy changes.
Systematic review and meta-analyses
We will use the following search string: (2020:2023[Date - Publication] AND "hasabstract" AND meta-analysis[Filter] AND ("Annals of Internal Medicine"[Journal] OR "BMC Medicine"[Journal] OR "BMJ"[Journal] OR "CMAJ"[Journal] OR "JAMA"[Journal] OR "JAMA Netw Open"[Journal] OR "Lancet"[Journal] OR "Nat Med"[Journal] OR "N Engl J Med"[Journal] OR "PLOS Med"[Journal])). 

After further development we will also try to use our IA Tool to identify all systematic review and meta-analyses in case it outperforms our search strategy.
Collection of relevant articles
We will use R [version 4.2.2 RRID:SCR_001905], and the packages rvest and RSelenium to create web scraping scripts that extract eligible articles (PDF) and any supplementary materials from the website of each of the ten journals included in our sample. Two main strategies will be adapted to each journal:
- A strategy for open access articles, as detailed with an example using articles from The BMJ (Appendix 1); 
- A strategy for articles under a paywall, that will require institutional access and the use of another method using RSelenium packages, as detailed with an example using articles from JAMA (Appendix 2). 
In addition, for each eligible article, the XML file will be extracted either from PubMed, PubMed Central, Unpaywall or Alexandria3k if available, or through the API of the respective journal if accessible.
In case no solution is found to retrieve the pdf, XML and/ or the supplementary materials, we will contact the different journals to ask for an access to all the needed documents. 
Manual extraction of data in research articles
In order to determine the diagnostic performance of each tool to detect each open science practice, we will use a manual extraction of open science practices as a gold standard. Therefore 3 distinct manual extractions will be performed for 1/ key transparency practices, 2/ registration practices for clinical trials following policy changes and 3/ data sharing practices for clinical trials following policy changes (see below for more details including sample size computations). Because of time constraint (availability of researchers in charge of data extraction) data extractions have started on 29/04/2024 prior the registration of the protocol and are being performed by two independent reviewers (MS, FJA), but no interventions began before it. The first 100 extracted articles will be triple checked by CV and disagreements identified in the following articles will be resolved by consensus with a third reviewer (FN or CV). About 150 articles were extracted by one researcher as of registration and extractions are still on going without any analysis being carried out when the protocol is registered. 
Identification of automatized tools
We will identify existing automatic tools that can monitor any of the previously established 19 core open science practices (5). Those tools will be searched by reviewing the ScreenIT pipeline (20) curated by the Automated Screening Working Groups and by contacting several public (e.g., Rtransparent and ODDPub) and private groups (e.g., SciScore and DataSeer) performing open science monitoring. Box 1 lists some tools/initiatives identified in preliminary searches. 

RTransparent: R package that automatically identifies and extracts indicators of transparency (data availability, code availability, statements of conflicts of interest, statements of funding, statements of protocol registration) from TXT and PMC XML versions of published research articles in biomedicine (https://github.com/serghiou/rtransparent). For data and code, rTransparent uses ODDPub effectively with only small adjustments. We will expand the code for registration to cover OSF and PROSPERO and if needed all other)*.
ODDPub: R package that facilitates detecting open data and open code (https://github.com/quest-bih/oddpub)**.
ctRegistries: Checking for registration statement for clinical trials (https://journals.sagepub.com/doi/10.1177/17407745221087456, https://github.com/maia-sh/trn-pop).*
DataSeer: Artificial Intelligence based tool that facilitates checking data sharing policies (https://dataseer.ai/).*
SciScore: Machine learning based tool to evaluate items in the MDAR checklist and use of key biological resources (https://sciscore.com/). 
ContriBOT: A R tool, that detects and extracts Author contribution statements and acknowledgements from scientific papers. (https://github.com/quest-bih/ContriBOT) **.
IA Tool: We are currently developing and IA tool (see our protocol for both development and validation https://doi.org/10.17605/OSF.IO/3V6JW ) based on a large language model (LLM) such as Mixtral to extract the presence and absence of open science practices inside medical journal articles. Ideally this tool will be automatized, open source (or failing that, open weights). It should be able to identify journal articles and supplementary material that is in .pdf format and to extract core open science practices. The task of extracting content out of a scholar article with an LLM involves optional innovations to ensure accuracy, robustness, and efficiency, such as embedding of raw data into vector stores, refining the method of prompting the LLM, and employing techniques like retrieval augmented generation (RAG) or fine-tuning*.
Box 1: Potential tools/initiatives that we have identified in preliminary searches Some of those tools are overlapping, such as ODDpub and Rtransparent.
*This tool will use as input PDF or XML **This tool will use as input PDF only 

Table 3 lists the core set of the 19 previously established open science practices to be monitored as well as examples of existing automatic tools that could track each practice. We do not expect that all open science practices will be covered by existing tools but tools are already available for many of these practices such as data sharing and pre-registratio 

Table 3: see attachement 

Table 3: List of the 19 core open science practices and various solutions identified in preliminary searches 
Some of the core practices go beyond what can be extracted from articles. As our project focuses on research articles only, NA indicates that these practices may not be applicable in our project. 
* data identifier : dbSNP, dbVar, Sequence Read Archive, BioProject, Protein Circular Dichroism Data Bank, ArrayExpress, GEO, European Genome-phenome Archive, Japanese Genotype-phenotype Archive, MassIVE, MetaboLights, PeptideAtlas, ProteomeXchange, FlowRepository, Image Data Resource, European Nucleotide Archive, UniProt, dbGaP, Biostudies, and ClinVar, data repositories and software tools with RRIDs 
Outcomes 
Key transparency practices
For each open science practice (Table 3) in which an automated tool will be identified, we will use the tool on the sample of papers from 2020 to 2023 (included, Table 2). In case certain tools only apply to open access articles, we will restrict our analysis using those specific tools to those articles and document the number of open access articles used and the number of articles that were not used. We will therefore record the open access status of all research articles included in the study. In order to assess the diagnostic accuracy of the tools, we will randomly sample 10 articles from each articles type (RCT, meta-analysis or other research article) per journal for manual assessment (i.e. a total of 300 research articles) from 2022-2023. This sample size ensures a precision of ± 6% to estimate percentages in the worst-case scenario for precision estimations (i.e. if the percentage is 50%). These 300 articles will be manually extracted and will serve as the gold standard. This manual extraction will also be used to estimate how many papers retrieved using our search strategy are actually false positives (i.e. how many are not research articles). 
Registration practices for clinical trials following policy changes
In order to explore whether the tools are able to detect a change following the 2005 ICMJE policy that mandated registration, we will monitor registration practices for all available randomized control trials published 2 years before and 5 years after the policy (i.e. 01/07/2005) in all selected journals (when applicable), which means from 01/07/2003 to 01/07/2010 (Table 2). 
In order to assess the diagnostic accuracy of the tools, we will randomly select 15 articles per journal for manual assessment from 2003 to 2010. This sample size (150) ensures a precision of ± 8% to estimate percentages in the worst-case scenario for precision estimations (i.e. if the percentage is 50%). These 150 articles will be manually extracted and will serve as the gold standard. This manual extraction will also be used to estimate how many papers retrieved using our search strategy are actually false positives (i.e. how many are not randomised controlled trials). 
Data sharing practices for clinical trials following policy changes
In order to explore whether the tools are able to detect changes following implementation of the various data sharing policies, we will monitor data sharing practices for all available randomized control trials (RCT): 
- Published 2 years before and after the policy (i.e. 01/04/2007 (11)) that encouraged data sharing in Annals of Internal Medicine which means from 01/04/2005 to 01/04/2009;
- Published 2 years before and after the policies that encouraged (i.e. 25/03/2009 (12)), mandated for drug and devices (i.e. 01/01/2013 (13)) and then mandated for all trials (i.e. 01/07/2015 (14)), in The BMJ which means from 25/03/2007 to 01/07/2017;
- Published 2 years before and after the policy (i.e. 01/04/2014 (16)) that mandated data sharing in PLOS Medicine which means from 01/04/2012 to 01/04/2016;
- Published 2 years before and up to date after the ICMJE policy that encouraged data sharing (i.e. 01/07/2018 (7)), in all remaining journals which means from 01/07/2016 to 01/07/2020;
Longitudinal descriptions will be performed separately for each journal.
In order to assess the diagnostic performance of the tools, we will assess 50 articles randomly sampled articles from all 4 groups defined above (Annals, BMJ, PLOS, and ICMJE) over the selected period, i.e. a total of 200 articles. This sample size ensures a precision of ± 7% to estimate percentages in the worst-case scenario for precision estimations (i.e. if the percentage is 50%). These 200 articles will be manually extracted and will serve as the gold standard. This manual extraction will also be used to estimate how many papers retrieved using our search strategy are actually false positives (i.e. how many are not randomised controlled trials). 
Analysis
General approach
For each open science indicator and for each specific validation sample, we will use a 2x2 table (e.g. table 4) with the manual extraction as gold standard and the extraction by automatic tool as intervention. Due to the lack of consensus on a gold standard for automatic extraction, manual extraction is considered the most reliable method for obtaining information and will be regarded as the gold standard.

We will estimate the diagnostic accuracy of each tool by deriving sensitivity, specificity per indicator and F-score (25), by using the value identified by manual inspection as the gold standard. Systematic differences between journals will be explored by performing a subgroup analysis by journal. Estimated parameters will be presented with their 95% confidence intervals. Performance of different tools tracking the same open practices may vary because tools have been optimized on different training data and may also rely on different operational definitions of what is being tracked. If certain articles are unavailable, we maintain the predetermined power level by replacing them with alternative articles chosen at random from the same journal, or, if needed, from another randomly selected journal with a similar publication type.



Table 4: see attachement 


Table 4: Describe of the performance of each automatic tool. 
 
Key transparency practices
A descriptive analysis of each of the parameters described above will be carried out by journal. Categorical variables will be described over the time period using number and percentages (with their 95% confidence intervals). For registration and data sharing, subgroup analyses will be specifically performed for clinical trials and for meta-analyses separately.
Registration practices for clinical trials following policy changes
A descriptive analysis of clinical trial registration will be carried for all journals around 01/07/2005. Categorical variables will be described over the time period using number and percentages for each year. This analysis will be descriptive with no specific strategy regarding statistical inference.

Data sharing practices for clinical trials following policy changes
A descriptive analysis of clinical trial data sharing will be carried for all journals around 01/07/2018. Categorical variables will be described over the time period using number and percentages for each year. This analysis will be descriptive with no specific strategy regarding statistical inference.

Exploratory analysis 
A secondary analysis will assess whether performance differs between pdf vs. XML files, for papers for which both are available in order to help to interpret data
PERSPECTIVES
Implications 
In this study, we will explore various automated tools to monitor open science practices at major generalist biomedical journals. This work will inform the development of a broader open science observatory of biomedical journals. The leading biomedical journals in this study will hence provide important information about these journals that are considered as trusted sources by the medical community. Last, by tracking the open science practices, we will be able to describe changes occurring around implementation of various policy changes on selected open science practices (clinical trial data sharing and registration). Finally, if the accuracy of the automatic tools is good, we will be able to expand the observatory to other journals to more holistically measure the landscape, depending validation. 
Limitations
Because it will be based on a selected sample of major journals, this study will lack external validity regarding core open science practices in other journals (including specialty journals) (10). But given the complexity of the task, it is important that, before further development of the observatory, we start with a small sample of journals. These journals were chosen as we are sure they publish all the different types of research articles that are targeted by the open science practices we are interested in monitoring. In addition, because some automated indicators only detect mention of a registration, or mention of data sharing, it is likely that we won’t be able to detect failure to comply with study registration and/or data sharing plans (26). Finally, because journals can change their reporting practices over time and software tools can improve, it is possible that the diagnostic properties assessed in our study will differ in the future evaluations. Regarding this specific point, the potential identification of challenges in line with reporting in our study may be useful to inform future journal policies in order to reach a better harmonization and to ease monitoring of core Open Science practices. 
 
TIMELINE
See attachement 
 
ACKNOWLEGEMENT: 
We thank Anita Bandrowski for her advices and feedback on the protocol. 
REFERENCES