TITLE: Open Science and COVID-19 RCTs
AUTHORS: John     Borghi

[DESCRIPTION]
Open science is an umbrella term that covers a wide variety of efforts focused on making scientific research more transparent and accessible. Practices related to open science exist along a continuum of openness and an individual research effort can be more or less open along a number of dimensions- including the accessibility of related publications, availability of data and code, and application of techniques such as pre-registration that enhance transparency related to project aims and methods. Among the many motivations for adopting open science practices is the link between openness and reproducibility - the more transparent the research effort, the more easily its conclusions can be examined, evaluated, and built upon by others.

This protocol outlines our methods for assessing the openness of randomized controlled trials (RCTs) related to the prevention and treatment of COVID-19. RCTs are commonly described as the "gold standard" of evidence in clinical research, which underlines the importance of their results are transparent.

[STEPS]
SECTION: Literature Searching
1. Search Strategies    

The following are the copy and paste-able versions of our literature searches.

PUBMED

(("randomized controlled trial" [pt] OR "controlled clinical trial" [pt] OR randomized [tiab] OR randomly [tiab] OR randomised [tiab] OR randomization [tiab] OR randomisation [tiab] OR placebo* [tiab] OR placebo* [ot] OR random* [ot] OR pragmatic [tiab] OR pragmatic [ot] OR blind* [ot] OR "Pragmatic Clinical Trial" [Publication Type] OR blind* [tiab]) AND ("covid 19" [tw] OR covid [tw] OR covid19 [tw] OR "ncov 2019" [tw] OR "novel coronavirus" [tw] OR "sars cov 2" [tw] OR "sars cov-2" [tw] OR "ncov 2019" [tw] OR sarscov2 [tw] OR (wuhan [tw] AND coronavirus* [tw]) OR "corona virus*" [tw] OR "coronavirus disease 2019" [tw] OR "coronavirus disease 19" [tw] OR "2019 ncov" [tw] OR 2019nCoV [tw] OR "coronavirus 2" [tw] OR "Coronavirus"[Mesh:NoExp] OR "SARS-CoV-2"[Mesh] OR "COVID-19 Testing"[Mesh] OR "COVID-19" [mesh] OR "COVID-19 Vaccines"[Mesh] OR "Receptors, Coronavirus"[Mesh] OR "Spike Glycoprotein, Coronavirus"[Mesh] OR "SARS-CoV-2 variants" [Supplementary Concept]) AND english [lang] NOT (review [pt] OR letter [pt] OR editorial [pt] OR "Systematic Review" [Publication Type] OR "Meta-Analysis" [Publication Type] OR "mental disorders" [mesh] OR depression [ti] OR psychology [sh] OR case reports [pt])

EMBASE

((randomized*:ti,ab,kw  OR randomly:ti,ab,kw  OR randomised:ti,ab,kw  OR randomization:ti,ab,kw  OR randomisation:ti,ab,kw  OR placebo*:ti,ab,kw OR blind*:ti,ab,kw OR pragmatic:ti,ab,kw OR 'randomized controlled trial'/exp OR  'adaptive clinical trial'/exp OR 'controlled clinical trial'/exp) AND ('coronavirus disease 2019'/exp OR 'SARS-Cov-2 variant 501Y.V1'/exp OR 'coronavirus spike glycoprotein'/exp OR 'novel coronavirus':ti,kw,ab OR 'ncov 2019':ti,kw,ab OR 2019nCoV:ti,kw,ab OR 'corono virus':ti,ab,kw OR 'corona virus disease 19':ti,ab,kw OR 'coronavirus 2':ti,ab,kw OR 'covid 19':ti,kw,ab OR 'COVID-19 testing'/exp OR  'SARS-CoV-2 vaccine'/exp OR 'coronavirus receptor'/exp OR ((covid NEAR/2 19):ti,kw,ab) OR ((sars NEAR/2 'cov 2'):ti,kw,ab) OR '2019 ncov':ti,kw,ab OR ((wuhan:ti,kw,ab OR hubei:ti,kw,ab) AND coronavirus*:ti,kw,ab) OR ([2019-2020]/py AND (new:ti,kw,ab OR novel:ti,kw,ab OR pandemic:ti,kw,ab OR epidemic:ti,kw,ab) AND ('coronavirus infection'/exp OR coronavirus*:ti,kw,ab OR coronavirinae/de OR 'corona virus*':ti,kw,ab))) AND 'human'/de AND [embase]/lim NOT ([embase]/lim AND [medline]/lim) AND [english]/lim) NOT ('mental disease'/exp OR depression:ti OR 'systematic review'/exp OR 'review'/exp OR 'editorial'/exp OR 'meta analysis'/exp OR 'case report'/exp OR 'letter'/exp)

COCHRANE 

(randomized OR randomly OR randomised OR randomization OR randomisation OR placebo* OR pragmatic OR blind*) AND ("covid 19" OR covid OR covid19 OR "ncov 2019" OR "novel coronavirus" OR "sars cov 2" OR "sars cov-2" OR "ncov 2019" OR sarscov2 OR (wuhan AND coronavirus*) OR "corona virus*" OR "coronavirus disease 2019" OR "coronavirus disease 19" OR "2019 ncov" OR 2019nCoV OR "coronavirus 2") NOT (depression)

Filter to ("Journal" or Academic Journal"):PT

SECTION: Data Extraction/Data Availability
4. Locating Data Availability Statements

Data availability statements are (typically) short statements that describe how the data and other materials that underlie a given work can be accessed. The ICMJE requires such statements for any published clinical trial.

To find a data availability statement for a given paper:
Check PubMed Central - PMC has a discrete section for data availability statements. The PMCID for each paper will be found using the PMCID - PMID - Manuscript ID - DOI Converter and will be available in your reporting sheet.
Check the supplementary materials. Typically if it's here it'll be labeled "data availability". Papers published in NEJM, JAMA, and other journals will have statements in supplementary materials, so please make sure you check.
For some papers, the statement won't be separated on in PubMed Central or the paper won't be in PubMed Central at all. If that's the case, search for "Data Sharing", "Data Availablity" and variants in the full text (if available).


  
  
For some journals, the statement will be included as a supplementary file.
SECTION: Data Extraction/Data Availability
5. Coding Data Availability Statements - Availability

If you can locate a data availability statement, enter "Yes" in the "data_avail_statement" column of the reporting sheet. Otherwise, enter "No".

Once a data availability statement is located, it should be copied to the  "data_avail_statement_text" column. If you can not find a data availability statement, leave this column blank.

We will delete this column before sharing the data for copyright (??) reasons.
SECTION: Literature Searching
3. Screening 

The deduplication and screening will be done through Covidance. 

All papers will be screened by two reviewers to ensure they meet inclusion/exclusion criteria. In cases where there is a disagreement between reviewers, a third will adjudicate.
SECTION: Finding Preprints
7. Finding Preprints

PubMed has begun to index preprints as part of the NIH Preprints Pilot. Please use the following workflow to try to identify if a paper has a corresponding preprint. 

If a preprint is found:
Enter "Found" in the "preprint_found" column, otherwise enter "Not Found".
Copy the DOI of the preprint in the "preprint_doi" column
Enter the name of the preprint server in the "preprint_server" column.
SECTION: Literature Searching
2. Inclusion and Exclusion Criteria

Inclusion

The trial must be described in a published manuscript (i.e. not a preprint - We'll get to these)
The study must be examining a pharmaceutical intervention designed to prevent or treat COVID-19.
The study design must be a randomized controlled trial (RCT) - Meaning participants must be prospectively and randomly assigned to different groups.


Exclusion

The report is not a published manuscript (i.e. a preprint)
The study examines an intervention that is not designed to prevent or treat COVID-19.
The study examines treatments/procedures that are impacted by COVID but don't directly bear on COVID (e.g. studies of "treating XYZ in the time of COVID").
Published manuscripts that cover studies that are not RCTs (observational studies, retrospective studies, etc)
Meta-analyses/reviews that synthesize the results of previously published trials.
Protocols and other works published in advance of data collection.
SECTION: Data Extraction/Data Availability
6. Coding Data Availability Statements - Mechanism

If the data availability statement indicates that the data is available:

If the data is available via request to an author, put a "yes" in "data_avail_request_author"
If the data is available via request to a study sponsor, IRB, or other entity, put a "yes"  in "data_avail_request_other"
If the data is included in the manuscript and/or supplementary materials, put a  "yes" in  "data_avail_paper"
If the data is available via a data repository, put a  "yes" in "data_avail_repo"
If the data availability statement points to a trial registry, put a  "yes" in "data_avail_registry"
If the data availability statement states that the data is available through some other means, put a "yes"  in "data_avail_other"
If it is unclear how the data is available, put a "yes" in "data_avail_unclear"

Data may be made available through multiple mechanisms, please mark every method that is described in the data availability statement.

If the data availability statement indicates that data is available in a repository, enter the name of the repository in "data_avail_repo_name"


SECTION: Finding Study Info
8. Registration Information

After clicking through the full text of the article, search for the term "registration" and variants.

If you are able to find a registration, enter Yes in the "register" column in the reporting sheet. Otherwise, enter No.
If you are able to find a registration, enter the ID in the "register_id" column.

0.17. If a data availability statement is located, it should be coded. Please do the following when coding each statement.

If the statement includes language that indicates that the data includes sensitive information or statements related to protecting the privacy of the participants, put a "Yes" in the "data_availability_sensitive" column. Otherwise, put a "No".
SECTION: Data Extraction/Data Availability
5.1. Please indicate the availability of data in the "data_availability" column using the following criteria:

If the data is available (by any mechanism) enter "yes"
If the data is explicitly not available enter "no"

Not that sometimes authors will state that data is not available but will then indicate that data is available through a certain mechanism (e.g. "Upon request"). In these cases, mark "yes".

In general, take a very (very) liberal view here. If the statement does not indicate that the data is NOT UNAVAILABLE, we're going to count that as available.
SECTION: Data Extraction/Data Availability
5.2. If the statement indicates that the data will be made available later:

If the information is given- record when it will be made available (in months post-publication) in the "data_later_months" column. 
If the data will be made available after "final publication", put a 1 in the "data_later_publication" column
If the data will be made available after a product approval, put a 1 in the "data_later_product" column.
SECTION: Finding Preprints
7.1. Check to see if the paper has an associated preprint in Europe PMC. The PMCID for the corresponding paper will be in the reporting sheet.

Europe PMC makes linked preprints very obvious, click through the link to identify the preprint server. Here is an example of where this works.

   

SECTION: Finding Preprints
7.2. If no preprint is found using the above method, it may be that it was posted with a slightly different title. Here is an example of a paper and a preprint with different names.

To check, either click through the first author's name to see their publications or create a search string that contains the author's first and last name as well as (SRC:PPR) in the manner shown below. This will show the list of all the author's indexed preprints. 

  
Take a quick scan through the results to see if there any titles that are like the paper's. If there are, put a Yes in the "preprint_possible" column.

We'll check on these later.
17. 
We're not really doing any critical appraisal of these studies, but some information about the research design will potentially be helpful for the later analyses.

Record the intervention/interventions being studied under the "intervention" column.
Record the total number of participants (across all arms) that were included in the study (after exclusion) in the  "sample_size" column.
SECTION: Finding Study Info
8.1. Study Information

Though we're not doing the critical appraisal, some information about the study design will be helpful for our later analyses. In the "intervention" column, record the intervention being studied. We will normalize names (e.g. BNT162b2 vs tozinameran) later