TITLE: Standards-Compliant General Protocol for Systematic Reviews
AUTHORS: Paul   Whaley, Stephen M Wattam

[DESCRIPTION]
This general protocol is designed to support the planning of SRs that are concerned with the effects of environmental exposures on human health outcomes, or the prevention or mitigation thereof.

It has been designed specifically to help researchers follow the COSTER recommendations for the conduct of systematic reviews, in a way that complies with the ROSES and PRISMA-P checklists for reporting SR protocols, PRISMA-S for search strategies, and provides data for PROSPERO records.

Note that due to some inconsistencies between the standards and checklists, not every element of every checklist is covered. A cross-walk of the standards and checklists will be provided with forthcoming protocol documentation. Code is being written to help generate automated draft protocol reports.

If you use the protocol in a systematic review, please cite not only this instance of the protocol template, but also the parent COSTER manuscript, DOI 10.1016/j.envint.2020.105926.

Training in use of the protocol is available, and comments are welcome. Please contact the author, Paul Whaley, for more information.

[BEFORE_START]
Note on standards compliance: Simply following this template does not guarantee the same degree of compliance with all steps of the SR protocol development process. This is because some steps not only require something to be done, but for it to be done well. Compare, for example, selecting team members to fill competencies (step 2) with defining "appropriate" eligibility criteria (step 9). Author teams are recommended to implement some form of external auditing in their protocol development process, to help ensure that the planned methods are robust (see step 27).

[GUIDELINES]
Protocols.io has not yet been optimised as a means for reporting what was done in response to complex instructions such as those found in this protocol. Feedback on use of the protocol, and how to develop it to facilitate reporting of planned methods, would be very much appreciated.

[STEPS]
SECTION: A. Securing capacity, competencies, and tools
1. [osf_record] Create an Open Science Framework (OSF) record for the project. 
PRISMA-P 1a, 1b / PROSPERO 1

Create an OSF record for the project (this template may help https://osf.io/zjevp/ - click the fork button, then click "duplicate template", and rename the template with your project name).
Name the protocol "XYZ: a systematic review protocol" or "XYZ: a protocol for an update of a systematic review", as appropriate
Paste the OSF record URL into the box below.

Mark the step as complete when the OSF record URL has been inserted below.
 
.
 

SECTION: A. Securing capacity, competencies, and tools
2. [competencies] Assign team members to core systematic review competencies. 
COSTER 1.1.1 / PRISMA-P 3b 

Assign a team member to each of the competencies listed below. 

Mark the step as complete when at least one team member has been assigned to each competency.
 

SECTION: A. Securing capacity, competencies, and tools
3. [info_manage] Identify information management practices and tools for each stage of the review. 
COSTER 1.1.2. / PRISMA-P 11a / PROSPERO 26

State the software and relevant packages to be used in the review, using the form below.

Mark the step as complete when all the software to be used in the review has been catalogued.
 

SECTION: A. Securing capacity, competencies, and tools
4. [funding] State the funding sources of the systematic review.
PRISMA-P 5a, 5b, 5c / PROSPERO 12

Mark the step as complete when information about funding sources has been provided.
 

SECTION: A. Securing capacity, competencies, and tools
5. [interests] List the potential conflicts of interest of the authors, and manage any identified apparent conflicts of interest. 
COSTER 1.1.3 and 1.1.4. / ROSES for SR Protocols item 40 / PROSPERO 13

Create a .docx file declaring the financial and non-financial interests of each of the contributors to the SR (a template that may be useful is here: https://osf.io/ejcyw/)
Name each file "DisclosureOfInterest_LastnameFirstname"
Evaluate the contributor interests for apparent conflicts
Exclude from decision-making processes authors who have apparent conflicts of interest
Summarise the apparent conflicts in a file named "ApparentConflicts" (even if it is to declare none were identified), including steps taken to remove authors with apparent conflicts from relevant decision-making roles

Mark the step as complete after a full set of author-completed Conflict of Interest Disclosure forms have been uploaded to the OSF record (#2), and the summary of apparent conflicts document has been uploaded (#5).
 


SECTION: B. Setting the research question (&quot;problem formulation&quot;)
6. [need] Demonstrate the need for a new review. 
COSTER 1.2.1 / PRISMA-P item 6 / ROSES for SR Protocols items 6 and 7

Create a .docx format document called "DemonstrationOfNeed" (a template that may be useful is here: https://osf.io/vp7ak/)
Describe the scientific value of the question(s), i.e. why it is important that it be investigated.
Describe the importance to stakeholders and/or affected parties of the question(s) being asked
Summarise existing primary research and reviews that justify conducting a new systematic review
Upload the document to the OSF record

Mark the step as complete when a document detailing the above items has been uploaded to the OSF record.
 

SECTION: B. Setting the research question (&quot;problem formulation&quot;)
7. [rationale] Articulate the scientific rationale for each question via development of a theoretical framework.
COSTER 1.2.2. / PRISMA-P 6 / ROSES for SR Protocols item 6

Create a .docx format document called "TheoreticalFramework"
Present the theoretical framework for each question in the document.
Upload the document to the OSF record

Mark the step as complete when the theoretical framework document has been uploaded to the OSF record.
 

SECTION: B. Setting the research question (&quot;problem formulation&quot;)
8. [question] State the research question or questions to be answered by the systematic review. For each research question, define a statement of the research objective in terms of Population, Exposure or Intervention, Comparator, and Outcome. 
COSTER 1.2.3. / PRISMA-P 7 / ROSES for SR Protocols items 7 and 8

Define the PECO statement or equivalent via completing each substep below.

Mark this step as complete when all relevant substeps are complete.
 

SECTION: B. Setting the research question (&quot;problem formulation&quot;)
8.1. [q_pop] Define the target Population of interest. These are the entities which are being subjected to the exposures or interventions of interest. 

Mark this step as complete when relevant population information has been provided.
 

SECTION: B. Setting the research question (&quot;problem formulation&quot;)
8.2. [q_exp] Define the target Exposure or Intervention of interest. This concerns the administered or observed change in conditions of the objects of investigation. It should include timing, duration and dose.

Exposure: What is the exposure or intervention?
Timing: When does the exposure or intervention happen?
Duration: For how long does the exposure or intervention last?
Dose: What is the dose regimen (amount, frequency)?
Additional: Are there any other relevant characteristics of the exposure or intervention?

Mark this step as complete when relevant exposure / intervention information has been entered.
 

SECTION: B. Setting the research question (&quot;problem formulation&quot;)
8.3. [q_comp] Define the target Comparator of interest. This concerns the characteristics of the exposure or intervention being used as the comparator to which the target exposure or intervention is being compared.

Mark this step as complete when relevant comparator information has been entered.
 

SECTION: B. Setting the research question (&quot;problem formulation&quot;)
8.4. [q_outcome] Define the target Outcome(s) of interest. This concerns the change being measured in the exposure or intervention group. These should be the primary outcomes of interest to the systematic review which form the hypothesis or hypotheses being tested. Secondary outcomes can also be listed.

Mark this step as complete when the primary and secondary outcomes have been entered.
 

SECTION: C. Defining the eligibility criteria and designing the process for screening evidence for inclusion
9. [eligibility] Define and justify unambiguous and appropriate eligibility criteria for each component of the objective statement. 
COSTER section 1.3 / PRISMA-P 8 / ROSES item 20

Create a .docx format document named "EligibilityCriteria"
Provide a complete list of design and reporting criteria criteria for deciding whether to include or exclude a study (the EligibilityCriteria template on the template OSF archive may be useful)
Upload the document to the OSF record

Mark the step as complete when the EligibilityCriteria document has been uploaded to the OSF record.
 

SECTION: C. Defining the eligibility criteria and designing the process for screening evidence for inclusion
10. [screening] Define the method for screening each piece of evidence for inclusion. COSTER 3.1 recommends this be conducted by at least two people working independently, with an appropriate process (e.g. third-party arbitration) for identifying and settling disputes. 
COSTER 3.1 / PRISMA-P 11b / ROSES 18, 19, 20

Mark this step as complete when the screening methodology has been described below.
 

SECTION: C. Defining the eligibility criteria and designing the process for screening evidence for inclusion
11. [multi_pubs] Define the strategy for handling multiple publications of the same study (e.g. multiple publications, conference abstracts etc.). 
COSTER 3.4

Mark this step as complete when the multiple-publication strategy has been described.
 
.
 


SECTION: D. Defining the strategy for searching for evidence relevant to the review objectives
12. [search] Design sufficiently sensitive search strategies, so that studies which meet the eligibility criteria of the review are not inadvertently excluded. Document the search methods in sufficient detail to render them transparent and reproducible. 
COSTER 1.4.1, 2.6 / PROSPERO 17 / PRISMA-P 9, 10

Create a .docx format document named "LiteratureSearch" (or use the template provided at https://osf.io/2ysk3)
Complete each of the sub-steps below
Upload the complete document to the OSF record

Mark this step as complete when all relevant sub-steps are complete and the LiteratureSearch document has been uploaded to the OSF Record.
 

SECTION: D. Defining the strategy for searching for evidence relevant to the review objectives
12.1. [s_databases] Search all the key scientific databases for the topic, including national, regional and subject-specific databases.
COSTER 2.1 / PRISMA-P 9 / ROSES 9

List the databases, platforms, and study registries being searched in the LiteratureSearch document

Mark this step as complete when this information is added to the LiteratureSearch document.
SECTION: D. Defining the strategy for searching for evidence relevant to the review objectives
12.2. [s_strings] Define search strings for each database, electronic and other source, using appropriate controlled vocabulary, free-text terms and logical operators in a manner which prioritises sensitivity.  
COSTER 2.3, 2.6 / PRISMA-P 10 / ROSES 10

Provide a complete search string for each database, platform, and registry in the LiteratureSearch document

Mark this step as complete when this information is added to the LiteratureSearch document.
 

SECTION: D. Defining the strategy for searching for evidence relevant to the review objectives
12.3. [s_grey] Define reproducible strategies for identifying and searching sources of grey literature (databases, websites etc.). Document the search methods and results in sufficient detail to render them transparent and reproducible. 
COSTER 2.2 and 2.6 / ROSES 9, 15 / PRISMA-P 10

Provide a complete search method for each grey literature source in the LiteratureSearch document

Mark this step as complete when this information is added to the LiteratureSearch document.
SECTION: D. Defining the strategy for searching for evidence relevant to the review objectives
12.4. [s_hand] Search within the reference lists of included studies and other reviews relevant to the topic (“hand-searching”) and consider searching in the reference lists of documents which have cited included studies. 
COSTER 2.4 / PRISMA-P 10 / ROSES 9

Provide a citation searching method in the LiteratureSearch document

Mark this step as complete when this information is added to the LiteratureSearch document.
SECTION: D. Defining the strategy for searching for evidence relevant to the review objectives
12.5. [s_contact] Search by contacting relevant individuals and organisations. 
COSTER 2.5 / PRISMA-P 10 / ROSES 9

Provide information about how individuals and organisations will be contacted for relevant evidence in the LiteratureSearch document

Mark this step as complete when this information is added to the LiteratureSearch document.
SECTION: D. Defining the strategy for searching for evidence relevant to the review objectives
13. [rerun_search] Plan for re-running all searches and screen the results for potentially eligible studies within 12 months prior to publication of the review (screening at least at the level of title plus abstract). 
COSTER 2.7 / ROSES 17

Mark this step as complete when the search re-run plan has been described in the boxes below.
 

SECTION: E. Methods for Appraising Individual Studies
14. [rob] Define the risk of bias assessment methods to be used for evaluating the internal validity of the included research. If observational studies are included, this should cover identification of plausible confounders. 
COSTER 1.4.3 / PRISMA-P 14 / ROSES 22 / PROSPERO 27

Create a .docx format document named "BiasAssessment" (or use the template provided at https://osf.io/h3egq/) 
Complete each of the sub-steps below
Upload the complete document to the OSF record

Mark this step as complete when all relevant sub-steps are completed and the BiasAssessment document has been uploaded to the OSF Record.
 

SECTION: E. Methods for Appraising Individual Studies
14.1. [rob_roles] Conduct risk of bias assessment with at least two people working independently, with an appropriate process (e.g. third-party arbitration) for identifying and settling disputes. 
COSTER 5.3 / PROSPERO 27
 

SECTION: E. Methods for Appraising Individual Studies
14.2. [rob_tool] Select and modify as necessary a risk of bias assessment instrument.
COSTER 1.4.3 / PRISMA-P 14 / ROSES 22 / PROSPERO 27

Add tool selection and modification information to the BiasAssessment document

Mark this step as complete when this information is added to the BiasAssessment document.
 

SECTION: E. Methods for Appraising Individual Studies
14.3. [rob_summary] Define how the results of the risk of bias assessment will be summarised for each included study.
COSTER section 5 / PRISMA-P 14 / ROSES 23 

Add this information to the BiasAssessment document

Mark this step as complete when this information is added to the BiasAssessment document.
SECTION: E. Methods for Appraising Individual Studies
14.4. [rob_use] Define how the results of the risk of bias assessment will be used in developing the results and findings of the systematic review.
COSTER section 5 / PRISMA-P 14 / ROSES 23 / PROSPERO 27

Add this information to the BiasAssessment document

Mark this step as complete when this information is added to the BiasAssessment document.
SECTION: E. Methods for Appraising Individual Studies
15. [other_appr] Define any other methods for critically appraising the included studies, any instruments for so doing, how the results of the appraisal of each study will be summarised, and how the results will be used in developing the results and findings of the systematic review.
COSTER 7.8 / ROSES 22 

Create a .docx format document named "OtherAppraisal" (or use the template provided at https://osf.io/2z6bf) 
Complete each section of the template (it follows the pattern of the BiasAssessment document)
Upload the complete document to the OSF record

Mark this step as complete when the OtherAppraisal document is uploaded to the OSF record (or skip this step if not relevant to the SR).
SECTION: F. Evidence Synthesis Methods
16. [char_incl] Design the "characteristics of included studies" table. 
COSTER 1.4.2

Create a .docx format document named "CharacteristicsOfStudies" (or use the template provided at https://osf.io/h9sd8) 
Provide information about the data to be presented in the characteristics of included studies table
Upload the complete document to the OSF record

Mark this step as complete when the CharacteristicsOfStudies document is uploaded to the OSF record.
 

SECTION: F. Evidence Synthesis Methods
17. [extract_forms] Design the data extraction forms. 
COSTER 1.4.7 / PRISMA-P 12

Design the data extraction forms
Create a .docx format document named "ExtractionForms"
Paste screenshots of the forms into the document
Upload the complete document to the OSF record

Mark this step as complete when the ExtractionForms document is uploaded to the OSF record.
SECTION: F. Evidence Synthesis Methods
18. [synth] Design the methods for synthesising the included studies. 
COSTER 1.4.4, section 6 / ROSES 30-35 / PRIMSA-P 15-16

This should cover: 
qualitative and quantitative methods (with full consideration given to synthesis methods to be used when meta-analysis is not possible); 
assessment of heterogeneity; 
choice of effect measure (e.g. RR, OR etc.); 
methods for meta-analysis and other quantitative synthesis; 
pre-defined, appropriate effect modifiers for sub-group analyses.


Create a .docx format document named "SynthesisMethods" (or use the template provided at https://osf.io/8a43q) 
Describe in the document the synthesis methods to be used (these are complex, so use of the template is recommended)
Upload the complete document to the OSF record.

Mark this step as complete when the SynthesisMethods document is uploaded to the OSF record.
 

SECTION: F. Evidence Synthesis Methods
19. [certainty] Define the methods for determining how, given strengths and limitations of the overall body of evidence, confidence in the results of the synthesis of the evidence for each outcome is to be captured and expressed. (For reviews which include multiple streams of evidence, this may need to be defined for each stream.) 
COSTER 1.4.5 / ROSES 22 / PRISMA-P 17

Create a .docx format document named "CertaintyAssessment" (or use the template provided at https://osf.io/vh29q/) 
Describe in the document the certainty assessment methods to be used
Upload the complete document to the OSF record.

Mark this step as complete when the CertaintyAssessment document is uploaded to the OSF record.
SECTION: F. Evidence Synthesis Methods
20. [integration] For reviews which include multiple streams of evidence (e.g. animal and human studies), define the methods for integrating the individual streams into an overall result. 
COSTER 1.4.6

This should include a description of the relative relevance of populations (e.g. species, age, comorbidities etc.), exposures (e.g. timing, dose), and outcomes (direct or surrogate, acute or chronic model of disease, etc.), as appropriate, per which inferences about predicted effects in target populations can be made from observed effects in study populations.

Create a .docx format document named "EvidenceIntegration" (or use the template provided at https://osf.io/gc8ep/) 
Describe in the document the evidence integration methods to be used
Upload the complete document to the OSF record.

Mark this step as complete when the EvidenceIntegration document is uploaded to the OSF record.

SECTION: G. Pilot Testing (COSTER 1.4.7)
21. [pilot_screening] Pilot test the screening process. Make any necessary changes to the documentation of the screening process, described from  , as needed (Section C). 
COSTER 1.4.7

A general protocol for piloting the screening stage of a systematic review is available here:
https://www.protocols.io/view/a-general-protocol-for-pilot-testing-the-screening-bkc9ksz6

Mark this step as complete when the screening process has been piloted and changes to methods finalised.
 

SECTION: G. Pilot Testing (COSTER 1.4.7)
22. [pilot_extraction] Pilot test data extraction forms. Make changes to the documentation of the data extraction process, described in  , as needed. 
COSTER 1.4.7 / ROSES 25, 26 / PRISMA-P 11c 

Mark this step as complete when the screening process has been piloted and changes to methods finalised.
 

SECTION: G. Pilot Testing (COSTER 1.4.7)
23. [pilot_rob] Pilot test the risk of bias assessment. Make changes to the documentation of the risk of bias assessment process, described in  , as needed.
COSTER 1.4.7

Mark this step as complete when the risk of bias assessment methodology has been piloted and changes to methods finalised.
SECTION: G. Pilot Testing (COSTER 1.4.7)
24. [pilot_certainty] Pilot the process for the assessment of confidence in the results of the synthesis of the evidence. Make changes to the documentation of the certainty assessment process, described in  , as needed
COSTER 1.4.7

Mark this step as complete when the certainty assessment methodology has been piloted and changes to methods finalised.
SECTION: H. Registering the protocol
25. [write] Create a protocol document from the steps and documents created above.
 

SECTION: H. Registering the protocol
26. [registration] Create a permanent public record of intent to conduct the review (e.g. by registering the protocol in an appropriate registry). Do this prior to conducting the literature search. 
COSTER 1.5.1 / PRISMA-P 2 

Mark this step as complete when a unique identifier and link to the registration record has been added below.
 

SECTION: I. Publishing the protocol
27. [peer_review] As appropriate for review planning and question formulation, secure peer-review and public feedback on the draft version of the protocol, incorporating comments into the final version of the protocol. 
COSTER 1.5.2

Mark this step as complete when peer-review comments have been incorporated into the protocol.
SECTION: I. Publishing the protocol
28. [publication] Publish the final version of the protocol in a public archive. Do this prior to screening studies for inclusion in the review. This should be a read-only, date-stamped record on a third-party controlled website.
COSTER 1.5.3

Mark this step as complete when a unique identifier and registration record has been added below.
 

Well done! You have completed your SR protocol and can start screening the literature and following your meticulously-planned methods!