TITLE: A General Protocol for Pilot-Testing the Screening Stage of a Systematic Review (Manual)
AUTHORS: Paul Whaley, Ruth Garside, Jacqualyn  Eales

[DESCRIPTION]
<div class = "text-blocks"><div class = "text-block">Studies of potential relevance to the objectives of a systematic reviews, as found by the search strategy, should be screened in duplicate for their eligibility. This screening process should be piloted. This protocol provides a step-by-step formula for piloting the screening process of a systematic review. The formula is also applicable to the conduct of systematic evidence maps.</div><div class = "text-block">There are generally three reasons for piloting the screening process of a systematic review:</div><div class = "text-block"><ol style = "list-style-type: decimal;"><li style = "counter-reset:ol0;">Are the eligibility criteria expressed clearly enough?</li><li style = "counter-reset:ol0;">Are the screeners on the review team interpreting the criteria consistently?</li><li style = "counter-reset:ol0;">Are there papers which should be included in the review but had not been anticipated as being relevant?</li></ol></div><div class = "text-block">This protocol presents a process for answering each of these questions in the affirmative and preparing to ensure that the screening step of the systematic review is a success, i.e. it does not result in exclusion of evidence which is relevant to the research question.</div></div>

[STEPS]
?. [Preparing for Training]
Define the initial set of eligibility criteria in discussion with topic experts, attempting to be as clear and unambiguous as possible.
?. [Preparing for Training]
Identify the reviewers who will conduct the screening process.
?. [Preparing for Training]
Select a random subset of the full set of studies to use as the training set, which is large enough that it will be representative of the topic areas in the full set.
This is particularly important when the reviewers have different backgrounds and areas of topic expertise. It might be necessary to divide the training set into multiple sub-sets to allow e.g. modified criteria to be tested on an unseen set of studies.
?. [Doing the Training]
Screen the studies in the training set. Each study should be assessed by all the reviewers who are conducting the screening process.
?. [Doing the Training]
Discuss each disagreement and difficult decision between the reviewers who are conducting the screening.
?. [Preparing for Training]
Select the software to be used for the screening process.
?. [Doing the Training]
Come to a consensus on each decision, using the disagreement resolution process defined above.
?. [Doing the Training]
Clarify and add detail to the eligibility criteria as required.
?. [Doing the Training]
Determine whether the success conditions for completing the training process have been fulfilled.
?. [Completing]
Finish. Document what was done in each of the above steps.
In the review itself, it is important to continue to check consistency of decisions throughout the screening process, particularly for reviews which screen large numbers of studies.
?. [Preparing for Training]
Define the success conditions for the training process.
The "success conditions" are the point at which the training process will have been considered sufficiently successful that it can be considered complete and the systematic review can proceed.
?. [Revising]
Return to Step 7.
?. [Preparing for Training]
Define the process for resolving disagreements in screening decisions. If this is arbitration by a third party, identify who this will be.