TITLE: How is the practice to use the Risk Of Bias In Non-randomised Studies - of Interventions (ROBINS-I) tool in systematic reviews: a meta-epidemiological study protocol
AUTHORS: Masahiro Banno, Yasushi Tsujimoto, Takashi Ariie, Yuji Okazaki, Yoshinosuke Shimamura, Kyosuke Kamijo, Yuki Kataoka

[DESCRIPTION]
<div class = "text-blocks"><div class = "text-block">ABSTRACT</div><div class = "text-block">Objectives. To evaluate the practice to use the Risk Of Bias In Non-randomised Studies - of Interventions (ROBINS-I) tool in systematic reviews (SRs) including non-randomized studies of interventions (NRSI). </div><div class = "text-block">Methods. This study is a meta-epidemiological study. We will include SRs, which include NRSI, and which used the ROBINS-I tool to assess the risk of bias (RoB) for NRSI. We will exclude SRs if the results have no NRSI even if review authors plan to include NRSI. We will exclude protocols. We will limit SRs published since 2020. We will search PubMed to collect eligible SRs using the search term “ROBINS[tiab]” AND the Systematic Review filter. We will perform random sampling and collect 100 samples from eligible non-Cochrane reviews. We will investigate eligible Cochrane reviews and the 100 non-Cochrane reviews and compare the characteristics. </div><div class = "text-block">Ethics & Dissemination. This study does not need ethical approval. We registered this study protocol. We will publish the findings in a peer-reviewed journal and may present them at conferences. </div><div class = "text-block">Discussion. This study will clarify how is the practice to assess the RoB for non-randomized studies of interventions. The results will inform the researchers, reviewers, and readers of the points that need special attention.</div><div class = "text-block">Registration: Protocols.io</div></div>

[STEPS]
?. [ABSTRACT]
Objectives. To evaluate the practice to use the Risk Of Bias In Non-randomised Studies - of Interventions (ROBINS-I) tool in systematic reviews (SRs) including non-randomized studies of interventions (NRSI). Methods. This study is a meta-epidemiological study. We will include SRs, which include NRSI, and which used the ROBINS-I tool to assess the risk of bias (RoB) for NRSI. We will exclude SRs if the results have no NRSI even if review authors plan to include NRSI. We will exclude protocols. We will limit SRs published since 2020. We will search PubMed to collect eligible SRs using the search term “ROBINS[tiab]” AND the Systematic Review filter. We will perform random sampling and collect 100 samples from eligible non-Cochrane reviews. We will investigate eligible Cochrane reviews and the 100 non-Cochrane reviews and compare the characteristics. Ethics & Dissemination. This study does not need ethical approval. We registered this study protocol. We will publish the findings in a peer-reviewed journal and may present them at conferences. Discussion. This study will clarify how is the practice to assess the RoB for non-randomized studies of interventions. The results will inform the researchers, reviewers, and readers of the points that need special attention.Registration: Protocols.io
?. [BACKGROUND]
Non-randomized controlled studies of interventions (NRSI) are defined as “any quantitative study estimating the effectiveness of an intervention that does not use randomization to allocate units to intervention groups” [1]. NRSI is becoming an important part of systematic reviews (SRs) to investigate long-term outcomes and adverse outcomes. Indeed, 63 % of SR protocols included NRSI [2-4]. Systematic reviewers are encouraged to use the Risk Of Bias In Non-randomised Studies - of Interventions (ROBINS-I) for the assessment of the risk of bias (RoB) in each outcome in each NRSI according to the Cochrane handbook [1, 5, 6]. ROBINS-I is a domain-based assessment tool in which several types of biases to assess separately. The tool has a comprehensive manual to instruct users whereas previous tools for RoB in NRSI, such as the Newcastle-Ottawa tool and the Downs-Black tool, do not meet such characteristics [7, 8]. ROBINS-I is one of the most used RoB tools in SR protocols including NRSI [4].However, there have been several inappropriate uses for the assessment of the risk of bias tools other than ROBINS-I, such as the misuse of outcome-based evaluation for study-based evaluation, and the few sensitivity analyses/ subgroup analyses for the primary outcomes derived from the result of the RoB assessment [9, 10]. We hypothesize that the same may be true for ROBINS-I. The objective of this study is to describe how is the practice to assess RoB with ROBINS-I in SRs including NRSI.
?. [MATERIALS AND METHODS]
This publication is the full study protocol. This protocol has been registered in protocols.io (https://www.protocols.io/).Study designThis study is a meta-epidemiological study. Eligible criteriaWe will include all SRs, which included NRSI, and assessed RoB by ROBINS-I. We will exclude SRs if the results have no NRSI even if review authors plan to include NRSI. We will exclude protocols. We will limit SRs published since 2020 because Cochrane Handbook recommended using ROBINS-I for NRSI since 2019 [1]. Searches and study selectionWe will search PubMed with the keyword “ROBINS[tiab]” and restrict the search using the Systematic Review filter implemented in PubMed. Two review authors will independently screen the title and abstract for the search results. Disagreements will be solved through discussion. If necessary, other review authors will arbitrate disagreements. After excluding some records in screening for title and abstract, we will check the eligibility criteria of full text whether reviews included NRSI and used ROBINS-I. Disagreements will be solved through discussion. If necessary, other review authors will arbitrate disagreements. ROBINS-IROBINS-I has seven domains: (1) bias due to confounding, (2) bias in selection of participants into the study, (3) bias in classification of intervention, (4) bias due to deviations from intended intervention, (5)bias due to missing data, (6) bias in measurement of outcomes, (7) bias in selection of the reported result [5]. The tool has signaling questions. The response options in each domain are (1) yes; (2) probably yes; (3) probably no; (4) no; and (5) no information. These response options of signaling questions will result in RoB value judgments. RoB value judgments are low risk of bias, moderate risk of bias, serious risk of bias, critical risk of bias, and no information [5]. Overall RoB judgments by ROBINS-I will be derived from RoB judgments for each domain as follows. A low risk of bias for overall risk of bias means “the study is judged to be at low risk of bias for all domains” [5]. Moderate risk of bias for overall risk of bias means “the study is judged to be at low or moderate risk of bias for all domains” [5]. Serious risk of bias for overall risk of bias means “the study is judged to be at serious risk of bias in at least one domain, but not at critical risk of bias in any domain” [5]. Critical risk of bias for overall risk of bias means “the study is judged to be at critical risk of bias in at least one domain” [5]. No information for overall risk of bias means “there is no clear indication that the study is at serious or critical risk of bias and there is a lack of information in one or more key domains of bias” [5]. Sample sizeWe will perform random sampling and collect 100 samples from eligible non-Cochrane reviews. We will investigate eligible Cochrane reviews and the 100 non-Cochrane reviews in this study.Data extractionWe will extract the following information: number of authors, impact factors, intervention type (pharmacological, nonpharmacological, both pharmacological and nonpharmacological), number of domains, names of domains, overall ratings for ROBINS-I assessment, units for ROBINS-I assessment, and including the critical risk of bias in a meta-analysis. Table 1 shows detail of the definition of each item. We will also extract other information as shown in Table 1. Two reviewers will independently extract the data. Disagreements will be solved through discussion. If necessary, the third reviewer will arbitrate disagreements. We will limit our study to the results of the ROBINS-I assessment, which first appeared in the paper, and extract the results.The outcome of the present study is the distribution of the variables listed in Table 1 among SRs that used ROBINS-I. Data analysisWe will compare the differences of variables listed in Table 1 between Cochrane and non-Cochrane reviews. We will employ the Mann-Whitney test and Pearson’s chi-squared test for continuous and categorical variables. We will consider two-sided P-value less than 0.05 is statistically significant.We will use Stata ver. 15.1 (StataCorp LLC, College Station, Texas, United States of America).EthicsEthics approval will not be essential because we only use openly available data.
?. [DISCUSSION]
This study will clarify how is the practice to assess the RoB for non-randomized studies of interventions. The results will inform the researchers, reviewers, and readers of the points that need special attention.
?. [ACKNOWLEDGEMENTS]
We would like to thank researchers for developing ROBINS-I.
?. [CONTRIBUTORS]
MB, YT, TA, YO, YS, KK, and YK contributed to the conception and design of the research. MB is fully responsible for writing the protocol. All authors gave final approval of the protocol before submission. After the publication of the protocol, we plan for the following contributions by each author: MB, YT, TA, YO, YS, KK, and YK will conduct screen articles, and extract data. MB will conduct the data analysis. MB, YT, TA, YO, YS, KK, and YK will write the manuscript.
?. [FUNDING]
This protocol was supported by no funder.
?. [COMPETING INTERESTS]
All authors have no competing interests.
?. [Provenance and peer review]
Not peer reviewed.
?. [Patient consent for publication]
Not required.
?. [Data Availability Statement]
We have no additional data.
?. [Table 1. Summary of variables]
Variables  &nbsp;(Category)    Definition    Number of authors    The number of authors in the review.
  We will consider a group name as one author.    Impact factors    Impact factors in the journal in
  which the review was published. We will extract Journal Impact Factor 2020 of
  the journal in the Journal Citation Reports 2021.    Intervention type   (pharmacological,
  nonpharmacological, or both pharmacological and nonpharmacological)    Whether the intervention of the
  review is pharmacological, nonpharmacological intervention, or both
  pharmacological and nonpharmacological intervention.    Number of domains  &nbsp;(0 to 7, 8 or more)    The number of domains of ROBINS-I
  reported in the reviews.    Names of domains  &nbsp;(all seven domains, change/ deletion/ addition, no
  description)    Whether the name of each seven domain
  of ROBINS-I is presented in the results of the reviews.  &nbsp;All seven domains: all seven domains are
  presented in the results of the reviews.  &nbsp;Change/ deletion/ addition: any change of
  the name, deletion, addition of domains is performed.   No
  description: no domain is presented in the results of the reviews.    Overall ratings for ROBINS-I
  assessment   (low,
  moderate, serious, critical, no information, unofficial judgment, or no
  description)     We will refer to the table or traffic
  light plot of the ROBINS-I assessment listed first in the main text. We will
  extract the ROBINS-I assessment listed at the top or left of the table or
  traffic light plot.   Official
  categories of judgment defined in the ROBINS-I detailed guidance (low,
  moderate, serious, critical, no information)&nbsp;
    Unofficial
  judgment: any categories of judgment, such as “high”, that were not defined
  in the original ROBINS-I detailed guidance.&nbsp;
    No
  description: overall risk of bias assessed by ROBINS-I is not presented in
  the review.     Units for ROBINS-I assessment   (outcome, or
  study)     Outcome: one assessment was conducted
  for a specific outcome.&nbsp;&nbsp;   Study: other
  than above. For example, one assessment per study was conducted in the paper
  without stating the subject of the assessment.    Variables  &nbsp;(Category)    Definition    Conflicts of interest for assessors
  of ROBINS-I  &nbsp;(no conflicts of interest, yes, or no)    Whether the way to address the potential
  effect of conflicts of interest for assessors' judgments in ROBINS-I
  assessment was presented in the review.  Yes: the name of the ROBINS-I
  evaluators was listed, and it was stated that the evaluators had no conflicts
  of interest. We will also select “yes” if all authors were listed as having
  no conflicts of interest.  No: other than above.    Including the critical risk of bias
  in a meta-analysis  &nbsp;(yes, or no)    Yes: the overall risk of bias in the
  ROBINS-I assessment included critical risk of bias. In addition,
  meta-analyses for any outcomes including the study with critical risk of bias
  were being conducted.  &nbsp;No: other than above. For example, whether
  the overall risk of bias in the ROBINS-I assessment included critical risk of
  bias was unclear, details of meta-analyses were unclear, or meta-analyses
  were not being conducted.    Sensitivity analyses, subgroup
  analyses, or meta-regression using the ROBINS-I assessment  &nbsp;(yes, or no)    We will search only the main text and
  supplementary files.   Whether the
  results of sensitivity analyses, subgroup analyses, or meta-regression for
  the primary outcomes using the result of the ROBINS-I assessment are
  presented in results. We will select "yes" if planned for
  sensitivity analyses/ subgroup analyses/ meta-regression but not performed
  due to the limited data.&nbsp;     Interrater reliability (yes, or no)    Whether interrater reliability for ROBINS-I
  assessment such as a kappa statistic was reported.    Discussing the ROBINS-I assessment  &nbsp;(yes, or no)    Whether results for ROBINS-I
  assessment were interpreted in discussion (e.g., in strength or limitations).    Variables  &nbsp;(Category)    Definition    Number of included trials in an
  outcome  &nbsp;    The number of included trials in the
  target outcome. The target outcome means the outcome whose result of the
  ROBINS-I assessment the earliest appeared in the main text of the review.    Judgment for each domain  &nbsp;(yes, or no)    Whether judgments regarding all seven
  domains of ROBINS-I in the target outcome were presented in the review (e.g.,
  in traffic light plots or tables).     Supporting comments for each domain
  rating  &nbsp;(yes, or no)    Whether comments supporting judgments
  regarding all seven domains of ROBINS-I in the target outcome were presented
  in the review (e.g., in tables). We will decide "no" if comments
  supporting judgments regarding any domains of ROBINS-I were not presented in
  the review.    Judgment for each signaling question  &nbsp;(yes, or no)    Whether the answers to all signaling
  questions of ROBINS-I in the target outcome were presented.     Supporting comments for each
  signaling question judgment  &nbsp;(yes, or no)    Whether supporting comments for
  answers to all signaling questions of ROBINS-I in the target outcome were
  presented in the review (e.g., in tables).     Confounding factors  &nbsp;(yes, or no)    Whether confounding factors for the
  target outcome were presented.     Co-interventions  &nbsp;(yes, or no)    Whether co-interventions for the
  target outcome were presented.     Effects of interest  &nbsp;(intention-to-treat effect, per-protocol
  effect, or no description)    Whether investigating the
  intention-to-treat effect or the per-protocol effect by ROBINS-I assessment
  in the target outcomes was presented in the review.
Variables  &nbsp;(Category)    Definition    Number of authors    The number of authors in the review.
  We will consider a group name as one author.    Impact factors    Impact factors in the journal in
  which the review was published. We will extract Journal Impact Factor 2020 of
  the journal in the Journal Citation Reports 2021.    Intervention type   (pharmacological,
  nonpharmacological, or both pharmacological and nonpharmacological)    Whether the intervention of the
  review is pharmacological, nonpharmacological intervention, or both
  pharmacological and nonpharmacological intervention.    Number of domains  &nbsp;(0 to 7, 8 or more)    The number of domains of ROBINS-I
  reported in the reviews.    Names of domains  &nbsp;(all seven domains, change/ deletion/ addition, no
  description)    Whether the name of each seven domain
  of ROBINS-I is presented in the results of the reviews.  &nbsp;All seven domains: all seven domains are
  presented in the results of the reviews.  &nbsp;Change/ deletion/ addition: any change of
  the name, deletion, addition of domains is performed.   No
  description: no domain is presented in the results of the reviews.    Overall ratings for ROBINS-I
  assessment   (low,
  moderate, serious, critical, no information, unofficial judgment, or no
  description)     We will refer to the table or traffic
  light plot of the ROBINS-I assessment listed first in the main text. We will
  extract the ROBINS-I assessment listed at the top or left of the table or
  traffic light plot.   Official
  categories of judgment defined in the ROBINS-I detailed guidance (low,
  moderate, serious, critical, no information)&nbsp;
    Unofficial
  judgment: any categories of judgment, such as “high”, that were not defined
  in the original ROBINS-I detailed guidance.&nbsp;
    No
  description: overall risk of bias assessed by ROBINS-I is not presented in
  the review.     Units for ROBINS-I assessment   (outcome, or
  study)     Outcome: one assessment was conducted
  for a specific outcome.&nbsp;&nbsp;   Study: other
  than above. For example, one assessment per study was conducted in the paper
  without stating the subject of the assessment.
Variables  &nbsp;(Category)    Definition    Conflicts of interest for assessors
  of ROBINS-I  &nbsp;(no conflicts of interest, yes, or no)    Whether the way to address the potential
  effect of conflicts of interest for assessors' judgments in ROBINS-I
  assessment was presented in the review.  Yes: the name of the ROBINS-I
  evaluators was listed, and it was stated that the evaluators had no conflicts
  of interest. We will also select “yes” if all authors were listed as having
  no conflicts of interest.  No: other than above.    Including the critical risk of bias
  in a meta-analysis  &nbsp;(yes, or no)    Yes: the overall risk of bias in the
  ROBINS-I assessment included critical risk of bias. In addition,
  meta-analyses for any outcomes including the study with critical risk of bias
  were being conducted.  &nbsp;No: other than above. For example, whether
  the overall risk of bias in the ROBINS-I assessment included critical risk of
  bias was unclear, details of meta-analyses were unclear, or meta-analyses
  were not being conducted.    Sensitivity analyses, subgroup
  analyses, or meta-regression using the ROBINS-I assessment  &nbsp;(yes, or no)    We will search only the main text and
  supplementary files.   Whether the
  results of sensitivity analyses, subgroup analyses, or meta-regression for
  the primary outcomes using the result of the ROBINS-I assessment are
  presented in results. We will select "yes" if planned for
  sensitivity analyses/ subgroup analyses/ meta-regression but not performed
  due to the limited data.&nbsp;     Interrater reliability (yes, or no)    Whether interrater reliability for ROBINS-I
  assessment such as a kappa statistic was reported.    Discussing the ROBINS-I assessment  &nbsp;(yes, or no)    Whether results for ROBINS-I
  assessment were interpreted in discussion (e.g., in strength or limitations).
Variables  &nbsp;(Category)    Definition    Number of included trials in an
  outcome  &nbsp;    The number of included trials in the
  target outcome. The target outcome means the outcome whose result of the
  ROBINS-I assessment the earliest appeared in the main text of the review.    Judgment for each domain  &nbsp;(yes, or no)    Whether judgments regarding all seven
  domains of ROBINS-I in the target outcome were presented in the review (e.g.,
  in traffic light plots or tables).     Supporting comments for each domain
  rating  &nbsp;(yes, or no)    Whether comments supporting judgments
  regarding all seven domains of ROBINS-I in the target outcome were presented
  in the review (e.g., in tables). We will decide "no" if comments
  supporting judgments regarding any domains of ROBINS-I were not presented in
  the review.    Judgment for each signaling question  &nbsp;(yes, or no)    Whether the answers to all signaling
  questions of ROBINS-I in the target outcome were presented.     Supporting comments for each
  signaling question judgment  &nbsp;(yes, or no)    Whether supporting comments for
  answers to all signaling questions of ROBINS-I in the target outcome were
  presented in the review (e.g., in tables).     Confounding factors  &nbsp;(yes, or no)    Whether confounding factors for the
  target outcome were presented.     Co-interventions  &nbsp;(yes, or no)    Whether co-interventions for the
  target outcome were presented.     Effects of interest  &nbsp;(intention-to-treat effect, per-protocol
  effect, or no description)    Whether investigating the
  intention-to-treat effect or the per-protocol effect by ROBINS-I assessment
  in the target outcomes was presented in the review.
?. [References]
[1] Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al. Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). 2021 [accessed March 18 2021].[2] Hummel N, Debray T, Didden E-Mea. Methodological guidance, recommendations and illustrative case studies for (network) meta-analysis and modelling to predict real-world effectiveness using individual participant and/or aggregate data. https://www.imi-getreal.eu/Portals/1/2017-03-30%20-%20WP4%20-%20Methodological%20guidance%2C%20recommendations%20and%20illustrative%20case%20studies.pdf; 2017 [accessed May 17, 2021.[3] Golder S, Loke YK, Bland M. Meta-analyses of adverse effects data derived from randomised controlled trials as compared to observational studies: methodological overview. PLoS Med 2011;8:e1001026. 10.1371/journal.pmed.1001026.[4] Farrah K, Young K, Tunis MC, Zhao L. Risk of bias tools in systematic reviews of health interventions: an analysis of PROSPERO-registered protocols. Syst Rev 2019;8:280. 10.1186/s13643-019-1172-8.[5] Sterne JA, Hernan MA, Reeves BC, Savovic J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 2016;355:i4919. 10.1136/bmj.i4919.[6] Sterne JAC, Savovic J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 2019;366:l4898. 10.1136/bmj.l4898.[7] Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health 1998;52:377-84. 10.1136/jech.52.6.377.[8] Wells G, Shea B, O’Connell D, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp; 2008 [accessed May 15, 2021.[9] Minozzi S, Cinquini M, Gianola S, Gonzalez-Lorenzo M, Banzi R. The revised Cochrane risk of bias tool for randomized trials (RoB 2) showed low interrater reliability and challenges in its application. J Clin Epidemiol 2020;126:37-44. 10.1016/j.jclinepi.2020.06.015.[10] Jorgensen L, Paludan-Muller AS, Laursen DR, Savovic J, Boutron I, Sterne JA, et al. Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials: overview of published comments and analysis of user practice in Cochrane and non-Cochrane reviews. Syst Rev 2016;5:80. 10.1186/s13643-016-0259-8.