# Impact of a short version of the CONSORT checklist for peer reviewers to improve the reporting of randomised controlled trials published in biomedical journals: study protocol for a randomised controlled trial

Benjamin Speich, Sara Schroter, Matthias Briel, David Moher, Iratxe Puebla, Alejandra Clark, Michael Maia Schlüssel, Philippe Ravaud, Isabelle Boutron, Sally Hopewell

► To cite this version:

Benjamin Speich, Sara Schroter, Matthias Briel, David Moher, Iratxe Puebla, et al. Impact of a short version of the CONSORT checklist for peer reviewers to improve the reporting of randomised controlled trials published in biomedical journals: study protocol for a randomised controlled trial. BMJ Open, 2020, 10 (3), pp.e035114. 10.1136/bmjopen-2019-035114. hal-02985929

HAL Id: hal-02985929

https://hal.inrae.fr/hal-02985929

Submitted on 9 Nov 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# BMJ Open Impact of a short version of the CONSORT checklist for peer reviewers to improve the reporting of randomised controlled trials published in biomedical journals: study protocol for a randomised controlled trial

Benjamin Speich¹,¹,² Sara Schroter¹,³ Matthias Briel,²,⁴ David Moher¹,⁵,⁶ Iratxe Puebla¹,⁷ Alejandra Clark,⁷ Michael Maia Schlüssel¹,⁸ Philippe Ravaud,⁹,¹⁰ Isabelle Boutron,⁹,¹⁰ Sally Hopewell¹

**To cite:** Speich B, Schroter S, Briel M, et al. Impact of a short version of the CONSORT checklist for peer reviewers to improve the reporting of randomised controlled trials published in biomedical journals: study protocol for a randomised controlled trial. *BMJ Open* 2020;10:e035114. doi:10.1136/bmjopen-2019-035114

► Prepublication history and additional material for this paper are available online. To view these files, please visit the journal online (http://dx.doi.org/10.1136/bmjopen-2019-035114).

Received 21 October 2019
Revised 05 February 2020
Accepted 10 February 2020

## ABSTRACT

**Introduction** Transparent and accurate reporting is essential for readers to adequately interpret the results of a study. Journals can play a vital role in improving the reporting of published randomised controlled trials (RCTs). We describe an RCT to evaluate our hypothesis that asking peer reviewers to check whether the most important and poorly reported CONSOLIDATED Standards of Reporting Trials (CONSORT) items are adequately reported will result in higher adherence to CONSORT guidelines in published RCTs.

**Methods and analysis** Manuscripts presenting the primary results of RCTs submitted to participating journals will be randomised to either the intervention group (peer reviewers will receive a reminder and short explanation of the 10 most important and poorly reported CONSORT items; they will be asked to check if these items are reported in the submitted manuscript) or a control group (usual journal practice). The primary outcome will be the mean proportion of the 10 items that are adequately reported in the published articles. Peer reviewers and manuscript authors will not be informed of the study hypothesis, design or intervention. Outcomes will be assessed in duplicate from published articles by two data extractors (at least one blinded to the intervention). We will enrol eligible manuscripts until a minimum of 83 articles per group (166 in total) are published.

**Ethics and dissemination** This pragmatic RCT was approved by the Medical Sciences Interdivisional Research Ethics Committee of the University of Oxford (R62779/RE001). If this intervention is effective, it could be implemented by all medical journals without requiring large additional resources at journal level. Findings will be disseminated through presentations in relevant conferences and peer-reviewed publications. This trial is registered on the Open Science Framework (https://osf.io/c4hn8).

## Strengths and limitations of this study

► Pragmatic randomised controlled trial (RCT) with individual randomisation of real manuscripts describing RCTs submitted to a variety of journals.

► Main outcomes will be assessed from publicly available sources (ie, published articles).

► If this simple intervention is effective, it could be implemented by journals without requiring large additional resources at journal level.

► The intervention could not be included within the email from journal with the link to the manuscript for review, risking peer reviewers will potentially ignore the separate email containing the CONSOLIDATED Standards of Reporting Trials reminder.

## INTRODUCTION

### Background and rationale

There is substantial agreement that well conducted and reported randomised controlled trials (RCTs) generate the most trustworthy evidence when evaluating newly developed or existing clinical interventions.¹⁻³ For clinicians, scientists and decision makers, published articles are often the only way to know how a study was conducted. In order to judge the internal and external validity of RCTs, it is crucial that these articles present transparent, accurate and unbiased information about the methods and conduct of the RCT.

To improve the quality and transparency of clinical and epidemiological research, the EQUATOR (Enhancing the Quality and Transparency of Research) Network was founded in 2006 and officially launched in 2008.⁴⁻¹⁰ This international network, which

<figure><img src="image_2.png" /></figure>

Check for updates

© Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY. Published by BMJ.

For numbered affiliations see end of article.

**Correspondence to**
Dr Benjamin Speich;
benjamin.speich@ndorms.ox.ac.uk
assists in the development of reporting guidelines and actively promotes their use, consists of methodologists, epidemiologists, reporting guideline developers, statisticians, clinicians and journal editors.

The CONSOLIDated Standards of Reporting Trials (CONSORT) is perhaps the most prominent reporting guideline, designed to help improve the transparency and quality of reporting of RCTs.<sup>11-13</sup> It guides authors, peer reviewers and journal editors on the minimum information to be included in published reports of RCTs to facilitate critical judgement and interpretation of results and consists of 25 items and a flow diagram. The last update of the CONSORT Statement was published simultaneously in 10 leading medical journals in 2010<sup>13</sup> and currently CONSORT is endorsed by over 600 journals worldwide.<sup>14</sup>

Despite some improvement in reporting following the endorsement of the CONSORT statement, there remain major reporting deficiencies in published RCTs.<sup>3 15-21</sup> For example, a study of 1122 RCTs indexed in PubMed in December 2012 found that many did not define the primary outcome (31%), state the sample size calculation (45%) or explain the method of allocation concealment (50%).<sup>22</sup> This lack of transparency is a major limiting factor for readers who assess an article in order to find the answer to a specific question; it is also a major problem for scientists who perform systematic reviews and meta-analyses.

## Evidence to date

Journals can play a vital role in improving the reporting of published RCTs. For example, a survey of journals' 'Instructions to Authors' in 2014 found that 63% (106 of 168) of biomedical journals mentioned CONSORT<sup>23</sup>; however of those journals only 38 (36%) required a completed CONSORT checklist on submission. Such implementation indicates some improvement over time compared with an assessment in 2007 when only 17 of 62 (27%) journals requested the CONSORT checklist on

submission.<sup>24</sup> A study using interrupted time series analysis and assessing if the CONSORT checklist for reporting abstracts of RCTs had an effect on reporting quality found that results were better reported in journals which had an active editorial policy to implement the checklist.<sup>25</sup>

A scoping review conducted in 2017 by Blanco and colleagues summarised different interventions aimed at improving adherence to reporting guidelines.<sup>26</sup> They identified a number of different interventions, some of which had been evaluated at journals. However, all the interventions, except requesting submission of checklists from authors, required additional resources from the journal (eg, internal peer review by editorial assistants or an additional peer-reviewer round conducted by a senior statistician using appropriate reporting guidelines.<sup>27-29</sup> Therefore, it is unlikely that these interventions will be implemented in the majority of journals, especially smaller journals with limited resources. Another study found that providing authors with a web-based CONSORT tool, which combined different CONSORT extensions and provided authors with a customised checklist, did not improve reporting when used at the manuscript revision stage.<sup>30</sup> However, a study examining 'the nature and extent of changes made to manuscripts after peer review, in relation to the reporting of methodological aspects of RCTs' and 'the type of changes requested by peer reviewers' found that peer review did lead to some improvement in reporting.<sup>27</sup>

The role of peer reviewers and expectations of them is varied.<sup>31</sup> While CONSORT checklists are sometimes available for peer reviewers to check, they are not typically instructed to assess this information as part of their review and there have been no studies evaluating the effect of asking them to do this. We plan to evaluate the impact of giving peer reviewers a short version of the CONSORT checklist together with a brief explanation of the items and asking them to check if they are adequately reported.

## METHODS AND ANALYSIS

### Objective

The objective of this study is to evaluate the impact of giving peer reviewers, during the standard peer review process, a short version of the CONSORT checklist (C-short) together with a brief explanation of the items and asking them to check if they are adequately reported in the manuscript.

### Study design

This study is a multicentre superiority RCT with submitted manuscripts as the unit of randomisation (figure 1; allocation ratio 1:1). This study protocol was written in adherence to the SPIRIT guidelines (online supplementary file).<sup>32</sup>

### Study setting and eligibility criteria

The population will be defined on two levels: included journals and included manuscripts.

<figure><img src="image_3.png" /><figcaption>**Figure 1** Study flow chart. CONSORT, CONSOLIDated Standards of Reporting Trials; SPIRIT: Standard Protocol Items: Recommendations for Interventional Trials.</figcaption></figure>
## Inclusion criteria for journals

Included journals must: (i) endorse the CONSORT statement by mentioning it in the journals' Instruction to Authors; (ii) have published primary results of at least five RCTs in 2017 (identified using a PubMed search). To be efficient, we plan to contact (via email) the editors of eligible journals from specific publishers (eg, BMJ Publishing Group; Public Library of Science (PLOS)) instead of separate journals. A description of the requirements for participation and a short summary information sheet will be included as part of the email invitation sent to journal editors. If a journal is eligible, and the editor agrees to take part, the editor will need to provide access to their editorial system (eg, ScholarOne, Editorial Manager) to enable the external researcher (BS) to screen and randomise eligible manuscripts. In cases where this is not possible, we will explore with individual journals if it would be possible to grant limited access (eg, only rights to screen studies) or to handle the different steps without access to the editorial system (eg, screening through automated reports; intervention provided by a journal staff member) and that the emails for the intervention would be sent by a member of the editorial team.

## Inclusion criteria for manuscripts

* All new manuscript submissions reporting the primary results of RCTs, which the journal editor has decided to send out for external peer review. Since the 10 chosen CONSORT checklist items (C-short) are applicable to different study designs, we will include all manuscripts reporting the primary results of RCTs regardless of study design (eg, parallel group trial, cluster trial, superiority trial, non-inferiority/equivalence trials).

## Exclusion criteria for manuscripts

* Manuscripts clearly presenting secondary trial results, additional time points, economic analyses or any other analyses.

* Manuscripts which are clearly labelled as a pilot or feasibility study or animal studies.

* Manuscripts not sent for peer review.

Details of journal manuscript submission and peer review processes, including consent and potential confidentiality issues, will be discussed in detail with each journal by teleconference and/or face to face prior to the journal agreeing to take part to ensure that randomisation of manuscripts is feasible.

In participating journals, the external researcher (BS) will check at least twice a week (by screening automated submission lists) all research manuscripts that are sent out for external peer review. As soon as the first invited peer reviewer accepts the invitation to review, the manuscript will be randomised to the intervention or control arm (see 'Randomisation' for more details). It is possible that this process might be slightly different among various included journals (eg, that team members of a journal

might be involved in the screening if limited or no access to the journal's editorial system is granted).

## Interventions

### Control group: usual practice

After accepting to review a manuscript, peer reviewers will receive the automated, journal specific standard email with general information as per each journal's usual practice (eg, where to access the manuscript, date the peer review report is due).

### Intervention group: C-short plus usual practice

After accepting to review a manuscript, peer reviewers will receive the automated, journal specific standard email with general information (identical to control group). In addition, peer reviewers will receive an additional email from the editorial office that includes a short version of the CONSORT checklist (C-short) together with a brief explanation of the items either as a table within the email or as an attachment based on the preferences and possibilities of the journal (table 1, online supplementary appendix 1). Peer reviewers will be asked to check whether the items in the C-short checklist are addressed in the manuscript and to request authors to include these items if they are not adequately reported. This second email (see online supplementary appendix 1), containing the C-short checklist together with a brief explanation, is not generated automatically within the existing journal editorial systems (eg, ScholarOne or Editorial Manager); it will be sent manually by a researcher (BS) from the journal's editorial system or by a member of the journal's staff. In both cases the email will appear to have come from the editorial office (not the researcher).

## Development of the C-short checklist and explanation of items

For the development of C-short we chose the 10 most important and poorly reported CONSORT items as identified by a group of CONSORT experts in a previous study conducted by Hopewell and colleagues.³⁰ The selection of the items was based on expert opinion and empirical evidence whenever available.³⁰ In addition, to enable peer reviewers to better understand the items, we added a short explanation for each of the 10 items. These short explanations were extracted and amended from the CONSORT explanation and elaboration paper¹¹ and from COBWEB which is an online writing aid tool.³³ The short explanation was discussed and adapted by the scientific committee.

## Outcomes

### Primary outcome

The primary outcome of this study will be the difference in the mean proportion of adequately reported C-short items in published articles between the two groups.

### Secondary outcomes

Secondary outcomes will include the following:

* Mean proportion of adequately reported C-short items in published articles considering each item separately.
<table><caption>**Table 1** The 10 most important and poorly reported consort items as defined by a group of experts on the CONSORT statement.³⁰ for better understanding key features were summarised within a short explanation (extracted from the CONSORT explanation and elaboration paper¹¹ as well as from the COBWEB tool)³³</caption><thead><tr><td>Item</td><td>Section</td><td>CONSORT item</td><td>Short explanation</td></tr></thead><tbody><tr><td>1</td><td>Outcomes (6a)</td><td>Completely defined pre-specified primary outcome measure, including how and when it was assessed</td><td>Is it clear (i) what the primary outcome is (usually the one used in the sample size calculation), (ii) how it was measured (if relevant; eg, which score used), (iii) at what time point and (iv) what the analysis metric was (eg, change from baseline, final value)?</td></tr><tr><td>2</td><td>Sample size (7a)</td><td>How sample size was determined</td><td>Is there a clear description of how the sample size was determined, including (i) the estimated outcomes in each group, (ii) the α (type I) error level, (iii) the statistical power (or the β (type II) error level) and (iv) for continuous outcomes, the SD of the measurements?</td></tr><tr><td>3</td><td>Sequence generation (8a)</td><td>Method used to generate random allocation sequence</td><td>Does the description make it clear if the 'assigned intervention is determined by a chance process and cannot be predicted'?</td></tr><tr><td>4</td><td>Allocation concealment (9)</td><td>Mechanism used to implement random allocation sequence (such as sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned</td><td>Is it clear how the care provider enrolling participants was made ignorant of the next assignment in the sequence (different from blinding)? Possible methods can rely on centralised or 'third-party' assignment (ie, use of a central telephone randomisation system, automated assignment system, sealed containers).</td></tr><tr><td>5</td><td>Blinding (11a)</td><td>If done, who was blinded after assignment to interventions (eg, participants, care providers, those assessing outcomes)</td><td>Is it clear if (i) healthcare providers, (ii) patients and (iii) outcome assessors are blinded to the intervention? General terms such as 'double-blind' without further specifications should be avoided.</td></tr><tr><td>6</td><td>Outcomes and estimation (17a/b)</td><td>For the primary outcome, results for each group, and the estimated effect size and its precision (such as 95% CI)</td><td>Is the estimated effect size and its precision (such as SD or 95% CI) for each treatment arm reported? When the primary outcome is binary, both the relative effect (risk ratio, relative risk or OR) and the absolute effect (risk difference) should be reported with CI.</td></tr><tr><td>7</td><td>Harms (19)</td><td>All-important harms or unintended effects in each group</td><td>Is the number of affected persons in each group, the severity grade (if relevant) and the absolute risk (eg, frequency of incidence) reported? Are the number of serious, life threatening events and deaths reported? If no adverse event occurred this should be clearly stated.</td></tr><tr><td>8</td><td>Registration (23)</td><td>Registration number and name of trial registry</td><td>Is the registry and the registration number reported? If the trial was not registered, it should be explained why.</td></tr><tr><td>9</td><td>Protocol (24)</td><td>Where trial protocol can be accessed</td><td>Is it stated where the trial protocol can be assessed (eg, published, supplementary file, repository, directly from author, confidential and therefore not available)?</td></tr><tr><td>10</td><td>Funding (25)</td><td>Sources of funding and other support (such as supply of drugs) and role of funders</td><td>Are (i) the funding sources, and (ii) the role of the funder(s) described?</td></tr></tbody></table>

COBWEB, Consort-based WEB tool; CONSORT, CONSOLIDATED STANDARDS OF REPORTING TRIALS.

► Difference in mean proportion of adequately reported C-short items in published articles considering each sub-item (see 'Assessment of outcomes') as a separate item.

► Time from assigning an editor to the first decision (as communicated to the author after the first round of peer review).
* Proportion of manuscripts rejected after the first round of peer review.

* Proportion of manuscripts that will be published in the journal under study.

## Additional outcomes

* Exploratory analysis of available peer reviewer comments (ie, any references to CONSORT).

For journals where peer reviewers' comments are subsequently published alongside the published article, we will examine the peer reviewers' comments for any reference to CONSORT and trial reporting. We will contact those journals which do not make peer reviewers' comments publicly available, to see if reviews could be provided for such analyses under the condition that only anonymised data will be published.

## Assessment of outcomes

The outcomes will be assessed independently by two (blinded or at least partially blinded; see 'blinding') outcome assessors with expertise in the design and reporting of clinical trials. Any disagreement will be resolved by consensus or if necessary by consulting a third assessor. To ensure consistency between reviewers, we will first pilot the data extraction form; any disparities in the interpretation will be discussed and the data extraction form will be modified accordingly.

Adequate reporting of items will be assessed in duplicate from published full-text publications following the same instructions as provided by the CONSORT C-short checklist.¹¹ The following checklist items have, due to their complexity, sub-items which will be extracted separately. The sub-items are highlighted in the short explanation of the intervention (see table 1 and online supplementary appendix 1):

* Outcomes (item 6a): (i) define primary outcome, (ii) how it was measured, (iii) at what time point and (iv) the analysis metric (eg, change from baseline, final value).

* Sample size (item 7a): (i) the estimated outcomes in each group, (ii) the α (type I) error level, (iii) the statistical power (or the β (type II) error level), (iv) for continuous outcomes, the SD of the measurements.

* Blinding (item 11a): Is the blinding status clear for the following persons: (i) healthcare provider, (ii) patients and (iii) outcome assessors.

* Funding (item 25): (i) the funding source, and (ii) the role of funder in the design, conduct, analysis and reporting.

All items will be judged as either 'yes' meaning adequately reported, 'no' meaning not adequately reported or not reported at all, or 'NA' meaning that this sub-item is not applicable for this RCT. Items with different sub-items will only be judged as adequately reported if all relevant sub-items were adequately reported.

The outcomes 'time from assigning an editor to the first decision', 'proportion of manuscripts rejected after the first round of peer-review' and 'proportion of

manuscripts that will be published in the journal under study' will be extracted directly from the journal's editorial system or provided by the journal.

## Participant timeline

The overview of the study schedule, including enrolment, intervention and assessments, is presented in table 2.

## Sample size

For the sample size calculation, we hypothesised in a first scenario (table 3) that the intervention C-short will result in a 25% relative increase in adequate reporting compared with the control (meaning that 70% of items will be adequately reported in the intervention group and 56% in the control group). This is based on a proportion of adequate reporting of 0.56 for the 10 most important and poorly reported items found in the control group of a previous study (meaning that a mean of 56% of the 10 most important and poorly reported items were reported).³⁰ The SD in the same study was 0.23. However, we calculated our sample size to account for a slightly larger variability in our data (SD=0.25). To demonstrate a significant difference with a power of 90% and a type 1 error at 5%, a total of 136 published articles will be required in this scenario (68 per treatment arm; based on a two sided t-test).

Two authors of this protocol, working for PLOS ONE (IP and AC), one of the participating journals, pointed out that 3 out of the 10 assessed items (ie, item 'Registration', 'Protocol' and 'Funding') should always be implemented in submissions to their journal given their policy requirements for clinical trials. Assuming that this journal will recruit a high proportion of manuscripts, and that also other journals might update their templates, we increased the sample size in a second scenario, in which all these three items would have an overall adherence of 90% in the control arm (table 3). This would entail an overall baseline adherence with the 10 C-short items of 71%. Based on a two sided t-test, a sample size of 166 (83 per treatment arm) will have a power of 80% to find a 15% relative increase (71% adherence in control group; 82% adherence in intervention group; SD=0.25; a type 1 error at 5%).

Since the final sample size will be based on the number of articles published, rather than on the number of manuscripts randomised, eligible manuscripts will be randomised until 83 articles are published in each arm (resulting in no less than 166 articles), to avoid loss of power due to potential imbalance between arms. Recruitment will be stopped as soon as both arms reach the sample size of 83. After recruitment has stopped we will wait 3 months so that manuscripts, which are still in production, can be published. Manuscripts which are published after the 3-month period will be excluded.

## Randomisation and blinding

Manuscripts meeting the eligibility criteria and sent out for external peer review by the journals will be randomised
<figure><img src="image_4.png" /><figcaption>**Table 2** Study schedule</figcaption></figure>

<table><thead><tr><th></th><th>Enrolment</th><th>Allocation and intervention</th><th>Intervention</th><th>Post-intervention</th><th></th></tr></thead><tbody><tr><td>Time point</td><td>Studies which are sent out for peer review</td><td>After first peer reviewer accepts invitation</td><td>Whenever an additional peer reviewer accepts invitation</td><td>First decision by journal</td><td>Published manuscripts</td></tr><tr><td>Eligibility screen</td><td>X</td><td></td><td></td><td></td><td></td></tr><tr><td>Allocation</td><td></td><td>X</td><td></td><td></td><td></td></tr><tr><td>Intervention:</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>C-short + usual care</td><td></td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>Usual care</td><td></td><td>X</td><td>X</td><td></td><td></td></tr><tr><td>Assessment of trial characteristics:</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Funding source</td><td></td><td></td><td></td><td></td><td>X</td></tr><tr><td>Study centres (single centre or multicentre)</td><td></td><td></td><td></td><td></td><td>X</td></tr><tr><td>Sample size</td><td></td><td></td><td></td><td></td><td>X</td></tr><tr><td>Study design (eg, parallel arm, crossover)</td><td></td><td></td><td></td><td></td><td>X</td></tr><tr><td>Hypothesis (eg, superiority, non-inferiority)</td><td></td><td></td><td></td><td></td><td>X</td></tr><tr><td>Medical field</td><td></td><td></td><td></td><td></td><td>X</td></tr><tr><td>Intervention tested</td><td></td><td></td><td></td><td></td><td>X</td></tr><tr><td>Number of trial arms</td><td></td><td></td><td></td><td></td><td>X</td></tr><tr><td>Number of peer reviewers</td><td></td><td></td><td></td><td></td><td>X</td></tr><tr><td>Journal which published the manuscript</td><td></td><td></td><td></td><td></td><td>X</td></tr><tr><td>Number of journals requesting CONSORT adherence (submission of checklist mandatory)</td><td></td><td></td><td></td><td></td><td>X</td></tr><tr><td>Assessment of outcomes:</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Time from assigning an academic editor until the first decision</td><td></td><td></td><td></td><td>X</td><td></td></tr><tr><td>Proportion of manuscripts directly rejected after the first round of peer review</td><td></td><td></td><td>X</td><td></td><td></td></tr><tr><td>Proportion of manuscripts that will be published in the journal under study</td><td></td><td></td><td></td><td>X</td><td></td></tr><tr><td>Adherence to CONSORT items and sub-items</td><td></td><td></td><td></td><td></td><td>X</td></tr></tbody></table>

CONSORT, CONSOLIDATED STANDARDS OF REPORTING TRIALS.

into one of the two groups (allocation 1:1). The randomisation list will be created by the Study-Randomizer system³⁴ using random block sizes between 2 and 8 and stratified by journal. As soon as the first peer reviewer

accepts the invitation, the manuscript will be included and randomised to one of the two study arms. One of the investigators (BS) will log onto the Study-Randomizer system³⁴ and enter the study identification number (ID;
<table><caption>**Table 3** Assumptions for sample size calculations in two different scenarios</caption><thead><tr><th>Item</th><th>CONSORT item</th><th>Scenario 1.<br>adequate reporting<br>as published in<br>WebCONSORT<sup>30</sup></th><th>Scenario 2.<br>adapted from<br>scenario 1</th></tr></thead><tbody><tr><td>1</td><td>Outcomes<br>(6a)</td><td>77% (79 of 103)</td><td>77% (79 of 103)</td></tr><tr><td>2</td><td>Sample size<br>(7a)</td><td>83% (85 of 103)</td><td>83% (85 of 103)</td></tr><tr><td>3</td><td>Sequence<br>generation<br>(8a)</td><td>76% (78 of 103)</td><td>76% (78 of 103)</td></tr><tr><td>4</td><td>Allocation<br>concealment<br>(9)</td><td>55% (57 of 103)</td><td>55% (57 of 103)</td></tr><tr><td>5</td><td>Blinding (11a)</td><td>35% (36 of 103)</td><td>35% (36 of 103)</td></tr><tr><td>6</td><td>Outcomes<br>and<br>estimation<br>(17a)</td><td>44% (45 of 103)</td><td>44% (45 of 103)</td></tr><tr><td>7</td><td>Harms (19)</td><td>71% (73 of 103)</td><td>71% (73 of 103)</td></tr><tr><td>8</td><td>Registration<br>(23)</td><td>69% (71 of 103)</td><td>90%</td></tr><tr><td>9</td><td>Protocol (24)</td><td>19% (20 of 103)</td><td>90%</td></tr><tr><td>10</td><td>Funding (25)</td><td>34% (35 of 103)</td><td>90%</td></tr><tr><td colspan="2"><strong>Overall</strong></td><td><strong>56%</strong></td><td><strong>71%</strong></td></tr></tbody></table>

CONSORT, CONsolidated Standards for Reporting Trials

provided by the journal), the study title and the journal the study was submitted to. Subsequently, all additional peer reviewers accepting the invitation to review the same manuscript will receive the same group assignment as the first peer reviewer.

Authors will be blinded to the intervention. Editors will not be actively informed about the randomisation (possible exception listed under 'Interventions'). To avoid potential bias, peer reviewers and manuscript authors will not be informed of the study hypothesis, design and intervention.

Outcomes will be assessed in duplicate (see 'Assessment of outcomes'). At least one outcome assessor will be blinded. Due to restricted resources the investigator conducting the randomisation (BS) might be involved in the data extraction from published manuscripts.

## Data analysis

All quantitative variables will be described using means and SD, or medians and interquartile ranges in case severe departures from a normal distribution are identified. Data distributions will be inspected visually (ie, by histograms) instead of performing formal statistical tests for normality. Categorical variables will be described using frequencies and percentages. For the primary and secondary outcomes, we will estimate the mean difference

between the two groups and report them with respective 95% CI. No interim analysis will be conducted.

## Populations of analysis

The main population for analysis will be all manuscripts randomised and accepted for publication in the participating journals. In contrast to RCTs conducted with patients, where losses to follow-up need to be carefully considered (eg, multiple imputation of missing data), we are only interested in the reporting adherence of RCTs that are published. As such, we will exclude randomised manuscripts that were not published from the main analysis. All outcomes will be calculated based on the main population. The secondary outcome 'Time to the first decision' will additionally be calculated considering all randomised manuscripts (including the ones which were not published). For all analyses a p-value of 0.05 (5% significance level) will be used to indicate statistical significance. Exact p-values will be presented up to three decimal places. We anticipate there will be no missing data in this study, neither at the individual C-short items, nor at the manuscript level. This is due to the study design, which will include only the randomised manuscripts that are accepted for publication. We will analyse if the rate of manuscripts rejected after the first round of peer review and if the proportion of manuscripts that will be published differentiate among the two study arms (both secondary results).

## Analysis of primary endpoint

The effect of the intervention will be estimated as the mean difference in the proportion of C-short items adequately reported between the study arms. If the data on the primary outcome are normally distributed, groups will be compared using an unpaired Student's t-test. If the data are not normally distributed, comparisons will be performed using a non-parametric equivalent test (ie, Wilcoxon-Mann-Whitney test).

## Analysis of secondary endpoints

To investigate the effect of the intervention on the secondary outcomes, mean differences with respective 95% CI will be reported. If normality is not observed for any of the continuous secondary outcomes, the same strategy adopted for the primary outcome (use of a non-parametric equivalent to the Student's t-test) will be used.

## Pre-specified subgroup analysis

No formal subgroup comparative analysis is planned for the primary or secondary outcomes. However, the effect of the intervention on the primary outcome within subgroups will be presented using forest plots to visually examine whether it may differ according to some variables such as: (i) journals that actively implement the CONSORT Statement (defined as requiring authors to submit a completed CONSORT checklist alongside their manuscript) versus journals that are not actively implementing the CONSORT statement, (ii) sample size of included RCTs (n<100 vs n≥100) and (iii) impact factor
(<5, 5.1–10;>10) as there is evidence that higher impact factor and higher sample size are associated with higher adherence to reporting guidelines.35 Subgroup analysis at the journal level will only be conducted when sufficient journals are in each group so that no results of individual journals are revealed. All analyses will be exploratory, with the aim of supporting new hypothesis generation, rather than being conclusive.

## Data management and confidentiality

Outcomes from publications will be assessed and extracted in duplicate. Since this information is not confidential, we will use freely available online forms (eg, Google forms) for data extraction from published RCTs. Data entered will be validated for completeness.

Data from the journal's editorial system (eg, title of manuscript, first author, randomisation ID, journal, date when manuscript was assigned to an editor, date when the final decision was made, final editorial decision, number of peer reviewers who reviewed the manuscript, the peer review reports (if available)) will be extracted (by BS or a member of the journal's staff), anonymised and entered in password protected files which are saved on a server from the University of Oxford. Data will be managed and curated according to University of Oxford regulations, which includes regular back-up (on a daily basis) of the virtual drives where the data are stored. No auditing or data monitoring is planned (as outcomes are directly extracted from journal's editorial system or in duplicate from published RCTs).

The raw data extracted from the included published manuscripts can be made openly accessible in an anonymised way (ie, giving the included RCT a number instead of identifying them). Derived/aggregated data, including anonymised information generated from the journal's editorial system, will be stored and made available to the research community when the project ends (see also 'Publication policy and access to data'). Where appropriate, the researcher who has access to the journal's editorial system (BS) and anyone else who will see the identifiable data will sign a confidentially agreement with the participating journals, confirming that they will not share identifiable data with any other party. Publishers such as the British Medical Journal (BMJ) state in their Company Privacy Statement that reviews and manuscripts may be used for quality improvement purposes and that is the nature of this research. Furthermore, peer reviewers for all BMJ journals receive the following statement in their invitation letter 'We are constantly trying to find ways of improving the peer review system and have an ongoing programme of research. If you do not wish your review entered into a study please let us know by emailing [...] as soon as possible.'

## Trial registration

This trial was denied registration on ClinicalTrials.gov as the study is not a clinical study that assesses a health outcome in human subjects. Instead we registered the trial

on the Open Science Framework (https://osf.io/c4hn8). The first manuscript was randomised in July 2019. We expect that recruitment will be finished in summer 2021.

## Patient and public involvement

Given the specific study topic, the steering committee agreed that patient or public involvement is not needed for this study.

## DISCUSSION

RCTs are the current gold standard for evaluating any new intervention in evidence-based medicine. Unfortunately, not all RCTs are of high quality. In fact, there are several well-known shortcomings with respect to reporting.3 15–20 It is important to note that adhering to the CONSORT Statement does not mean that the study is of high quality. However, reporting all items from the CONSORT checklist will enable readers to adequately judge the quality of RCTs.

In this RCT we will test if a simple intervention in the form of asking peer reviewers to check whether selected CONSORT items are adequately addressed will increase the proportion of reporting completeness in the published RCTs in the participating journals. A multi-centre parallel arm RCT with randomisation at the individual manuscript level was chosen instead of a cluster RCT because the risk of 'contamination' at journal level was judged as low as the intervention will be implemented by an external researcher (ie, BS) or a member of the journal staff (eg, personnel from editorial services). The likelihood of contamination due to peer reviewers being invited to assess several RCTs and therefore becoming exposed to both intervention arms was judged small and therefore we do not plan to adjust for clustering by journal. Originally we planned to implement the intervention within the original instruction to peer reviewer email which is sent out as soon as a peer reviewer accepts the invitation from the journal. However, as these emails are sent automatically by the journal's editorial system we would have needed to modify the software from each journal to make sure that only half of the manuscripts administered the intervention. After our first discussion with journal editors and journal staff, we realised that this approach is not feasible and therefore decided to implement the intervention in the form of a separate email. We intended to conduct this RCT in a pragmatic way so that results 'would also be relevant to [...] people who decide whether to implement the intervention on the basis of its results'.36 Hence we chose to assess outcomes from published articles and not from manuscripts after the first round of revisions. Ideally, the full impact of the intervention would also be measured including all versions of randomised manuscripts in the final statistical analysis. However, due to confidentiality issues and limited resources we will not be able to evaluate manuscript versions prior to publication.
A selection of CONSORT items was chosen instead of the entire CONSORT checklist as we did not want to put too high a burden on peer reviewers, which could increase the risk that peer reviewers ignore our reminder.

Should the proposed intervention be successful in improving the reporting quality of published RCTs, as measured by the adherence to CONSORT, the intervention could be implemented at the journal level without requiring a large amount of additional resources. In addition, very similar interventions for other article types (eg, systematic reviews, trial protocols) and corresponding guidelines (eg, PRISMA [Preferred Reporting Items for Systematic Reviews and Meta-Analysis], SPIRIT) could be easily implemented too.

## Ethics and dissemination

Ethical approval has been obtained from the Medical Sciences Interdivisional Research Ethics Committee of the University of Oxford (R62779/RE001). The original approved study protocol is available in online supplementary appendix 2. The WHO trial registration data set is available in online supplementary appendix 3.

The results from this study will be published in a peer-reviewed journal irrespective of the study results. Authorship of publications will be granted according to the criteria of the International Committee of Medical Journal Editors. We plan to make the anonymised data set, including the data from the published articles, available as a supplementary file of the main publication.

### Author affiliations

**Twitter** Benjamin Speich @benjaminspeich, David Moher @dmoher and Michael Maia Schlüssel @m_schlüssel

**Contributors** SH, BS, IB, MB, DM and PR had the study idea and designed the study. SS, IP and AC provided expertise to ensure implementation at the journal level was possible. MMS was responsible for statistical aspects, including the sample size calculation and the data analysis plan. BS and SH wrote the first draft of the study protocol. All authors critically revised the protocol and approved the final version.

**Funding** BS is supported by an Advanced Postdoc.Mobility grant from the Swiss National Science Foundation (P300PB_177933). DM is supported by a University Research Chair, University of Ottawa. MMS is funded by Cancer Research UK. The funders had no role in designing the study and will also have no role in conducting the study, or analysing and reporting study results.

**Competing interests, and roles of the steering committee** SS is employed by the British Medical Journal (BMJ). IP and AC are employed by the Public Library of

Science. DM, SH and IB are members of the CONSolidated Standards for Reporting Trials (CONSORT) executive and authors of the CONSORT 2010 statement. DM and PR are members of the EQUATOR (Enhancing the Quality and Transparency of Research) network steering group. MMS is a meta-researcher and reporting guideline developer, enthusiast and disseminator; he may therefore overestimate the importance of this project. All authors have declared that no other competing interests exist: the principal investigator (BS) is responsible for the preparation and the revisions of the study protocol, organising meetings of the steering committee, recruiting and randomising eligible manuscripts as well as the publication of study reports. The steering committee (IB, MB, SH, DM, PR, BS, MMS and SS) is responsible for revising the protocol, defining and validating the additional short explanation for each CONSORT item, advising on study implementation and for publishing the results of this study. MMS is responsible for the sample size calculation and the statistical analyses.

**Patient and public involvement** Patients and/or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research.

**Patient consent for publication** Not required.

**Provenance and peer review** Not commissioned; externally peer reviewed.

**Open access** This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.

### ORCID iDs

Benjamin Speich http://orcid.org/0000-0002-3301-8085

Sara Schroter http://orcid.org/0000-0002-8791-8564

David Moher http://orcid.org/0000-0003-2434-4206

Iratxe Puebla http://orcid.org/0000-0003-1258-0746

Michael Maia Schlüssel http://orcid.org/0000-0002-1711-9310

## REFERENCES

1. Duley L, Antman K, Arena J, et al. Specific barriers to the conduct of randomized trials. *Clin Trials* 2008;5:40–8.

2. Collins R, MacMahon S. Reliable assessment of the effects of treatment on mortality and major morbidity, I: clinical trials. *Lancet* 2001;357:373–80.

3. Chalmers I. Unbiased, relevant, and reliable assessments in health care: important progress during the past century, but plenty of scope for doing better. *BMJ* 1998;317:1167–8.

4. Altman DG, Simera I, Hoey J, et al. EQUATOR: reporting guidelines for health research. *Lancet* 2008;371:1149–50.

5. Simera I. EQUATOR network collates resources for good research. *BMJ* 2008;337:a2471.

6. Simera I, Altman DG. Writing a research article that is "fit for purpose": EQUATOR Network and reporting guidelines. *Evid Based Med* 2009;14:132–4.

7. Simera I, Altman DG, Moher D, et al. Guidelines for reporting health research: the EQUATOR network's survey of guideline authors. *PLoS Med* 2008;5:e139.

8. Simera I, Moher D, Hirst A, et al. Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR network. *BMC Med* 2010;8:24.

9. Simera I, Moher D, Hoey J, et al. The EQUATOR network and reporting guidelines: helping to achieve high standards in reporting health research studies. *Maturitas* 2009;63:4–6.

10. Altman DG, Simera I. A history of the evolution of guidelines for reporting medical research: the long road to the EQUATOR network. *J R Soc Med* 2016;109:67–77.

11. Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. *BMJ* 2010;340:c869.

12. Schulz KF, Altman DG, Moher D, et al. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. *BMC Med* 2010;8:18.

13. Schulz KF, Altman DG, Moher D, et al. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. *BMJ* 2010;340:c332.

14. CONSORT transparent reporting of trials. Endorsers. journals and organizations. Available: http://www.consort-statement.org/about-consort/endorsers [Accessed 27 Feb 2019].

<footnote n="1">¹Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK</footnote>

<footnote n="2">²Basel Institute for Clinical Epidemiology and Biostatistics, Department of Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland</footnote>

<footnote n="3">³The BMJ, London, UK</footnote>

<footnote n="4">⁴Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada</footnote>

<footnote n="5">⁵Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada</footnote>

<footnote n="6">⁶School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada</footnote>

<footnote n="7">⁷PLOS ONE, Public Library of Science, Cambridge, UK</footnote>

<footnote n="8">⁸The EQUATOR Network, Oxford, UK</footnote>

<footnote n="9">⁹Université de Paris, CRESS, Inserm, INRA, F75004, Paris, France</footnote>

<footnote n="10">¹⁰Centre d'Épidémiologie clinique, Hôpital Hôtel Dieu, Assistance Publique des Hôpitaux de Paris, Paris, France</footnote>
15 Chan A-W, Hróbjartsson A, Haahr MT, et al. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. *JAMA* 2004;291:2457–65.

16 Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. *Lancet* 2009;374:86–9.

17 Ioannidis JP, Lau J. Completeness of safety reporting in randomized trials: an evaluation of 7 medical areas. *JAMA* 2001;285:437–43.

18 Glasziou P, Meats E, Heneghan C, et al. What is missing from descriptions of treatment in trials and reviews? *BMJ* 2008;336:1472–4.

19 Boutron I, Dutton S, Ravaud P, et al. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. *JAMA* 2010;303:2058–64.

20 Hopewell S, Dutton S, Yu L-M, et al. The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. *BMJ* 2010;340:c723.

21 Turner L, Shamseer L, Altman DG, et al. Does use of the CONSORT statement impact the completeness of reporting of randomised controlled trials published in medical journals? A cochrane review. *Syst Rev* 2012;1:60.

22 Odutayo A, Emdin CA, Hsiao AJ, et al. Association between trial registration and positive study findings: cross sectional study (epidemiological study of randomized Trials-ESORT). *BMJ* 2017;356:j917.

23 Shamseer L, Hopewell S, Altman DG, et al. Update on the endorsement of CONSORT by high impact factor journals: a survey of journal "Instructions to Authors" in 2014. *Trials* 2016;17:301.

24 Hopewell S, Altman DG, Moher D, et al. Endorsement of the CONSORT Statement by high impact factor medical journals: a survey of journal editors and journal 'Instructions to Authors'. *Trials* 2008;9:20.

25 Hopewell S, Ravaud P, Baron G, et al. Effect of editors' implementation of consort guidelines on the reporting of Abstracts in high impact medical journals: interrupted time series analysis. *BMJ* 2012;344:e4178.

26 Blanco D, Altman D, Moher D, et al. Scoping review on interventions to improve adherence to reporting guidelines in health research. *BMJ Open* 2019;9:e026589.

27 Hopewell S, Collins GS, Boutron I, et al. Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study. *BMJ* 2014;349:g4145.

28 Cobo E, Selva-O’Callaghan A, Ribera J-M, et al. Statistical reviewers improve reporting in biomedical articles: a randomized trial. *PLoS One* 2007;2:e332.

29 Cobo E, Cortés J, Ribera JM, et al. Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical Journal: masked randomised trial. *BMJ* 2011;343:d6783.

30 Hopewell S, Boutron I, Altman DG, et al. Impact of a web-based tool (WebCONSORT) to improve the reporting of randomised trials: results of a randomised controlled trial. *BMC Med* 2016;14:199.

31 Glonti K, Cauchi D, Cobo E, et al. A scoping review on the roles and tasks of peer reviewers in the manuscript review process in biomedical journals. *BMC Med* 2019;17:118.

32 Chan A-W, Tetzlaff JM, Gotzsche PC, et al. Spirit 2013 explanation and elaboration: guidance for protocols of clinical trials. *BMJ* 2013;346:e7586.

33 Barnes C, Boutron I, Giraudeau B, et al. Impact of an online writing aid tool for writing a randomized trial report: the COBWEB (CONSORT-based web tool) randomized controlled trial. *BMC Med* 2015;13:221.

34 Study Randomizer. Available: https://studyrandomizer.com/ [Accessed 8 July 2019].

35 Jin Y, Sanger N, Shams I, et al. Does the medical literature remain inadequately described despite having reporting guidelines for 21 years? - A systematic review of reviews: an update. *J Multidiscip Healthc* 2018;11:495–510.

36 Loudon K, Treweek S, Sullivan F, et al. The PRECIS-2 tool: designing trials that are fit for purpose. *BMJ* 2015;350:h2147.