Development of ARCADIA: a tool for assessing the quality of peer-review reports in biomedical research
Authors: David Blanco
Cecilia Superchi
Darko Hren
José Antonio González
Roser Rius
Isabelle Boutron
Alessandro Recchioni
Journal: BMJ Open (2020)
Discipline: CLINICAL MEDICINE
Keywords: epidemiology; protocols & guidelines; public health; statistics & research methods.
Affiliations: Université de Paris, CRESS, INSERM, INRA, Paris, France ., Statistics and Operations Research Department, Barcelona-Tech, UPC, Barcelona, Spain., Department of Psychology, School of Humanities and Social Sciences, University of Split, Split, Croatia., Université de Paris, CRESS, INSERM, INRA, Paris, France., Springer Nature, Berlin, Germany., Centre d'épidémiologie Clinique, Hôpital Hôtel-Dieu, Assistance Publique Hôpitaux de Paris (APHP), Paris, France.

Abstract:  Objective: To develop a tool to assess the quality of peer-review reports in biomedical research. Methods: We conducted an online survey intended for biomedical editors and authors. The survey aimed to (1) determine if participants endorse the proposed definition of peer-review report quality; (2) identify the most important items to include in the final version of the tool and (3) identify any missing items. Participants rated on a 5-point scale whether an item should be included in the tool and they were also invited to comment on the importance and wording of each item. Principal component analysis was performed to examine items redundancy and a general inductive approach was used for qualitative data analysis. Results: A total of 446 biomedical editors and authors participated in the survey. Participants were mainly male (65.9%), middle-aged (mean=50.3, SD=13) and with PhD degrees (56.4%). The majority of participants (84%) agreed on the definition of peer-review report quality we proposed. The 20 initial items included in the survey questionnaire were generally highly rated with a mean score ranging from 3.38 (SD=1.13) to 4.60 (SD=0.69) (scale 1-5). Participants suggested 13 items that were not included in the initial list of items. A steering committee composed of five members with different expertise discussed the selection of items to include in the final version of the tool. The final checklist includes 14 items encompassed in five domains (Importance of the study, Robustness of the study methods, Interpretation and discussion of the study results, Reporting and transparency of the manuscript, Characteristics of peer reviewer's comments). Conclusion: Assessment of Review reports with a Checklist Available to eDItors and Authors tool could be used regularly by editors to evaluate the reviewers' work, and also as an outcome when evaluating interventions to improve the peer-review process. 

Retracted: No

PubPeer comments:
 Dear authors, I was excited to see your publication titled: Development of ARCADIA: a tool for assessing the quality of peer-review reports in biomedical research, but was saddened by the content found within the article. I would like to point out several, in my opinion, grave oversights of your paper. 1) You (based on the fact that majority of survey participants agreed with your initially proposed definition) defined the quality of a peer-review REPORT as u2018the extent to which a peer-review report helps, first, editors make an informed and unbiased decision about the manuscriptsu2019 outcome and, second, authors improve the quality of the submitted manuscriptu2019. Yet your tool to assess the quality of peer review report does neither: A) assess how much did or could a peer review report help an editor make an informed and unbiased decision; nor B) if the report improved or could improve the quality of a submitted manuscript. Therefore, it seems you failed to make a connection between your tool and the definition you proposed. 2) By your definition, you also imply that the reportu2019s quality is only as good as so far it is judged by the editor(s) or the authors of the paper that the review is supposed to evaluate. Therefore, if a good peer review report is written, but disregarded by the authors or the editor(s), from your definition it would appear it is of low or no quality. Should not a third party of independent evaluators or peers be able to evaluate quality of a peer review report? And does such a third party have to base their judgment on the quality of a report on how much it helped the editors or the authors? This is not discussed by you. Nor is the fact that some reports might be judged good by the authors, and bad by the editors. And in those rare cases when reviewers do not ask for changes, but find the paper to be acceptable for publication as it is u2013 would that also mean, based on your definition, that the quality of a report is not optimal as it only helped the editors? 3) You stated that your tool is the u201cthe only tool that clearly defines the quality of peer-review reports, as its definition was based on the perspectives of 446 authors and editorsu201d. In light of my comments above, and known differences in operational and theoretical definitions, is there evidence that your u201csystematicu201d approach to define quality of peer review is superior to other approaches of making definitions, or that it was successful? I find that what you call u201cqualityu201d is better captured by the word u201cusefulnessu201d (to editors and authors) : u201cthe extent to which it helps, editors make an informed and unbiased decision about the manuscriptsu2019 outcome and, second, authors improve the quality of the submitted manuscriptu201d. Usefulness to editors and authors might be a domain of quality, but surely not the only one. 4) You provide no scale of quality (e.g. 10.1001 jama.287.21.2790), nor discuss if your tool can be used in a binary way to say quality check passed or failed. Therefore, it is not clear, what you mean when you say quality assessment, nor do you perceive peer review as quality control or assurance. In my view your tool is a reporting guideline checklist, that could perhaps be best used by reviewers when writing their report. But following a reporting guideline does not imply that the quality of what is written is good. What if a peer review report failed to detect a grave oversight of a paper, or failed to detect a major methodological flaw, and yet mentioned all the items in your checklist? Would that imply that the report was of good or of bad quality? What about non-critical errors? You do not discuss this in your paper. Nor do you provide guidance if all items need to be assessed. Some tools clearly state u2013 u201cif the answer to any of these questions is u201cnou201d, you can save yourself the trouble of reading the rest of it., https: www.cebm.net 2014 06 critical-appraisal u201d. Can the similar be applied to your tool? 5) You also stated that one of the purposes of your tool could be to evaluate the impact of interventions aimed to improve the quality of the peer-review process u2013 and here you seem to equate the whole peer review process to the writing of a peer review report. I would have welcomed a discussion on editorial peer review, differences between reviewer opinions, and fighting the editorial decision? Did you consider if authors could use your checklist to point out why the editoru2019s decision was wrong? Or could editors use your checklist to justify not inviting a certain reviewer to review for them again, or to rank reviewers? Could it be used in training of peer reviewers, or in journal clubs? 6) Your tool name ARCADIA - Assessment of Review reports with a Checklist Available to eDItors and Authors - is a checklist that you refer throughout the paper as a tool. Yet you stated yourself: u201cThe tool is a checklistu201d, so is there a difference between the tool and the checklist. And was the reason for limiting its use to editors and authors based on your definition of quality? But even more worrying, is the fact that the name does not contain the word QUALITY. So were the exclusion of the word quality and the part u201cwith a checklistu201d just a means to make a flashy name, and therefore the title of your paper is just a rouse? 7) If I did not include the last sentence of my previous point, how would one rate my review with your tool? Is there a threshold of how many less polite comments are allowed in a review report for a yes score on constructiveness and politeness? Similarly, is there a threshold for any other items in your tool (e.g. the number of statements a reviewer can make without providing references for them)? 8) Going back to the name and purpose of your tool - in your reply to BMJ reviewers you stated: u201cwe decided to develop a checklist of items that could be used to show the independent influence of each item on the overall peer review report qualityu201d. I do not see that you provided a measurement of the overall report quality. And in your title, you stated ARCADIA is a tool to assess quality or peer review reports, not a checklist of 14 items that could independently influence the quality which you do not assess. 9) One of items in your tool is to state whether reviewers checked if study limitations are acknowledged? What if a paper states a limitation that in a reviewer s opinion completely invalidates the study? How should that be addressed? 10) It is nice that you acknowledged the tool is not validated, but I wonder why you deemed it good scientific practice to publish a non-validated tool. If you wanted to document the process of its creation, that is admirable, but then, in my opinion, you should have at least restrained from having an implication section that states your tool u201ccould be REGULARLY used by editors to evaluate the reviewersu2019 work,u201d or that u201cit can also be used as an outcome when evaluating interventions in order to improve the peer-review process.u201d With no proof of its usefulness, how scientific is it to make such recommendations, even when hiding behind the modality of verbs can and could? A discussion on guidance on the number of evaluators (i.e. should one worry about inter-rater agreement) and different appraisal by editors and authors and other stakeholders (which I already mentioned) wouldu2019ve been welcome. Additionally, you excluded in the implication section the use of your tool by authors, yet they are in your tools name. 11) Finally, I considered using your checklist to enrich my review, but I choose not to do so. I would not want it to be considered endorsement of the tool. And I observed it was also not used by the BMJ reviewers in their reports, nor did you use it when replying to their reviews. Additionally, as I provide here a post-publication peer review report, and those have not been discussed in your paper, it might be improper to use it for unintended purposes (e.g. preprints, postprint or reviews by friends colleagues before submission to a journal or upload to a server). In my opinion, researchers should do more critical thinking, simulations, piloting or testing before making claims as the ones you stated in this paper. Therefore, I would recommend you retract and republish your paper when you address its shortcomings. , 
 We would like to thank the reviewer for her his clear and constructive post publication review. We are very grateful for the valuable comments about our recent publication entitled u201cDevelopment of ARCADIA: a tool for assessing the quality of peer-review reports in biomedical researchu201d. We have provided a point-by-point response to the comments below. **1.tYou (based on the fact that majority of survey participants agreed with your initially proposed definition) defined the quality of a peer-review REPORT as u2018the extent to which a peer-review report helps, first, editors make an informed and unbiased decision about the manuscriptsu2019 outcome and, second, authors improve the quality of the submitted manuscriptu2019. Yet your tool to assess the quality of peer review report does neither: A) assess how much did or could a peer review report help an editor make an informed and unbiased decision; nor B) if the report improved or could improve the quality of a submitted manuscript. Therefore, it seems you failed to make a connection between your tool and the definition you proposed.** We respectfully disagree with the reviewer, since we believe we did not u201cfail to make a connection between the tool and the definition proposedu201d. ARCADIA as a checklist, provides a list of 14 items, which should be commented in a peer review report to u201chelp editors make an informed and unbiased decision about the manuscripts outcome and, authors improve the quality of the submitted manuscriptu201d. Therefore, the tool does not ask the user *to rate* the quality of a peer review report, but *to assess *if the reviewer appropriately comments the key aspects of a manuscript, which should be included in a peer review report, without using a scoring system. Similarly to other tools used for assessing peer review report quality [e.g., the Review Quality Instrument (RQI) developed by BMJ editors (1)], ARCADIA can be used to assess if a peer reviewer comments on key components of a manuscript (identified through the ARCADIA items) and not *how* the peer reviewer comments on these aspects. Therefore, it does not aim to assess the accuracy of the content of a peer review report (i.e., quality of the content), but it allows verification if the peer reviewer addresses the key elements of a manuscript in the peer review report in such a way that helps u201ceditors make an informed and unbiased decision about the manuscripts outcome and, authors improve the quality of the submitted manuscriptu201d (as it is clearly reported by the definition we provided). For all above reasons, we strongly believe that a clear and straightforward connection between ARCADIA and the definition proposed was, in fact, correctly made. **2.tBy your definition, you also imply that the reportu2019s quality is only as good as so far it is judged by the editor(s) or the authors of the paper that the review is supposed to evaluate. Therefore, if a good peer review report is written, but disregarded by the authors or the editor(s), from your definition it would appear it is of low or no quality. Should not a third party of independent evaluators or peers be able to evaluate quality of a peer review report? And does such a third party have to base their judgment on the quality of a report on how much it helped the editors or the authors? This is not discussed by you. Nor is the fact that some reports might be judged good by the authors, and bad by the editors. And in those rare cases when reviewers do not ask for changes, but find the paper to be acceptable for publication as it is u2013 would that also mean, based on your definition, that the quality of a report is not optimal as it only helped the editors?** We would like to give a brief overview about the peer review process in order to contextualize our work and better explain the reasons behind ARCADIA. The peer review process has three central actors: 1) journal editors, 2) authors and 3) peer reviewers. In a basic and traditional peer review process workflow, peer-reviewers are contacted by journal editors to write a peer review report to evaluate a submitted manuscript. The peer review report is hence used by authors to improve the quality of their manuscripts, and by editors to make an informed decision about the outcome of the manuscript. Therefore, we believe that the quality of a peer review report should be evaluated from an editor and authoru2019s perspective. However, we do not exclude that the perspectives of authors and editors on the same peer review report could be different. As we reported in our paper (2), a third party (e.g., researchers assessing an intervention aimed at improving the peer review process) can also use the tool taking into account both editor and authoru2019s perspective. With the word u2018authorsu2019 (also included in the name of the tool), we refer to all people conducting research and therefore publishing and or reviewing biomedical articles. **3.tYou stated that your tool is the u201cthe only tool that clearly defines the quality of peer-review reports, as its definition was based on the perspectives of 446 authors and editorsu201d. In light of my comments above, and known differences in operational and theoretical definitions, is there evidence that your u201csystematicu201d approach to define quality of peer review is superior to other approaches of making definitions, or that it was successful? I find that what you call u201cqualityu201d is better captured by the word u201cusefulnessu201d (to editors and authors): u201cthe extent to which it helps, editors make an informed and unbiased decision about the manuscriptsu2019 outcome and, second, authors improve the quality of the submitted manuscriptu201d. Usefulness to editors and authors might be a domain of quality, but surely not the only one.** We respectively disagree with the reviewer. We used a systematic approach to develop the tool based on the methods suggested by Streiner and Norman (3), consisting of three main steps: 1) definition of the aim of the tool; 2) generating items and 3) selecting items. We do not claim that the method used to define peer review report quality in our study is superior to the other approaches. Contrary to the common approach used to define the toolu2019s objective (consisting of a few people who agree on the definition), we preferred to directly involve authors and editors (i.e., central actors of this process) to ask them if they agreed on our definition and they were also invited to comment on it. Based on their perspectives and feedback, we then modified and revised the definition provided. As we showed in our methodological review (2019) (4), u2018qualityu2019 is a multidimensional and amorphous concept (5,6), which can be related to different aspects of a peer review report. As we reported in our methodological review (4), *Usefulness* was one of the many identified quality components of a peer review report. However, the steering committee of the present study (composed by five members with extensive expertise in clinical epidemiology, biostatistics, social science and editorial peer review) agreed not to include *Usefulness* of a peer review report as a component of quality because it represented the final objective of a review. A peer review report can be *useful* if the peer reviewer comments on the key elements of a manuscript such as the contribution of the study to scientific knowledge (ARCADIA item 1a), on whether the relevant literature was accurately reviewed (ARCADIA item 1b), etc. This is the reason why we excluded *Usefulness for decision-making* and *Usefulness for manuscript improvement* in the list of items to include in ARCADIA (as it is clearly reported in Figure 1 in the published paper) and we considered these items as part of the definition of peer review report quality. **4.tYou provide no scale of quality (e.g. 10.1001 jama.287.21.2790), nor discuss if your tool can be used in a binary way to say quality check passed or failed. Therefore, it is not clear, what you mean when you say quality assessment, nor do you perceive peer review as quality control or assurance. In my view your tool is a reporting guideline checklist, that could perhaps be best used by reviewers when writing their report. But following a reporting guideline does not imply that the quality of what is written is good. What if a peer review report failed to detect a grave oversight of a paper, or failed to detect a major methodological flaw, and yet mentioned all the items in your checklist? Would that imply that the report was of good or of bad quality? What about non-critical errors? You do not discuss this in your paper. Nor do you provide guidance if all items need to be assessed. Some tools clearly state u2013 u201cif the answer to any of these questions is u201cnou201d, you can save yourself the trouble of reading the rest of it., https: www.cebm.net 2014 06 critical-appraisal u201d. Can the similar be applied to your tool?** As we have formerly stated, ARCADIA is a checklist of 14 items, which should be commented in a peer review report to u201chelp editors make an informed and unbiased decision about the manuscripts outcome and, authors improve the quality of the submitted manuscriptu201d. ARCADIA does not aim to assess the content quality of a peer review report, but it allows verifying if the peer reviewer addresses the key aspects of a manuscript in the peer review report in such a way that helps u201ceditors make an informed and unbiased decision about the manuscripts outcome and, authors improve the quality of the submitted manuscriptu201d. Our aim was not to provide a scale of quality. We opted for developing a checklist instead of a scale for the reasons we reported in a previous study (4). It has been shown that scales are controversial tools in assessing quality, primarily because using a score u201cin summarization weightsu201d would cause a biased estimation of the measured object (7). Therefore, it is the ARCADIAu2019s user which considers if the peer revieweru2019s comment related to a specific item helps her him to u201cmake an informed and unbiased decision about the manuscripts outcomeu201d or u201cto improve the quality of the submitted manuscriptu201d by ticking u2018Yesu2019 or u2018Nou2019 in the checklist. A quality assessment can also be done without expressively saying u2018Failedu2019 or u2018Passedu2019. A quality assessment can simply be made by verifying through a checklist if the key aspects of a manuscript have been assessed (or not) by the peer reviewer as ARCADIA does. Contrary to the tool reported by the reviewer , ARCADIA does not state u201cif the answer to any of these questions is u201cnou201d, you can save yourself the trouble of reading the rest of itu201d. All the items included in ARCADIA (except for the items on the characteristics of the peer revieweru2019s comments encompassed in domain 5) should be commented by a peer reviewer. However, it is not always possible to assess all items. This depends on whether the item is covered in the study, and or the peer reviewer is qualified to comment on that specific item, as we reported in our paper (2). This is reason why we also provide the option u2018Not Applicable (NA)u2019 if an item cannot be assessed. This is the case, for instance, of theoretical essays. Theoretical essays do not have any methods and statistics, so the items corresponding to u2018study methodsu2019 (ARCADIA item 2a) and u2018statistical methodsu2019 (ARCADIA item 2b) should be ticked as u2018NAu2019. **5.tYou also stated that one of the purposes of your tool could be to evaluate the impact of interventions aimed to improve the quality of the peer-review process u2013 and here you seem to equate the whole peer review process to the writing of a peer review report. I would have welcomed a discussion on editorial peer review, differences between reviewer opinions, and fighting the editorial decision? Did you consider if authors could use your checklist to point out why the editoru2019s decision was wrong? Or could editors use your checklist to justify not inviting a certain reviewer to review for them again, or to rank reviewers? Could it be used in training of peer reviewers, or in journal clubs?** This is a valid point. However, we consider out of the paper scope to discuss about u201cthe differences between reviewer opinions, and fighting the editorial decisionu201d as the reviewer suggested (especially in case of words limit requested by the journal). We believe that ARCADIA could improve the transparency of the peer review process. This means that the tool could be used to verify if the editor based her his decision on an exhaustive peer review report. In addition, authors, commonly acting also as researchers and or peer reviewers, could use it to improve their peer review reports (i.e., in training programme as the reviewer suggested) and also as outcome to evaluate interventions to improve the peer review process (i.e., in RCTs). In journal clubs, researchers are used to discuss papers, and not peer review reports. We therefore suggest using other instruments, such as the reporting guidelines, to critically appraise manuscripts. **6. Your tool name ARCADIA - Assessment of Review reports with a Checklist Available to eDItors and Authors - is a checklist that you refer throughout the paper as a tool. Yet you stated yourself: u201cThe tool is a checklistu201d, so is there a difference between the tool and the checklist. And was the reason for limiting its use to editors and authors based on your definition of quality? But even more worrying, is the fact that the name does not contain the word QUALITY. So were the exclusion of the word quality and the part u201cwith a checklistu201d just a means to make a flashy name, and therefore the title of your paper is just a rouse?** ARCADIA is a new tool for assessing the quality of peer review reports, and its format is a checklist. We do not make a distinction between tool and checklist, they are one and the same. We stated this in the text (Results section) u201cThe tool is a checklistu201d and this is the reason why we included checklist in its name. Regarding the name of the tool, the word u2018qualityu2019 was intentionally not included because we did not want to emphasize this tricky and complex concept. However, we believe it is clearly implicit in the name. With u201cassessment of review reportsu201d in the toolu2019s name, we implicitly meant u201cassessment of review reports qualityu201d. u2018Qualityu2019 is a philosophical concept. It is a multidimensional and amorphous notion, which is extremely difficult to define especially from a scientific point of view. Robert M. Pirsig wrote in his book u2018Zen and the Art of Motorcycle Maintenanceu2019 (a great book tackling also the concept of u2018qualityu2019): u201cBut even though Quality cannot be defined, you know what Quality is!u201d We believe that journal editors and authors could generally recognize a high quality peer view report, but then they struggle to define its u2018qualityu2019. As we found in our methodological systematic review (4), the concepts evaluated by the identified tools vary widely. Despite the difficulties to define this amorphous concept, our research contributes to define it for a first time. The definition was based on an exhaustive systematic review of all tools used in biomedical research and also the perspectives of a large and heterogeneous sample of authors and editors. Finally, as we have explained in an earlier answer, ARCADIA is not limited only to authors and editors. **6.tIf I did not include the last sentence of my previous point, how would one rate my review with your tool? Is there a threshold of how many less polite comments are allowed in a review report for a yes score on constructiveness and politeness? Similarly, is there a threshold for any other items in your tool (e.g. the number of statements a reviewer can make without providing references for them)?** As we have previously stated, ARCADIA does not ask the user to rate the quality of a peer review report, but to assess if the reviewer appropriately comments the item included in the tool, which should be commented in a peer review report. As a checklist, it does not include a quality overall score or even a threshold. However, for the purpose of the validation study that we have already conducted (u2018working paperu2019), we included an *ARCADIA overall quality score* to calculate some measures of validity and reliability. We calculated the *ARCADIA overall quality score* (from 0 to 100) by dividing the total number of items ticked u2018Yesu2019 by the total number of items (n=14) excluding the items ticked as u2018NAu2019 and multiplying the final result by 100. A checklist can be easily converted to a scale. However, as we reported in our previous paper (4), we believe that it is not appropriate to assess the quality of peer review reports using a scale. **7.tGoing back to the name and purpose of your tool - in your reply to BMJ reviewers you stated: u201cwe decided to develop a checklist of items that could be used to show the independent influence of each item on the overall peer review report qualityu201d. I do not see that you provided a measurement of the overall report quality. And in your title, you stated ARCADIA is a tool to assess quality or peer review reports, not a checklist of 14 items that could independently influence the quality which you do not assess.** As we have previously explained, u2018qualityu2019 is a multidimensional concept. This means that multiple dimensions components are needed to assess quality in its complexity. In our case, we identified 14 components (defined as items). Each of them independently influences the quality of a peer review report as we reported in our reply to BMJ Open reviewers. So, ARCADIA as a checklist provides a list of 14 items for assessing the quality of a peer review report, without providing an overall score but showing if the peer reviewer appropriately comments on each of the quality components of a peer review report (i.e., ARCADIA items). **8.tOne of items in your tool is to state whether reviewers checked if study limitations are acknowledged? What if a paper states a limitation that in a reviewer s opinion completely invalidates the study? How should that be addressed?** u201cA study limitation that completely invalidates the studyu201d means that the study methods were not appropriate to the studyu2019s aims, as well as appropriately used. In this case, the ARCADIAu2019s user should check if the peer reviewer comments these flaws in the peer review report. ARCADIA includes an item on study methods (ARCADIA item 2a) defined as u201cthe peer reviewer should evaluate the soundness of the study methods, such as selection of the study design, assessment of the risk of bias, etc., to understand whether the methods were appropriate to the studyu2019s aims, as well as if they were properly used and reportedu201d (see supplementary file 7). **9.tIt is nice that you acknowledged the tool is not validated, but I wonder why you deemed it good scientific practice to publish a non-validated tool. If you wanted to document the process of its creation, that is admirable, but then, in my opinion, you should have at least restrained from having an implication section that states your tool u201ccould be REGULARLY used by editors to evaluate the reviewersu2019 work,u201d or that u201cit can also be used as an outcome when evaluating interventions in order to improve the peer-review process.u201d With no proof of its usefulness, how scientific is it to make such recommendations, even when hiding behind the modality of verbs can and could? A discussion on guidance on the number of evaluators (i.e. should one worry about inter-rater agreement) and different appraisal by editors and authors and other stakeholders (which I already mentioned) wouldu2019ve been welcome. Additionally, you excluded in the implication section the use of your tool by authors, yet they are in your tools name.** We have already validated ARCADIA and we are planning to publish the paper related to this study soon. We decided to publish the development and validation of ARCADIA in two separate articles because we believe that it is important to accurately report research in order to be easily replicated. Merging ARCADIAu2019s development and validation would result in a long and  crowded manuscript including potentially incomplete information due to wordcount restrictions. Moreover, there are several publications of non-validated tools in the literature (8u201311). We therefore respectfully disagree with the reviewer that u201cit is not a good scientific practice published to publish a non-validated toolu201d. As we have already stated, we used the word u2018authorsu2019 (included also in the name of the tool) to refer to all people conducting research and therefore publishing and or reviewing articles. So, ARCADIA is not limited to authors and editors, but it can also be used by researchers interested in improving their peer review reports or in conducting research to evaluate interventions aimed at improving the peer review process. **10.tFinally, I considered using your checklist to enrich my review, but I choose not to do so. I would not want it to be considered endorsement of the tool. And I observed it was also not used by the BMJ reviewers in their reports, nor did you use it when replying to their reviews. Additionally, as I provide here a post-publication peer review report, and those have not been discussed in your paper, it might be improper to use it for unintended purposes (e.g. preprints, postprint or reviews by friends colleagues before submission to a journal or upload to a server). In my opinion, researchers should do more critical thinking, simulations, piloting or testing before making claims as the ones you stated in this paper. Therefore, I would recommend you retract and republish your paper when you address its shortcomings.** In science, the word u2018retractionu2019 has a strong meaning. It implies that authors committed research misconduct such as plagiarism, fabrication or falsification of data . This is not the case. We therefore invite the reviewer to carefully read the guidelines on retractions by COPE (12) before making such recommendations. The reviewer mainly criticised the definition of peer review quality we provided and the language used, asking for more clarification on the scope of impact of the tool and future work. We believe that these are not valid reasons to ask journalu2019s editors to retract an article. We therefore consider the retraction recommendation inappropriate. Our paper reports the first version of a checklist developed following high methodological standards. As a first version, the checklist has some weaknesses, which have been also identified with the validation study that we have recently conducted. As it usually happens, more versions of a tool are developed to address its limitations. Therefore, we are more than happy to conduct more research (or invite other researchers to conduct more research on our tool) in order to address the weaknesses identified in the first version of ARCADIA. **References** 1. van Rooyen S, Black N, Godlee F. Development of the Review Q