<figure><img src="image_1.png" /></figure>

HAL
open science

# Reliability of citations of medRxiv preprints in articles published on COVID-19 in the world leading medical journals

Jean-François Gehanno, Julien Grosjean, Stefan Darmoni, Laetitia Rollin

► To cite this version:

Jean-François Gehanno, Julien Grosjean, Stefan Darmoni, Laetitia Rollin. Reliability of citations of medRxiv preprints in articles published on COVID-19 in the world leading medical journals. PLoS ONE, 2022, 17 (8), pp.e0264661. 10.1371/journal.pone.0264661. hal-03896489

HAL Id: hal-03896489

https://hal.science/hal-03896489

Submitted on 21 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_1.png" /></figure>

RESEARCH ARTICLE

# Reliability of citations of medRxiv preprints in articles published on COVID-19 in the world leading medical journals

Jean-Francois Gehanno<sup>1,2*</sup>, Julien Grosjean<sup>2,3</sup>, Stefan J. Darmoni<sup>2,3</sup>,
Laetitia Rollin<sup>1,2</sup>

1 Department of Occupational Medicine, Rouen University Hospital, Rouen, France, 2 Inserm, Rouen University, Sorbonne University, University of Paris 13, Laboratory of Medical Informatics and Knowledge Engineering in e-Health, LIMICS, Paris, France, 3 Department of Biomedical Informatics, Rouen University Hospital, Rouen France

* jf.gehanno@chu-rouen.fr

## Abstract

<figure><img src="image_2.png" /></figure>

OPEN ACCESS

**Citation:** Gehanno J-F, Grosjean J, Darmoni SJ, Rollin L (2022) Reliability of citations of medRxiv preprints in articles published on COVID-19 in the world leading medical journals. PLoS ONE 17(8): e0264661. https://doi.org/10.1371/journal.pone.0264661

**Editor:** Venkatesh Shankar Madhugiri, National Institute of Mental Health and Neuro Sciences, INDIA

**Received:** February 14, 2022

**Accepted:** July 25, 2022

**Published:** August 10, 2022

**Peer Review History:** PLOS recognizes the benefits of transparency in the peer review process; therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. The editorial history of this article is available here: https://doi.org/10.1371/journal.pone.0264661

**Copyright:** © 2022 Gehanno et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

**Data Availability Statement:** The datasets used and analysed during the current study are available at https://doi.org/10.5281/zenodo.5985960.

## Introduction

Preprints have been widely cited during the COVID-19 pandemics, even in the major medical journals. However, since subsequent publication of preprint is not always mentioned in preprint repositories, some may be inappropriately cited or quoted. Our objectives were to assess the reliability of preprint citations in articles on COVID-19, to the rate of publication of preprints cited in these articles and to compare, if relevant, the content of the preprints to their published version.

## Methods

Articles published on COVID in 2020 in the BMJ, The Lancet, the JAMA and the NEJM were manually screened to identify all articles citing at least one preprint from medRxiv. We searched PubMed, Google and Google Scholar to assess if the preprint had been published in a peer-reviewed journal, and when. Published articles were screened to assess if the title, data or conclusions were identical to the preprint version.

## Results

Among the 205 research articles on COVID published by the four major medical journals in 2020, 60 (29.3%) cited at least one medRxiv preprint. Among the 182 preprints cited, 124 were published in a peer-reviewed journal, with 51 (41.1%) before the citing article was published online and 73 (58.9%) later. There were differences in the title, the data or the conclusion between the preprint cited and the published version for nearly half of them. MedRxiv did not mentioned the publication for 53 (42.7%) of preprints.

## Conclusions

More than a quarter of preprints citations were inappropriate since preprints were in fact already published at the time of publication of the citing article, often with a different content.
<figure><img src="image_3.png" /></figure>

RESEARCH ARTICLE

# Reliability of citations of medRxiv preprints in articles published on COVID-19 in the world leading medical journals

Jean-Francois Gehanno<sup>1,2*</sup>, Julien Grosjean<sup>2,3</sup>, Stefan J. Darmoni<sup>2,3</sup>,
Laetitia Rollin<sup>1,2</sup>

1 Department of Occupational Medicine, Rouen University Hospital, Rouen, France, 2 Inserm, Rouen University, Sorbonne University, University of Paris 13, Laboratory of Medical Informatics and Knowledge Engineering in e-Health, LIMICS, Paris, France, 3 Department of Biomedical Informatics, Rouen University Hospital, Rouen, France

* jf.gehanno@chu-rouen.fr

## Abstract

<figure><img src="image_4.png" /></figure>

OPEN ACCESS

**Citation:** Gehanno J-F, Grosjean J, Darmoni SJ, Rollin L (2022) Reliability of citations of medRxiv preprints in articles published on COVID-19 in the world leading medical journals. PLoS ONE 17(8): e0264661. https://doi.org/10.1371/journal.pone.0264661

**Editor:** Venkatesh Shankar Madhugiri, National Institute of Mental Health and Neuro Sciences, INDIA

**Received:** February 14, 2022

**Accepted:** July 25, 2022

**Published:** August 10, 2022

**Peer Review History:** PLOS recognizes the benefits of transparency in the peer review process; therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. The editorial history of this article is available here: https://doi.org/10.1371/journal.pone.0264661

**Copyright:** © 2022 Gehanno et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

**Data Availability Statement:** The datasets used and analysed during the current study are available at https://doi.org/10.5281/zenodo.5985960.

## Introduction

Preprints have been widely cited during the COVID-19 pandemics, even in the major medical journals. However, since subsequent publication of preprint is not always mentioned in preprint repositories, some may be inappropriately cited or quoted. Our objectives were to assess the reliability of preprint citations in articles on COVID-19, to the rate of publication of preprints cited in these articles and to compare, if relevant, the content of the preprints to their published version.

## Methods

Articles published on COVID in 2020 in the BMJ, The Lancet, the JAMA and the NEJM were manually screened to identify all articles citing at least one preprint from medRxiv. We searched PubMed, Google and Google Scholar to assess if the preprint had been published in a peer-reviewed journal, and when. Published articles were screened to assess if the title, data or conclusions were identical to the preprint version.

## Results

Among the 205 research articles on COVID published by the four major medical journals in 2020, 60 (29.3%) cited at least one medRxiv preprint. Among the 182 preprints cited, 124 were published in a peer-reviewed journal, with 51 (41.1%) before the citing article was published online and 73 (58.9%) later. There were differences in the title, the data or the conclusion between the preprint cited and the published version for nearly half of them. MedRxiv did not mentioned the publication for 53 (42.7%) of preprints.

## Conclusions

More than a quarter of preprints citations were inappropriate since preprints were in fact already published at the time of publication of the citing article, often with a different content.
**Funding:** The authors received no specific funding for this work.

**Competing interests:** The authors have declared that no competing interests exist.

Authors and editors should check the accuracy of the citations and of the quotations of preprints before publishing manuscripts that cite them.

## Introduction

The scientific community has reacted rapidly to the medical challenges generated by the coronavirus disease 2019 (COVID-19) pandemic, with a steady increase of literature in peer-reviewed journals but also in preprint repositories [1–3]. Yet, a steep increase in the number of posted preprints was observed between January and December 2020, from 797 to 14 290, out of which 8858 (62.0%) were COVID-19-related [4].

However, preprints posted on preprint platforms are not subject to peer-review, and therefore critical appraisal, until they are submitted to peer-reviewed journals. Only a small proportion will be converted to scientific publications and the share of converted preprints shows a declining trend over time [4–7]. For those finally passing the peer-review process, peer-reviewers' comments and critiques lead authors to revise their manuscripts, substantially at times [4, 6, 8]. In that case, the preprint is not the latest version of the work anymore and identifying if it has been published, and referring to the published version instead of the preprint is an important issue as it provides readers with the latest version of a now certified work.

However, matching preprints to subsequent peer review publications is a challenge. Once a preprint is published in a peer-reviewed venue, the preprint server is supposed to update its web page, adding a prominent hyperlink leading to the newly published work. It has been observed that the preprint server MedRxiv reports only 39.7% of all existing publication links [9]. It is therefore probable that some studies are inadequately cited as preprints, although the work cited has been already published, and sometimes with a different content.

The first objective of our study was to assess the reliability of preprint citations in articles on COVID-19 published by the world leading medical journals. The second objective was to assess the rate of publication of preprints cited in articles appearing in those journals and the last objective was to compare, if relevant, the content of the preprints to their published version.

## Materials and methods

All research articles published on COVID-19 in the first 11 months of 2020 in the British Medical Journal (BMJ), The Lancet, the Journal of the American Medical Association (JAMA) and the New England Journal of Medicine (NEJM) were identified through the journal websites. We then manually screened these articles to identify all articles citing at least one preprint from MedRxiv, in the text as a footnote but also in the reference list since all these journals allow citation of preprints in the reference list of articles they published.

In September 2021, we searched PubMed, Google and Google Scholar to assess if each preprint had been published in a peer-reviewed journal, using the title and the first author in the search string. For each of those which had been published, we compared preprint version and associated journal article to identify changes in the title and two evidence components: study results (e.g. numeric changes in sample size, hazard ratio, odds ratio, event rate, or change in p-value) and abstract conclusions (staying positive, negative or neutral regarding the intervention effect, and reporting uncertainty in the findings or not). When different versions of the preprint were available, we used the version existing at the time the citing article was first published online. The results were considered different when the sizes of the samples, the main
**Funding:** The authors received no specific funding for this work.

**Competing interests:** The authors have declared that no competing interests exist.

Authors and editors should check the accuracy of the citations and of the quotations of preprints before publishing manuscripts that cite them.

## Introduction

The scientific community has reacted rapidly to the medical challenges generated by the coronavirus disease 2019 (COVID-19) pandemic, with a steady increase of literature in peer-reviewed journals but also in preprint repositories [1–3]. Yet, a steep increase in the number of posted preprints was observed between January and December 2020, from 797 to 14 290, out of which 8858 (62.0%) were COVID-19-related [4].

However, preprints posted on preprint platforms are not subject to peer-review, and therefore critical appraisal, until they are submitted to peer-reviewed journals. Only a small proportion will be converted to scientific publications and the share of converted preprints shows a declining trend over time [4–7]. For those finally passing the peer-review process, peer-reviewers' comments and critiques lead authors to revise their manuscripts, substantially at times [4, 6, 8]. In that case, the preprint is not the latest version of the work anymore and identifying if it has been published, and referring to the published version instead of the preprint is an important issue as it provides readers with the latest version of a now certified work.

However, matching preprints to subsequent peer review publications is a challenge. Once a preprint is published in a peer-reviewed venue, the preprint server is supposed to update its web page, adding a prominent hyperlink leading to the newly published work. It has been observed that the preprint server MedRxiv reports only 39.7% of all existing publication links [9]. It is therefore probable that some studies are inadequately cited as preprints, although the work cited has been already published, and sometimes with a different content.

The first objective of our study was to assess the reliability of preprint citations in articles on COVID-19 published by the world leading medical journals. The second objective was to assess the rate of publication of preprints cited in articles appearing in those journals and the last objective was to compare, if relevant, the content of the preprints to their published version.

## Materials and methods

All research articles published on COVID-19 in the first 11 months of 2020 in the British Medical Journal (BMJ), The Lancet, the Journal of the American Medical Association (JAMA) and the New England Journal of Medicine (NEJM) were identified through the journal websites. We then manually screened these articles to identify all articles citing at least one preprint from MedRxiv, in the text as a footnote but also in the reference list since all these journals allow citation of preprints in the reference list of articles they published.

In September 2021, we searched PubMed, Google and Google Scholar to assess if each preprint had been published in a peer-reviewed journal, using the title and the first author in the search string. For each of those which had been published, we compared preprint version and associated journal article to identify changes in the title and two evidence components: study results (e.g. numeric changes in sample size, hazard ratio, odds ratio, event rate, or change in p-value) and abstract conclusions (staying positive, negative or neutral regarding the intervention effect, and reporting uncertainty in the findings or not). When different versions of the preprint were available, we used the version existing at the time the citing article was first published online. The results were considered different when the sizes of the samples, the main
quantitative results or the level of significance were different. The conclusions were considered similar if the conclusions in the abstract of the published version remained positive, neutral, or negative regarding the effect of the intervention, or reported the same uncertainties. The conclusions were considered different otherwise. We finally collected the date of first posting of the preprint in MedRxiv, the date of publication of the article citing the preprint and the date of online publication in a journal of the preprint, if relevant.

Data were included in an Excel spreadsheet (Microsoft®).

## Results

The four major medical journals published 205 research articles on COVID until the 1st of December 2020, among which 60 (29.3%) cited at least one preprint from medRxiv (Table 1).

Overall, 182 preprints were cited, among which 124 (68.1%) have been published in a peer-reviewed journal (Fig 1). None of the preprints cited were withdrawn from medRxiv server.

However, among those 124 “preprints”, 51 (41.1%) had already been published when the citing article was published online, at least 2 months before for 33 of them, and were, in fact, not preprints anymore at that time.

Among these 51 superseded preprints, differences between the preprint version that was cited and the published version were observed in the data or the conclusions for 21 (40%) of them and in the title for 23 (45%) of them (Table 2).

For the 73 other preprints, i.e. the 73 (124–51) which were not already published when the citing article was made available online, differences between the preprint version that was cited and the published version were observed in the data or the conclusions for 33 (45%) of them and in the title for 24 (33%) of them (Table 2).

The differences in the results were often linked to different sample sizes. For example, in the study assessing factors associated with hospital admission and critical illness, the preprint version included 4,103 participants whereas the article published in the BMJ included 5,279 participants [10, 11]. This led to different conclusions, with an odds ratio for hospital admission among people > 75 years shifting from 66.8% (95% confidence interval 44.7 to 102.6) in the preprint to 37.9% (95% confidence interval 26.1 to 56.0) in the final article.

If we consider only the preprints that were not already published when the citing article was available online, the rate of publication was 55.7% (73 out of 131), with a median publication delay of 65 days (minimum 1 –maximum 486).

The fact that the preprint had been published in a peer-reviewed journal was not mentioned in medRxiv for 20 (39.2%) of the 51 superseded preprints and 33 (45.2%) of the 73 other preprints (p = 0.23).

Overall, the median interval between the first posting of a preprint in MedRxiv and its publication in a peer-reviewed journal was 94 days.

<table><caption>Table 1. Research articles published on COVID up to the 1st of December 2020, and citation of preprints.</caption><thead><tr><td></td><td>Number of articles published</td><td>Articles citing at least one preprint (%)</td><td>Total number of references in articles citing at least one preprint</td><td>Number of preprints among the references (%)</td></tr></thead><tbody><tr><td>BMJ</td><td>35</td><td>20 (57.1%)</td><td>956</td><td>91 (9.5%)</td></tr><tr><td>JAMA</td><td>89</td><td>14 (15.7%)</td><td>402</td><td>26 (6.5%)</td></tr><tr><td>Lancet</td><td>37</td><td>16 (43.2%)</td><td>647</td><td>53 (8.2%)</td></tr><tr><td>NEJM</td><td>44</td><td>10 (22.7%)</td><td>282</td><td>12 (4.3%)</td></tr><tr><td>Total</td><td>205</td><td>60 (29.3%)</td><td>2287</td><td>182 (8%)</td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0264661.t001
quantitative results or the level of significance were different. The conclusions were considered similar if the conclusions in the abstract of the published version remained positive, neutral, or negative regarding the effect of the intervention, or reported the same uncertainties. The conclusions were considered different otherwise. We finally collected the date of first posting of the preprint in MedRxiv, the date of publication of the article citing the preprint and the date of online publication in a journal of the preprint, if relevant.

Data were included in an Excel spreadsheet (Microsoft®).

## Results

The four major medical journals published 205 research articles on COVID until the 1st of December 2020, among which 60 (29.3%) cited at least one preprint from medRxiv (Table 1).

Overall, 182 preprints were cited, among which 124 (68.1%) have been published in a peer-reviewed journal (Fig 1). None of the preprints cited were withdrawn from medRxiv server.

However, among those 124 “preprints”, 51 (41.1%) had already been published when the citing article was published online, at least 2 months before for 33 of them, and were, in fact, not preprints anymore at that time.

Among these 51 superseded preprints, differences between the preprint version that was cited and the published version were observed in the data or the conclusions for 21 (40%) of them and in the title for 23 (45%) of them (Table 2).

For the 73 other preprints, i.e. the 73 (124–51) which were not already published when the citing article was made available online, differences between the preprint version that was cited and the published version were observed in the data or the conclusions for 33 (45%) of them and in the title for 24 (33%) of them (Table 2).

The differences in the results were often linked to different sample sizes. For example, in the study assessing factors associated with hospital admission and critical illness, the preprint version included 4,103 participants whereas the article published in the BMJ included 5,279 participants [10, 11]. This led to different conclusions, with an odds ratio for hospital admission among people > 75 years shifting from 66.8% (95% confidence interval 44.7 to 102.6) in the preprint to 37.9% (95% confidence interval 26.1 to 56.0) in the final article.

If we consider only the preprints that were not already published when the citing article was available online, the rate of publication was 55.7% (73 out of 131), with a median publication delay of 65 days (minimum 1 –maximum 486).

The fact that the preprint had been published in a peer-reviewed journal was not mentioned in medRxiv for 20 (39.2%) of the 51 superseded preprints and 33 (45.2%) of the 73 other preprints (p = 0.23).

Overall, the median interval between the first posting of a preprint in MedRxiv and its publication in a peer-reviewed journal was 94 days.

<table><caption>Table 1. Research articles published on COVID up to the 1st of December 2020, and citation of preprints.</caption><thead><tr><td></td><td>Number of articles published</td><td>Articles citing at least one preprint (%)</td><td>Total number of references in articles citing at least one preprint</td><td>Number of preprints among the references (%)</td></tr></thead><tbody><tr><td>BMJ</td><td>35</td><td>20 (57.1%)</td><td>956</td><td>91 (9.5%)</td></tr><tr><td>JAMA</td><td>89</td><td>14 (15.7%)</td><td>402</td><td>26 (6.5%)</td></tr><tr><td>Lancet</td><td>37</td><td>16 (43.2%)</td><td>647</td><td>53 (8.2%)</td></tr><tr><td>NEJM</td><td>44</td><td>10 (22.7%)</td><td>282</td><td>12 (4.3%)</td></tr><tr><td>Total</td><td>205</td><td>60 (29.3%)</td><td>2287</td><td>182 (8%)</td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0264661.t001
<figure><img src="image_3.png" /><figcaption>Fig 1. Flow chart of preprint citations and publications.</figcaption></figure>

https://doi.org/10.1371/journal.pone.0264661.g001

Discussion

We found that preprints are frequently cited in research articles on COVID-19 published in the world's leading medical journals. However, nearly half of the citations are inappropriate since the preprint was already published when the citing article was published online. Furthermore, many preprints cited will not be published in peer-reviewed journals, or will be but with different data.

Articles on COVID have been published in many different journals but we chose to investigate the BMJ, The Lancet, the JAMA and the NEJM because they are the medical journals that have published much of the research on Covid-19 [12, 13].
<figure><img src="image_5.png" /><figcaption>**Fig 1. Flow chart of preprint citations and publications.**</figcaption></figure>

https://doi.org/10.1371/journal.pone.0264661.g001

Discussion

We found that preprints are frequently cited in research articles on COVID-19 published in the world's leading medical journals. However, nearly half of the citations are inappropriate since the preprint was already published when the citing article was published online. Furthermore, many preprints cited will not be published in peer-reviewed journals, or will be but with different data.

Articles on COVID have been published in many different journals but we chose to investigate the BMJ, The Lancet, the JAMA and the NEJM because they are the medical journals that have published much of the research on Covid-19 [12, 13].
<table><caption>Table 2. Differences between the preprint and its published version.</caption><thead><tr><th></th><th>SP<sup>a</sup></th><th>P<sup>b</sup></th></tr></thead><tbody><tr><td>Same data / same conclusion</td><td>30 (59%)</td><td>40 (55%)</td></tr><tr><td>Same data / different conclusion</td><td>0 (0%)</td><td>0 (0%)</td></tr><tr><td>Different data / same conclusion</td><td>16 (31%)</td><td>20 (27%)</td></tr><tr><td>Different data / different conclusion</td><td>5 (10%)</td><td>13 (18%)</td></tr><tr><td>Different title</td><td>23 (45%)</td><td>24 (33%)</td></tr></tbody></table>

https://doi.org/10.1371/journal.pone.0264661.t002

Many preprint platforms exist, the oldest coming from the early 1990th in the physical sciences. We chose to use MedRxiv, which was launched in June 2019 to provide a dedicated platform and processes for preprints in medicine and health related sciences, because it became particularly popular during the pandemic and is the server that hosted the largest number of preprints on COVID-19 [1, 14].

More than 40% of these citations were inappropriate since, at the time of online publication of the citing article, the preprint was already published in a peer-reviewed journal.

The fact that a large proportion of preprints cited in articles published in peer-reviewed journals were not preprint anymore at the time of publication of the citing article has not been reported in the literature, to our knowledge. This would not be a problem if the contents of the preprint and its published version were similar. However, we observed differences in the data and even a different conclusion for nearly half and 10% of them, respectively. Therefore, searching for a potential peer-reviewed article before citing a paper as a preprint would better reflect the latest evidence available in many cases. However, identifying the subsequent publication of a preprint can be difficult since we observed, that MedRxiv often does not mention the publication of the preprints in a peer-reviewed journal, although it is supposed to display link to journal publication within a month [14], which confirms previous studies [9, 15]. Furthermore, in our study, nearly half of the preprints have a different title than their published version and just a copy and paste in PubMed or Google might not allow identifying the published version.

The interest and validity of preprints have been largely debated, with pros and cons, even before the COVID-19 pandemics [16]. This debate has largely increased about COVID preprints, balancing between the interests of early dissemination of knowledge and the risk of postulated claims without evidence, which will be used by scientists, healthcare workers, and the general public [4, 17, 18].

Preprints posted on platforms are not subject to peer-review and therefore critical appraisal, but some preprint platforms perform screening check, usually related to scope of the article (e. g. scientific content, not spam, relevant material, language), plagiarism and legal/ethical/societal issues and compliance [14]. Among the 44 preprint platforms identified by Kirkham et al. as having biomedical and medical scope, only three of them (Research Square, bioRxiv and medRxiv) check whether the content contains unfounded medical claims [14].

Despite these checks, MedRxiv platform states that “preprints are preliminary reports of work that have not been certified by peer review. They should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information »

However, preprints have been widely used during the COVID-19 pandemic where much of the preliminary evidence has been made available through preprints at the time of WHO

<footnote><sup>a</sup>SP: supersided preprints, preprint published before the publication of the citing article</footnote>

<footnote><sup>b</sup>P: preprint published after the publication of the citing article</footnote>
<table><caption>Table 2. Differences between the preprint and its published version.</caption><thead><tr><th></th><th>SP<sup>a</sup></th><th>P<sup>b</sup></th></tr></thead><tbody><tr><td>Same data / same conclusion</td><td>30 (59%)</td><td>40 (55%)</td></tr><tr><td>Same data / different conclusion</td><td>0 (0%)</td><td>0 (0%)</td></tr><tr><td>Different data / same conclusion</td><td>16 (31%)</td><td>20 (27%)</td></tr><tr><td>Different data / different conclusion</td><td>5 (10%)</td><td>13 (18%)</td></tr><tr><td>Different title</td><td>23 (45%)</td><td>24 (33%)</td></tr></tbody></table>

<sup>a</sup>SP: supersided preprints, preprint published before the publication of the citing article

<sup>b</sup>P: preprint published after the publication of the citing article

https://doi.org/10.1371/journal.pone.0264661.t002

Many preprint platforms exist, the oldest coming from the early 1990th in the physical sciences. We chose to use MedRxiv, which was launched in June 2019 to provide a dedicated platform and processes for preprints in medicine and health related sciences, because it became particularly popular during the pandemic and is the server that hosted the largest number of preprints on COVID-19 [1, 14].

More than 40% of these citations were inappropriate since, at the time of online publication of the citing article, the preprint was already published in a peer-reviewed journal.

The fact that a large proportion of preprints cited in articles published in peer-reviewed journals were not preprint anymore at the time of publication of the citing article has not been reported in the literature, to our knowledge. This would not be a problem if the contents of the preprint and its published version were similar. However, we observed differences in the data and even a different conclusion for nearly half and 10% of them, respectively. Therefore, searching for a potential peer-reviewed article before citing a paper as a preprint would better reflect the latest evidence available in many cases. However, identifying the subsequent publication of a preprint can be difficult since we observed, that MedRxiv often does not mention the publication of the preprints in a peer-reviewed journal, although it is supposed to display link to journal publication within a month [14], which confirms previous studies [9, 15]. Furthermore, in our study, nearly half of the preprints have a different title than their published version and just a copy and paste in PubMed or Google might not allow identifying the published version.

The interest and validity of preprints have been largely debated, with pros and cons, even before the COVID-19 pandemics [16]. This debate has largely increased about COVID preprints, balancing between the interests of early dissemination of knowledge and the risk of postulated claims without evidence, which will be used by scientists, healthcare workers, and the general public [4, 17, 18].

Preprints posted on platforms are not subject to peer-review and therefore critical appraisal, but some preprint platforms perform screening check, usually related to scope of the article (e. g. scientific content, not spam, relevant material, language), plagiarism and legal/ethical/societal issues and compliance [14]. Among the 44 preprint platforms identified by Kirkham et al. as having biomedical and medical scope, only three of them (Research Square, bioRxiv and medRxiv) check whether the content contains unfounded medical claims [14].

Despite these checks, MedRxiv platform states that “preprints are preliminary reports of work that have not been certified by peer review. They should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information »

However, preprints have been widely used during the COVID-19 pandemic where much of the preliminary evidence has been made available through preprints at the time of WHO
declaring the epidemic a public health emergency. Only a fraction of these preprints is subsequently published in peer-reviewed journals.

In our study, nearly half of the preprints which had not been published in a peer-reviewed journal before being cited as a preprint, were not subsequently published in peer-reviewed journals.

Our study may have underestimated the real rate of publication of cited preprints. However, among the preprints that were not finally published, all but one had been posted in MedRxiv before mid-August 2020 and we searched for publication more than one year after the posting. A subsequent publication is therefore improbable, considering the usual median delay of publication reported in other studies, ranging from 28 to 110 days [5–7, 19], with a mean of 68 days according to a large review [1]. In our study, all the preprints subsequently published were published within 10 months following the publication of the citing article, but one.

The rate of subsequent publications of preprints cited in the four world-leading medical journals is higher than the rate of publication of preprints reported in other studies, ranging from 6.9 to 21.1% [1, 4, 5, 7, 19–21]. However, the rates of publication reported in those studies are probably underestimated since they relied on information about subsequent publication in MedRxiv, which we observed to be largely inaccurate, confirming previous reports [9].

For the preprints which were finally published, the differences between the preprints and their published version were frequent and close to those reported in another study on 139 preprints, with changes in the study results or the abstract conclusions between the first preprint version to journal article for 38% and 29% of preprints, respectively [6]. However, when studying 67 studies on Covid-19 prevention or treatment, Bero et al. found only 23 articles (34%) that had no discrepancies in results reporting between preprints and journal publications [8].

The changes in the conclusions were less frequent than the changes in studies' result, mostly because conclusions were usually broad, such as "among patients hospitalized with Covid-19, those who received hydroxychloroquine did not have a lower incidence of death at 28 days than those who received usual care" [22].

These changes can be the consequence of publishing preliminary results in a preprint and final results in a peer-reviewed journal, of the fact that some preprints receive comments that allow authors to improve their manuscript prior to submission to a journal, or of the improvement of the quality of the report following the peer reviewed process.

Preprints on COVID-19 posted in preprint servers are shorter and contain less references than non-COVID-19 preprints, which suggests that authors tend to publish preliminary results [1].

Concerning gradual quality improving, although preprint servers offer authors the opportunity to post updated versions of a preprint, enabling them to incorporate feedback, correct mistakes, or add additional data and analysis, the majority of preprints on COVID-19 in preprint servers exist in fact as only a single version [1]. A study on bioRxiv preprints showed that less than 10% of preprints received at least one comment, and one third were posted by the preprint's authors themselves [23].

Although little empirical evidence is available to support the use of editorial peer review as mechanism to ensure quality of biomedical research [24], the peer review process is a still considered a cornerstone to improve the quality of scientific publications [25].

In light of that, in our study, when combining the 58 preprints that were published and the 56 that appeared finally in a peer-reviewed journal, but with different results or conclusions, we can consider that 62.6% of the quotations of preprints may not be fully reliable.

Another interest of preprints is that they allow free access to research findings, while a large proportion of journal articles often remain behind subscription paywalls. In response to the
declaring the epidemic a public health emergency. Only a fraction of these preprints is subsequently published in peer-reviewed journals.

In our study, nearly half of the preprints which had not been published in a peer-reviewed journal before being cited as a preprint, were not subsequently published in peer-reviewed journals.

Our study may have underestimated the real rate of publication of cited preprints. However, among the preprints that were not finally published, all but one had been posted in MedRxiv before mid-August 2020 and we searched for publication more than one year after the posting. A subsequent publication is therefore improbable, considering the usual median delay of publication reported in other studies, ranging from 28 to 110 days [5–7, 19], with a mean of 68 days according to a large review [1]. In our study, all the preprints subsequently published were published within 10 months following the publication of the citing article, but one.

The rate of subsequent publications of preprints cited in the four world-leading medical journals is higher than the rate of publication of preprints reported in other studies, ranging from 6.9 to 21.1% [1, 4, 5, 7, 19–21]. However, the rates of publication reported in those studies are probably underestimated since they relied on information about subsequent publication in MedRxiv, which we observed to be largely inaccurate, confirming previous reports [9].

For the preprints which were finally published, the differences between the preprints and their published version were frequent and close to those reported in another study on 139 preprints, with changes in the study results or the abstract conclusions between the first preprint version to journal article for 38% and 29% of preprints, respectively [6]. However, when studying 67 studies on Covid-19 prevention or treatment, Bero et al. found only 23 articles (34%) that had no discrepancies in results reporting between preprints and journal publications [8].

The changes in the conclusions were less frequent than the changes in studies' result, mostly because conclusions were usually broad, such as “among patients hospitalized with Covid-19, those who received hydroxychloroquine did not have a lower incidence of death at 28 days than those who received usual care” [22].

These changes can be the consequence of publishing preliminary results in a preprint and final results in a peer-reviewed journal, of the fact that some preprints receive comments that allow authors to improve their manuscript prior to submission to a journal, or of the improvement of the quality of the report following the peer reviewed process.

Preprints on COVID-19 posted in preprint servers are shorter and contain less references than non-COVID-19 preprints, which suggests that authors tend to publish preliminary results [1].

Concerning gradual quality improving, although preprint servers offer authors the opportunity to post updated versions of a preprint, enabling them to incorporate feedback, correct mistakes, or add additional data and analysis, the majority of preprints on COVID-19 in preprint servers exist in fact as only a single version [1]. A study on bioRxiv preprints showed that less than 10% of preprints received at least one comment, and one third were posted by the preprint's authors themselves [23].

Although little empirical evidence is available to support the use of editorial peer review as mechanism to ensure quality of biomedical research [24], the peer review process is a still considered a cornerstone to improve the quality of scientific publications [25].

In light of that, in our study, when combining the 58 preprints that were published and the 56 that appeared finally in a peer-reviewed journal, but with different results or conclusions, we can consider that 62.6% of the quotations of preprints may not be fully reliable.

Another interest of preprints is that they allow free access to research findings, while a large proportion of journal articles often remain behind subscription paywalls. In response to the
pandemic, a number of journal publishers began to alter their open-access policies in relation to COVID-19 manuscripts and made COVID-19 literature temporarily open access [1].

Finally, the quality of research had to be, in response to the pandemic, put in balance with the need to get rapidly new information to tackle this new threat. Communicating science through preprints allow to share research at a faster rate and with greater transparency than allowed by the current journal infrastructure. Nevertheless, according to the shortcomings of preprints, they should not appear in the reference list of journals but could be cited in the text as such, or, at least, the preprint status should be made clear in the text of a manuscript if citing the research and not just in the reference list [25, 26].

A promising alternative is the introduction of brief summaries of preprints with significant potential to impact clinical practice in a specific section of biomedical journals [27, 28]. Another one is to replace the word “preprint”, which can be misleading, by ’Unrefereed manuscript’, "Manuscript awaiting peer review" or "Non-reviewed manuscript" [29].

## Conclusions

More than 40% of preprints cited in the major medical journals were in fact already published at the time of publication of the citing article. Therefore, authors should check the accuracy of the citation and the quotations of preprints just before submitting the manuscript, and once again when signing the proofs. They should not rely only on the information displayed on MedRxiv website to identify subsequent publication. To overcome medRxiv weaknesses, authors should manually search bibliographic databases to determine if a preprint has been subsequently peer-reviewed and published, keeping in mind the nearly half of the preprints have a different title than their published version. Publishers should also check the accuracy of the citations of preprints before online publication of citing articles.

The debate on the interest of preprints has largely increased during the COVID preprints, balancing between the interests of early dissemination of knowledge and the risk of postulated claims without evidence. However, considering the significant number of changes in the content or even the conclusions between the preprints and their published version, quotations of preprints should be considered with caution, even in the articles published in the major medical journals.

## Supporting information

S1 File. https://doi.org/10.5281/zenodo.5985960. (XLSX)

## Acknowledgments

The authors thank Mrs Anna Gehanno for her help in editing the manuscript.

## Author Contributions

**Conceptualization:** Jean-Francois Gehanno, Laetitia Rollin.

**Data curation:** Jean-Francois Gehanno, Laetitia Rollin.

**Formal analysis:** Jean-Francois Gehanno, Julien Grosjean, Stefan J. Darmoni, Laetitia Rollin.

**Supervision:** Laetitia Rollin.

**Validation:** Jean-Francois Gehanno.

**Writing – original draft:** Jean-Francois Gehanno, Laetitia Rollin.
pandemic, a number of journal publishers began to alter their open-access policies in relation to COVID-19 manuscripts and made COVID-19 literature temporarily open access [1].

Finally, the quality of research had to be, in response to the pandemic, put in balance with the need to get rapidly new information to tackle this new threat. Communicating science through preprints allow to share research at a faster rate and with greater transparency than allowed by the current journal infrastructure. Nevertheless, according to the shortcomings of preprints, they should not appear in the reference list of journals but could be cited in the text as such, or, at least, the preprint status should be made clear in the text of a manuscript if citing the research and not just in the reference list [25, 26].

A promising alternative is the introduction of brief summaries of preprints with significant potential to impact clinical practice in a specific section of biomedical journals [27, 28]. Another one is to replace the word “preprint”, which can be misleading, by ’Unrefereed manuscript’, "Manuscript awaiting peer review" or "Non-reviewed manuscript" [29].

## Conclusions

More than 40% of preprints cited in the major medical journals were in fact already published at the time of publication of the citing article. Therefore, authors should check the accuracy of the citation and the quotations of preprints just before submitting the manuscript, and once again when signing the proofs. They should not rely only on the information displayed on MedRxiv website to identify subsequent publication. To overcome medRxiv weaknesses, authors should manually search bibliographic databases to determine if a preprint has been subsequently peer-reviewed and published, keeping in mind the nearly half of the preprints have a different title than their published version. Publishers should also check the accuracy of the citations of preprints before online publication of citing articles.

The debate on the interest of preprints has largely increased during the COVID preprints, balancing between the interests of early dissemination of knowledge and the risk of postulated claims without evidence. However, considering the significant number of changes in the content or even the conclusions between the preprints and their published version, quotations of preprints should be considered with caution, even in the articles published in the major medical journals.

## Supporting information

S1 File. https://doi.org/10.5281/zenodo.5985960. (XLSX)

## Acknowledgments

The authors thank Mrs Anna Gehanno for her help in editing the manuscript.

## Author Contributions

**Conceptualization:** Jean-Francois Gehanno, Laetitia Rollin.

**Data curation:** Jean-Francois Gehanno, Laetitia Rollin.

**Formal analysis:** Jean-Francois Gehanno, Julien Grosjean, Stefan J. Darmoni, Laetitia Rollin.

**Supervision:** Laetitia Rollin.

**Validation:** Jean-Francois Gehanno.

**Writing – original draft:** Jean-Francois Gehanno, Laetitia Rollin.
**Writing – review & editing:** Jean-François Gehanno, Julien Grosjean, Stefan J. Darmoni, Laëtitia Rollin.

References

1. Fraser N, Brierley L, Dey G, Polka JK, Pálfy M, Nanni F, et al. The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape. PLOS Biology. 2021; 19: e3000959. https://doi.org/10.1371/journal.pbio.3000959 PMID: 33798194

2. Ho MH-C, Liu JS. The swift knowledge development path of COVID-19 research: the first 150 days. Scientometrics. 2021; 1–9. https://doi.org/10.1007/s11192-020-03835-5 PMID: 33551516

3. Smyth AR, Rawlinson C, Jenkins G. Preprint servers: a 'rush to publish' or 'just in time delivery' for science? Thorax. 2020; 75: 532–533. https://doi.org/10.1136/thoraxjnl-2020-214937 PMID: 32312737

4. Ålgå A, Eriksson O, Nordberg M. The development of preprints during the COVID-19 pandemic. J Intern Med. 2021; 290: 480–483. https://doi.org/10.1111/joim.13240 PMID: 33560546

5. Krumholz HM, Bloom T, Sever R, Rawlinson C, Inglis JR, Ross JS. Submissions and Downloads of Preprints in the First Year of medRxiv. JAMA. 2020; 324: 1903–1905. https://doi.org/10.1001/jama.2020.17529 PMID: 33170232

6. Oikonomidi T, Boutron I, Pierre O, Cabanac G, Ravaud P. Changes in evidence for studies assessing interventions for COVID-19 reported in preprints: meta-research study. BMC Med. 2020; 18. https://doi.org/10.1186/s12916-020-01880-8

7. Añazco D, Nicolalde B, Espinosa I, Camacho J, Mushtaq M, Gimenez J, et al. Publication rate and citation counts for preprints released during the COVID-19 pandemic: the good, the bad and the ugly. PeerJ. 2021; 9: e10927. https://doi.org/10.7717/peerj.10927 PMID: 33717688

8. Bero L, Lawrence R, Leslie L, Chiu K, McDonald S, Page MJ, et al. Cross-sectional study of preprints and final journal publications from COVID-19 studies: discrepancies in results reporting and spin in interpretation. BMJ Open. 2021; 11: e051821. https://doi.org/10.1136/bmjopen-2021-051821 PMID: 34272226

9. Cabanac G, Oikonomidi T, Boutron I. Day-to-day discovery of preprint-publication links. Scientometrics. 2021; 126: 5285–5304. https://doi.org/10.1007/s11192-021-03900-7 PMID: 33897069

10. Petrilli CM, Jones SA, Yang J, Rajagopalan H, O'Donnell L, Chernyak Y, et al. Factors associated with hospitalization and critical illness among 4,103 patients with Covid-19 disease in New York City. medRxiv. 2020; 2020.04.08.20057794. https://doi.org/10.1101/2020.04.08.20057794

11. Petrilli CM, Jones SA, Yang J, Rajagopalan H, O'Donnell L, Chernyak Y, et al. Factors associated with hospital admission and critical illness among 5279 people with coronavirus disease 2019 in New York City: prospective cohort study. BMJ. 2020; 369: m1966. https://doi.org/10.1136/bmj.m1966 PMID: 32444366

12. Fassin Y. Research on Covid-19: a disruptive phenomenon for bibliometrics. Scientometrics. 2021; 1–15. https://doi.org/10.1007/s11192-021-03989-w PMID: 33994601

13. Khatter A, Naughton M, Dambha-Miller H, Redmond P. Is rapid scientific publication also high quality? Bibliometric analysis of highly disseminated COVID-19 research papers. Learned Publishing. 2021; 34: 568–577. https://doi.org/10.1002/leap.1403 PMID: 34226800

14. Kirkham JJ, Penfold NC, Murphy F, Boutron I, Ioannidis JP, Polka J, et al. Systematic examination of preprint platforms for use in the medical and biomedical sciences setting. BMJ Open. 2020; 10: e041849. https://doi.org/10.1136/bmjopen-2020-041849 PMID: 33376175

15. Clyne B, Walsh KA, O'Murchu E, Sharp MK, Comber L, Brien KKO, et al. Using preprints in evidence synthesis: Commentary on experience during the COVID-19 pandemic. Journal of Clinical Epidemiology. 2021; 138: 203–210. https://doi.org/10.1016/j.jclinepi.2021.05.010 PMID: 34022394

16. Maslove DM. Medical Preprints—A Debate Worth Having. JAMA. 2018; 319: 443. https://doi.org/10.1001/jama.2017.17566 PMID: 29192310

17. Ide K, Koshiba H, Hawke P, Fujita M. Guidelines Are Urgently Needed for the Use of Preprints as a Source of Information. J Epidemiol. 2021; 31: 97–99. https://doi.org/10.2188/jea.JE20200506 PMID: 33162425

18. Vlasschaert C, Topf JM, Hiremath S. Proliferation of Papers and Preprints During the Coronavirus Disease 2019 Pandemic: Progress or Problems With Peer Review? Advances in Chronic Kidney Disease. 2020; 27: 418–426. https://doi.org/10.1053/j.ackd.2020.08.003 PMID: 33308508

19. Gianola S, Jesus TS, Bargeri S, Castellini G. Characteristics of academic publications, preprints, and registered clinical trials on the COVID-19 pandemic. PLoS One. 2020; 15. https://doi.org/10.1371/journal.pone.0240123 PMID: 33022014
**Writing – review & editing:** Jean-François Gehanno, Julien Grosjean, Stefan J. Darmoni, Laëtitia Rollin.

References

1. Fraser N, Brierley L, Dey G, Polka JK, Pálfy M, Nanni F, et al. The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape. PLOS Biology. 2021; 19: e3000959. https://doi.org/10.1371/journal.pbio.3000959 PMID: 33798194

2. Ho MH-C, Liu JS. The swift knowledge development path of COVID-19 research: the first 150 days. Scientometrics. 2021; 1–9. https://doi.org/10.1007/s11192-020-03835-5 PMID: 33551516

3. Smyth AR, Rawlinson C, Jenkins G. Preprint servers: a 'rush to publish' or 'just in time delivery' for science? Thorax. 2020; 75: 532–533. https://doi.org/10.1136/thoraxjnl-2020-214937 PMID: 32312737

4. Ålgå A, Eriksson O, Nordberg M. The development of preprints during the COVID-19 pandemic. J Intern Med. 2021; 290: 480–483. https://doi.org/10.1111/joim.13240 PMID: 33560546

5. Krumholz HM, Bloom T, Sever R, Rawlinson C, Inglis JR, Ross JS. Submissions and Downloads of Preprints in the First Year of medRxiv. JAMA. 2020; 324: 1903–1905. https://doi.org/10.1001/jama.2020.17529 PMID: 33170232

6. Oikonomidi T, Boutron I, Pierre O, Cabanac G, Ravaud P. Changes in evidence for studies assessing interventions for COVID-19 reported in preprints: meta-research study. BMC Med. 2020; 18. https://doi.org/10.1186/s12916-020-01880-8

7. Añazco D, Nicolalde B, Espinosa I, Camacho J, Mushtaq M, Gimenez J, et al. Publication rate and citation counts for preprints released during the COVID-19 pandemic: the good, the bad and the ugly. PeerJ. 2021; 9: e10927. https://doi.org/10.7717/peerj.10927 PMID: 33717688

8. Bero L, Lawrence R, Leslie L, Chiu K, McDonald S, Page MJ, et al. Cross-sectional study of preprints and final journal publications from COVID-19 studies: discrepancies in results reporting and spin in interpretation. BMJ Open. 2021; 11: e051821. https://doi.org/10.1136/bmjopen-2021-051821 PMID: 34272226

9. Cabanac G, Oikonomidi T, Boutron I. Day-to-day discovery of preprint-publication links. Scientometrics. 2021; 126: 5285–5304. https://doi.org/10.1007/s11192-021-03900-7 PMID: 33897069

10. Petrilli CM, Jones SA, Yang J, Rajagopalan H, O'Donnell L, Chernyak Y, et al. Factors associated with hospitalization and critical illness among 4,103 patients with Covid-19 disease in New York City. medRxiv. 2020; 2020.04.08.20057794. https://doi.org/10.1101/2020.04.08.20057794

11. Petrilli CM, Jones SA, Yang J, Rajagopalan H, O'Donnell L, Chernyak Y, et al. Factors associated with hospital admission and critical illness among 5279 people with coronavirus disease 2019 in New York City: prospective cohort study. BMJ. 2020; 369: m1966. https://doi.org/10.1136/bmj.m1966 PMID: 32444366

12. Fassin Y. Research on Covid-19: a disruptive phenomenon for bibliometrics. Scientometrics. 2021; 1–15. https://doi.org/10.1007/s11192-021-03989-w PMID: 33994601

13. Khatter A, Naughton M, Dambha-Miller H, Redmond P. Is rapid scientific publication also high quality? Bibliometric analysis of highly disseminated COVID-19 research papers. Learned Publishing. 2021; 34: 568–577. https://doi.org/10.1002/leap.1403 PMID: 34226800

14. Kirkham JJ, Penfold NC, Murphy F, Boutron I, Ioannidis JP, Polka J, et al. Systematic examination of preprint platforms for use in the medical and biomedical sciences setting. BMJ Open. 2020; 10: e041849. https://doi.org/10.1136/bmjopen-2020-041849 PMID: 33376175

15. Clyne B, Walsh KA, O'Murchu E, Sharp MK, Comber L, Brien KKO, et al. Using preprints in evidence synthesis: Commentary on experience during the COVID-19 pandemic. Journal of Clinical Epidemiology. 2021; 138: 203–210. https://doi.org/10.1016/j.jclinepi.2021.05.010 PMID: 34022394

16. Maslove DM. Medical Preprints—A Debate Worth Having. JAMA. 2018; 319: 443. https://doi.org/10.1001/jama.2017.17566 PMID: 29192310

17. Ide K, Koshiba H, Hawke P, Fujita M. Guidelines Are Urgently Needed for the Use of Preprints as a Source of Information. J Epidemiol. 2021; 31: 97–99. https://doi.org/10.2188/jea.JE20200506 PMID: 33162425

18. Vlasschaert C, Topf JM, Hiremath S. Proliferation of Papers and Preprints During the Coronavirus Disease 2019 Pandemic: Progress or Problems With Peer Review? Advances in Chronic Kidney Disease. 2020; 27: 418–426. https://doi.org/10.1053/j.ackd.2020.08.003 PMID: 33308508

19. Gianola S, Jesus TS, Bargeri S, Castellini G. Characteristics of academic publications, preprints, and registered clinical trials on the COVID-19 pandemic. PLoS One. 2020; 15. https://doi.org/10.1371/journal.pone.0240123 PMID: 33022014
20. Shi X, Ross JS, Amancharla N, Niforatos JD, Krumholz HM, Wallach JD. Assessment of Concordance and Discordance Among Clinical Studies Posted as Preprints and Subsequently Published in High-Impact Journals. JAMA Netw Open. 2021; 4: e212110. https://doi.org/10.1001/jamanetworkopen.2021.2110 PMID: 33734411

21. McGuinness LA, Sheppard AL. A descriptive analysis of the data availability statements accompanying medRxiv preprints and a comparison with their published counterparts. PLoS One. 2021; 16: e0250887. https://doi.org/10.1371/journal.pone.0250887 PMID: 33983972

22. RECOVERY Collaborative Group, Horby P, Mafham M, Linsell L, Bell JL, Staplin N, et al. Effect of Hydroxychloroquine in Hospitalized Patients with Covid-19. N Engl J Med. 2020; 383: 2030–2040. https://doi.org/10.1056/NEJMoa2022926 PMID: 33031652

23. Malički M, Costello J, Alperin JP, Maggio LA. Analysis of single comments left for bioRxiv preprints till September 2019. Biochem Med (Zagreb). 2021; 31: 020201. https://doi.org/10.11613/BM.2021.020201 PMID: 33927548

24. Jefferson T, Rudin M, Brodney Folse S, Davidoff F. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev. 2007; MR000016. https://doi.org/10.1002/14651858.MR000016.pub3 PMID: 17443635

25. AMWA-EMWA-ISMPP joint position statement on medical publications, preprints, and peer review. Current Medical Research and Opinion. 2021; 37: 861–866. https://doi.org/10.1080/03007995.2021.1900365 PMID: 33789058

26. Hopkins C. Preprints—expediting access or compromising quality? International Forum of Allergy & Rhinology. 2021; 11: 835–836. https://doi.org/10.1002/alr.22732 PMID: 33089660

27. Zaidi M, Harper DM, Akhmanova A, Weigel D, Behrens TE, Eisen MB. Rigorous review and editorial oversight of clinical preprints. Elife. 2021; 10: e67528. https://doi.org/10.7554/eLife.67528 PMID: 34130793

28. Watching preprints evolve. Nat Rev Immunol. 2021; 21: 65–66. https://doi.org/10.1038/s41577-020-00489-5 PMID: 33432126

29. Ravinetto R, Caillet C, Zaman MH, Singh JA, Guerin PJ, Ahmad A, et al. Preprints in times of COVID19: the time is ripe for agreeing on terminology and good practices. BMC Med Ethics. 2021; 22: 106. https://doi.org/10.1186/s12910-021-00667-7 PMID: 34320970
20. Shi X, Ross JS, Amancharla N, Niforatos JD, Krumholz HM, Wallach JD. Assessment of Concordance and Discordance Among Clinical Studies Posted as Preprints and Subsequently Published in High-Impact Journals. JAMA Netw Open. 2021; 4: e212110. https://doi.org/10.1001/jamanetworkopen.2021.2110 PMID: 33734411

21. McGuinness LA, Sheppard AL. A descriptive analysis of the data availability statements accompanying medRxiv preprints and a comparison with their published counterparts. PLoS One. 2021; 16: e0250887. https://doi.org/10.1371/journal.pone.0250887 PMID: 33983972

22. RECOVERY Collaborative Group, Horby P, Mafham M, Linsell L, Bell JL, Staplin N, et al. Effect of Hydroxychloroquine in Hospitalized Patients with Covid-19. N Engl J Med. 2020; 383: 2030–2040. https://doi.org/10.1056/NEJMoa2022926 PMID: 33031652

23. Malički M, Costello J, Alperin JP, Maggio LA. Analysis of single comments left for bioRxiv preprints till September 2019. Biochem Med (Zagreb). 2021; 31: 020201. https://doi.org/10.11613/BM.2021.020201 PMID: 33927548

24. Jefferson T, Rudin M, Brodney Folse S, Davidoff F. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev. 2007; MR000016. https://doi.org/10.1002/14651858.MR000016.pub3 PMID: 17443635

25. AMWA-EMWA-ISMPP joint position statement on medical publications, preprints, and peer review. Current Medical Research and Opinion. 2021; 37: 861–866. https://doi.org/10.1080/03007995.2021.1900365 PMID: 33789058

26. Hopkins C. Preprints—expediting access or compromising quality? International Forum of Allergy & Rhinology. 2021; 11: 835–836. https://doi.org/10.1002/alr.22732 PMID: 33089660

27. Zaidi M, Harper DM, Akhmanova A, Weigel D, Behrens TE, Eisen MB. Rigorous review and editorial oversight of clinical preprints. Elife. 2021; 10: e67528. https://doi.org/10.7554/eLife.67528 PMID: 34130793

28. Watching preprints evolve. Nat Rev Immunol. 2021; 21: 65–66. https://doi.org/10.1038/s41577-020-00489-5 PMID: 33432126

29. Ravinetto R, Caillet C, Zaman MH, Singh JA, Guerin PJ, Ahmad A, et al. Preprints in times of COVID19: the time is ripe for agreeing on terminology and good practices. BMC Med Ethics. 2021; 22: 106. https://doi.org/10.1186/s12910-021-00667-7 PMID: 34320970