# Evidence of Use of Template-Based Peer-Review Reports and Concern About Review Mills

Cyril Labbé, Gilles Hubert, Wendeline Swart, Guillaume Cabanac

► To cite this version:

Cyril Labbé, Gilles Hubert, Wendeline Swart, Guillaume Cabanac. Evidence of Use of Template-Based Peer-Review Reports and Concern About Review Mills. PRC'25: 10th International Congress on Peer Review and Scientific Publication, Sep 2025, Chicago, United States. . [hal-05273020](https://hal.science/hal-05273020v1)

HAL Id: hal-05273020

https://hal.science/hal-05273020v1

Submitted on 22 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

Cyril Labbé¹, Gilles Hubert², Wendeline Swart², Guillaume Cabanac²,³

1 Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG

2 IRIT – University of Toulouse

3 Institut Universitaire de France

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

European Research Council
Established by the European Commission

## Objective

Test methods to investigate review mills¹: various coordinated reviewer misconducts.

We studied 4 datasets of peer-review reports in search for evidence of such practices.

<table><thead><tr><th>Datasets</th><th>Years</th><th>Number of reports</th><th>Number of articles</th><th>Report mean length</th><th>Number of reports with CST</th><th>Number of similar reports (≥0.75)</th></tr></thead><tbody><tr><td>MDPI (11 journals)</td><td>2018–2025</td><td>122,831</td><td>47,593</td><td>251.5</td><td>12,430 (10.12%)</td><td>1,604 (1.31%)</td></tr><tr><td>PeerJ² (7 journals)</td><td>2015–2022</td><td>12,959</td><td>6,292</td><td>504.8</td><td>1,137 (8.77%)</td><td>62 (0.48%)</td></tr><tr><td>NLPEER³ (5 conferences and F1000Research)</td><td>2016, 2017, 2020, 2022</td><td>12,061</td><td>NA</td><td>356.3</td><td>951 (7.88%)</td><td>122 (1.01%)</td></tr><tr><td>BMJ</td><td>2021–2024</td><td>308</td><td>97</td><td>530.4</td><td>37 (12.01%)</td><td>2 (0.65%)</td></tr></tbody></table>

## Design

We investigated plain-text reports only (excluding attached files). Considering 'Round 1' only. We computed statistics on:

* Report length,

* Common Sequences of Terms (CST),

* Similarity measures between reports.

## Conclusions

* Clear evidence of template-based peer-review reports practices (limitation: copy-paste of instructions).

* The extent of these practices varies from one dataset to another. This might be due to reviewers' practices, discrepancies in dataset sizes, to differences in scientific fields or to journals, editors or publishers' habits.

* More openly available review reports datasets would help further characterise and understand this phenomenon.

<figure><img src="image_10.png" /></figure>

## References

1. Oviedo-García, M.Á. The review mills, not just (self-)plagiarism in review reports, but a step further. *Scientometrics*. 2024; 129:5805–5813. doi: 10.1007/s11192-024-05125-w

2. Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X. MOPRD: A multidisciplinary open peer review dataset. *Neural Computing and Applications*. 2023; 35:24191–24206. doi: 10.1007/s00521-023-08891-5

3. Dycke, N., Kuznetsov, I., & Gurevych, I. NLPeer: A Unified Resource for the Computational Study of Peer Review. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics*, 2024; 1:5049–5073. doi: 10.18653/v1/2023.acl-long.277