# Bad practices in open peer review: lessons learned from mining MDPI open reports

Gilles Hubert

► To cite this version:

Gilles Hubert. Bad practices in open peer review: lessons learned from mining MDPI open reports. IX Annual conference of the Italian association for the promotion of open science, Associazione italiana per la promozione della scienza aperta (AISA), Nov 2024, Pisa (Italy), Italy. 10.5281/zenodo.14098794. hal-04786406

HAL Id: hal-04786406

https://ut3-toulouseinp.hal.science/hal-04786406v1

Submitted on 15 Nov 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Bad practices in open peer review:
lessons learned from mining MDPI open reports

Gilles Hubert

<figure><img src="image_2.png" /></figure>

UNIVERSITÉ
TOULOUSE III
PAUL SABATIER

<figure><img src="image_3.png" /></figure>

Institut de Recherche
en Informatique de Toulouse
CNRS - INP - UT3 - UT1 - UT2J
My collaborators (and friends)

<figure><img src="image_4.png" /><figcaption>Guillaume Cabanac</figcaption></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

European Research Council
Established by the European Commission

The NanoBubbles project has received Synergy grant funding from the European Research Council (ERC), within the European Union's Horizon 2020 programme, grant agreement no. 951393.

<figure><img src="image_7.png" /><figcaption>Cyril Labbé</figcaption></figure>

<figure><img src="image_8.png" /></figure>

UNIVERSITÉ
TOULOUSE III
PAUL SABATIER

<figure><img src="image_9.png" /></figure>

Institut de Recherche
en Informatique de Toulouse
CNRS - INP - UT3 - UT1 - UT2J

<figure><img src="image_10.png" /></figure>

institut
universitaire
de France

UGA
Université
Grenoble Alpes

LIG
LABORATOIRE D'INFORMATIQUE DE GRENoble
<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

THE PEER REVIEW PROCESS
AND IT'S IMPORTANCE

<figure><img src="image_13.png" /></figure>

www.imanagerpublications.com

* Key process in scientific publication

* Mostly unopened

* Confidence in peer reviewers and editors

* ...

* But faillible

* Bad practices in scientific papers (fake papers, fraudulent papers, retracted papers...)
<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>

Research paper mill

Article Talk

From Wikipedia, the free encyclopedia

"Academic paper mill" redirects here. Not to be confused with Wikipedia or its related projects.

In research, a paper mill is a business that publishes authorship.[1][2]

In some papers, thousands of reviews are received. It is a writing problem.

Plagiarism, paper mills and profit: These scientists are fighting the epidemic of fraudulent science research

Bad incentives in scientific publishing have led to bad science. Some researchers are working to sniff out the fraud and restore integrity to their field.

BY SABAHAT RAHMAN

JANUARY 31, 2024

<figure><img src="image_16.png" /></figure>

nature > news > article

NEWS | 19 January 2024

Science's fake-paper problem: high-profile effort will tackle paper mills

EXCLUSIVE: Poor-quality st
SIGN UP
businesses th

Feature

THE BATTLE
AGAINST
PAPER MILLS

Some journals have admitted to a problem with fake research papers. Now editors are trying to combat it. By Holly Else and Richard Van Noorden

<figure><img src="image_17.png" /></figure>

What is going on?

The paper mill business has repeatedly warned that some scientists buy papers from third-party firms. In a recent letter to the journal, RSC Advances, none of the papers had authors, or institutions cited, looked along similarly, says the paper mill's executive editor at the journal. "It was determined by us to get to the bottom of what was going on."

© 2024 Springer Nature Limited. All rights reserved.

A year later, in January 2021, Fisher wrote 45 papers from the journal, and editors at two other Royal Society of Chemistry (RSC) journals retracted one each over similar suspicions; 15 are still under investigation. Fisher had found what seemed to be the products of paper mills, companies that churn out fake scientific manuscripts in order. All the papers came from authors at Chinese hospitals. The journal's publisher, the RSC in London, announced in a statement that he had been the victim of what is believed to be "the systematic production of falsified research".

Paper mill detectors

A paper mill is a large, industrial building that produces large volumes of high-quality, dedicated images in papers, and one of the duties it performs their concerns about paper mills online.

But there are thousands more of these papers in the literature. The RSC's reservoir is significant for its operations, the says. "It's pretty embarrassing that so many papers have been falsified," adds Fisher. "Kadets to them to admit that they have been falsified."

Paper-mill detectors

In January 2021, Bk, and other image detectors who work under pseudonyms, like Stuart Clyde, Morley and Tiger 088 – posted, on a

SSB | Nature | 19 Mar 2021

NEWS

Review mills identified as a new form of peer-review fraud

BY

<figure><img src="image_18.png" /></figure>

JULIA ROBINSON | 5 FEBRUARY 2024

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>

A 'review mill' that appears to have produced at least 85 similar peer-review reports featuring coercive citation could be an indicator of a new organised form of academic malpractice. The review reports were discovered alongside articles published across several journals run by the open-access publisher, MDPI, and were brought to light by a volunteer-led investigation posted online by Predatory Reports – an organisation that aims to highlight unethical publishing practices.

The work was carried out by María de los Ángeles Oviedo García, a professor of business
# Review Mill at MDPI, by María de los Ángeles OVIEDO-GARCÍA, January 12, 2024

https://predatoryjournals.org/news/f/review-mill-at-mdpi

[...] a set of 85 very similar review reports in 23 journals published by MDPI [...] a standard text was copied and pasted to every manuscript regardless its content, following two patterns (type A and type B).

The manuscript "Analysis of Alkylphenol Ethoxylates in Tea by SPE Coupled to UPLC-MS/MS" is a very interesting work. This work describes the Alkylphenol ethoxylates (APEOs) is a kind of widely used non-ionic surfactant, as an adjuvant in pesticide formulation, which is considered to cause endocrine disrupting effect. In the current study, detection method for APEOs residue in tea was established, on the basis of SPE (sol-id-phase extraction) for simultaneous analysis of nonylphenol ethoxylates (NPEOs) and octylphenol ethoxylates (OPEOs) by UPLC-MS/MS. In the spiked concentrations from 0.024 µg/kg to 125.38 µg/kg for 36 monomers of APEOs (nEO = 3-20), the recoveries of APEOs ranged from 70.3-110.7% with RSD ≤16.9%, except OPEO20 (61.8%) and NPEO20 (62.9%). The results are consistent with the data and figures presented in the manuscript. While I believe this topic is of great interest to our readers, I think it needs major revision before it is ready for publication. So, I recommend this manuscript for publication with major revisions.

1. In this manuscript, the authors did not explain the importance of the Alkylphenol Ethoxylates in the introduction part. The authors should explain the importance of Alkylphenol Ethoxylates.

2) Title: The title of the manuscript is not impressive. It should be modified or rewritten.

3) Correct the following statement: "The detection rate and the range of monomers of NPEOs was generally higher than that of OPEOs. The current study would provide a theoretical basis to guide the rational use of APEOs as adjuvant in commercial pesticide production".

4) Keywords: there so many keywords and reduce them up to 5. So, modify the keywords.

5) Introduction part is not impressive. The references cited are very old. So, improve it with some latest literature like 10.3390/biom12010083, 10.3390/ph15101164

6) The authors should explain the following statement with recent references: "The total concentration of OPEOs was between 0.057 µg/kg and 12.9 µg/kg. The monomers with the highest detection concentration were OPEO9, OPEO10, and OPEO11".

7) Add space between magnitude and unit. For example, in synthesis "21.96g" should be 21.96 g. Make the corrections throughout the manuscript regarding values and units.

8) The author should provide reason about this statement, 50 tea samples in total were randomly collected from the market, including 24 white tea, 12 black tea, 5 dark tea, 4 green tea, 3 scented tea, and 3 oolong tea.

9) Comparison of the present results with other similar findings in the literature should be discussed in more detail. This is necessary in order to place this work together with other work in the field and to give more credibility to the present results.

10) Conclusion part is very long. Make it brief and improve by adding the results of your studies.

11) There are many grammatic mistakes. Improve the English grammar of the manuscript.

The manuscript "Green dynamic kinetic resolution-stereoselective acylation of secondary alcohols by enzyme-assisted ruthenium complexes" is very interesting work. In this study, The resulting catalytic system of the ruthenium racemization catalysts and enzymatic acylation led to chiral esters obtained by dynamic kinetic resolution. The immobilized catalytic system in the ionic liquid expresses the same activity and selectivity as the homogeneous system while allowing for convenient separation of the desired products. Additionally, the efficient reuse of the catalytic system has been demon strated fulfilling the paradigm of green processes. I believe this topic is of great interest to our reader, I think it needs minor revision before it is ready for publishing. So, I strongly recommend this manuscript for publication in the Journal of Catalysts with major revisions.

1. In this manuscript, the authors did not explain the importance of ruthenium complexes in the introduction part. The authors should explain the importance of ruthenium complexes.

2. The author should provide reason about this statement with recent references. The immobilized catalytic system in the ionic liquid expresses the same activity and selectivity as the homogeneous system while allowing for convenient separation of the desired products".

3. Introduction part is not impressive and systematic. Cite the following articles in the introduction part. (i) 10.3390/molecules27196580 (ii) 10.3389/fchem.2022.995820

4. The authors should justify the following statement: "The possibility of creating an efficient heterogeneous catalyst in which the ruthenium complex is non-covalently attached to support was also sought due to the simplicity of the system and the least influence of the catalytic properties".

5. The authors should explain regarding the recent literature why this study affects the activity of the ruthenium catalyst.

6. The authors should explain the statement with reference to the literature. This is because the catalyst is confined in a small volume of the IL as opposed to being dispersed throughout the entire volume in the absence of the addition of an IL.

7. Comparison of the present results with other similar findings in the literature should be discussed in more detail. This is necessary in order to place this work together with other work in the field and to give more credibility to the present results.

8. The conclusion part is very weak. Improve by adding the results of your studies.

9. The authors should pay more attention to the English grammar, and the abbreviation of journal names in Ref.
# Is it possible to automatically detect bad practices?

## Analysed dataset

* MDPI Open Peer Review Corpus 2 - Version 2.0

* Scraped by Miłkowski et al. (2023, https://doi.org/10.18150/shkp7b)

* 170 GB

* 135,437 articles and their associated reports (between 2011 and 2022)

## Chosen approach

* Focus on *Round 1* reports in plain text (i.e., excluding those uploaded as attached files)
  → 320,380 reports (/ 353,131)

* Statistics on report length

* «Strong» inter-textual similarity between reports

* Common word sequences (10+)

* Identified references (DOIs) suggested by reviewers with regular expressions
Result 1: Report length

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

Some very very long reports

* For example, Report 3 for paper pr10051002 is the same text pasted 50 times.

Quite a lot of short reports

* 223 ‘nano’ reports consisted of 1 word only (‘accept’, ‘none’, ‘Nil’ or ‘N.A.’)

* ‘Micro’ reports of less than 20 words account for 2.7% of the dataset.
‘Nano’ reports

Paper su14095543

Reviewer 2 Report

Author Response

Thank you very much for your support and affirmation of this article

Paper life12101650

Reviewer 1 Report

Nil

Author Response

Thank you to our affirmation

Paper agriculture12020303

Reviewer 1 Report

no

Author Response

Editor

MDPI-Agriculture

15th January 2022

Dear Editor,

Subject: Submission of revised paper ‘Climate Resilience and Environmental Sus
Dimensions of Water Security Modelling. Thank you for your email dated 8th Janu
above-mentioned manuscript. We appreciate the valuable comments and suggestio
anonymous reviewers
# Technical issues: bogus scraping and editorial management

Paper aerospace9100612

In the dataset

Reviewer 1:

Summary:

On the website

**Round 1**

Reviewer 1 Report

Summary:

The paper proposed a new complexity metric, which is an adaptation of Sinh and changes are supposed to make the metric suitable for the use with MBS in three case studies and the results are discussed.

Feedback:

Positive: + the paper addresses a timely topic and uses the right/suitable approach provides a comprehensive overview and the case studies add to the substan addressed: - the differences between the new metric and the one it is based advantageous and necessary - a general lack of comparison to recent literati distinction regarding the assumptions made is not clear regarding the sours ( lack of explanation of disadvantages/boundaries - format errors, missing cap pg. 4,7, ...), overlapping text (e.g. pg. 6) - very very wordy case study explains the results; the setups do not need to be explained in such an extensive way

<figure><img src="image_23.png" /></figure>

Paper agriculture11080744

In the dataset

Reviewer 2:

none

On the website

**Round 1**

Reviewer 1 Report

Good work addressing comments

Reviewer 2 Report

none

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

<figure><img src="image_24.png" /></figure>

**Round 1**

Reviewer 1 Report

The work is prepared quite diligently. I have no major remarks. However, I have a few observations:
'Micro' reports

Paper admsci12030097

Reviewer 1 Report

Update the literature review/references.

Author Response

The author had updated literature related to the
[30]

Paper aerospace8070179

Reviewer 1 Report

This paper is good and acceptable quality.

Author Response

Thank you for your support.

Paper adolescents1010001

Reviewer 1 Report

Authors should be congratulated.

Author Response

Many thanks for your kind comments
# Result 2: Inter-textual similarities between reports

## Regarding Report-report similarities

* 48,626 pairs of reports with inter-textual similarity >= 90%
  0.8 % of reports (2493 reports)

* 74,170 pairs of reports with inter-textual similarity >= 75%
  1.6 % of reports (5191 reports)

Very short reports

<figure><img src="image_25.png" /></figure>

*Only first 300 nodes are displayed
# Result 2: Inter-textual similarities between reports

Regarding report-report similarities

* reports with more than 3000 characters and inter-textual similarity >= 75% with at least one other report

<figure><img src="image_26.png" /></figure>
Report-report similarities show identical reports for the same paper

ijms232112760

<figure><img src="image_27.png" /></figure>

Reviewer 2 Report

In this article, Washburn et al. assess the activation of the complement system in response to the allograft of Sertoli cells. They observed that the Sertoli cells showed substantial protection against humoral immunity. The microarray experiment with the mouse Sertoli cells showed enhanced expression of complement inhibitory proteins (CIPs). They conclude by suggesting that since Sertoli cells exhibit protection from complement-mediated tissue rejection, it could be a novel strategy in diabetes for enhancing the success of islet grafts.

Overall, the study seems to highlight the protective mechanism exhibited by Sertoli cells' graft in treating diabetes. The study is in line with a previous study (Fallarino et al., 2009) and one from the same group (Kaur et al., 2018). Although the findings are interesting and certainly pave the path for exciting avenues, at the current stage, several factors limit the enthusiasm in this study.

...

Reviewer 3 Report

In this article, Washburn et al. assess the activation of the complement system in response to the allograft of Sertoli cells. They observed that the Sertoli cells showed substantial protection against humoral immunity. The microarray experiment with the mouse Sertoli cells showed enhanced expression of complement inhibitory proteins (CIPs). They conclude by suggesting that since Sertoli cells exhibit protection from complement-mediated tissue rejection, it could be a novel strategy in diabetes for enhancing the success of islet grafts.

Overall, the study seems to highlight the protective mechanism exhibited by Sertoli cells' graft in treating diabetes. The study is in line with a previous study (Fallarino et al., 2009) and one from the same group (Kaur et al., 2018). Although the findings are interesting and certainly pave the path for exciting avenues, at the current stage, several factors limit the enthusiasm in this study.
Report-report similarities show groups of reports almost identical

ma13235361

Reviewer 5 Report

1) The originality and the scientific value of the subject are good. Indeed, an important problem having direct applications is treated.

2) The Abstract is concrete as it gives the summary of this research work in a concise manner. In addition, it is sufficiently supported by the results obtained during research.

3) The Introduction Section in its current form is not adequate. In this context, I recommend the authors to further analyze and discuss the results of Refs. [1-3], [4-10], [11,12] and [13-15]. Besides, the differences/advantages of the present investigation compared to other literature works should be written out at the end of this Section in a much more detailed and comprehensive manner.

4) The materials, their applications, applied methods and especially the use of the investigated material are explained in detail. The composition, the origin of the material used, dimensions of specimens etc are all mentioned.

5) Presentation of the experimental work is very thorough. Process and prerequisites of sample preparation are clearly mentioned. However, the authors are kindly recommended to provide some further technical details about the laboratory equipment that they used to carry out their experiments.

6) The performance and clarity of results and data are good. Yet,

ma13245739

Reviewer 2 Report

1) The originality and the scientific value of the subject are good. Indeed, an important problem having direct applications is treated.

2) The Abstract is concrete as it gives the summary of this research work in a concise manner. In addition, it is sufficiently supported by the results obtained during research.

Nonetheless, given that the overall text contains many abbreviations, it is the reviewer's opinion that the authors should definitely add a nomenclature at the beginning of the manuscript.

3) The Introduction Section in its current form is not adequate. In this context, I recommend the authors to further analyze and discuss the results of Refs. [1-3], [4-10], [11,12] and [13-15].

Besides, the differences/advantages of the present investigation compared to other literature works should be written out at the end of this Section in a much more detailed and comprehensive manner.

4) The materials, their applications, applied methods and especially the use of the investigated material are explained in detail. The composition, the origin of the material used, dimensions of specimens etc are all mentioned.

5) Presentation of the experimental work is very thorough. Process and prerequisites of sample preparation are clearly mentioned.

ma13163503

Reviewer 3 Report

1) The originality and the scientific value of the subject are very good. An important problem having direct applications is treated.

2) The Abstract in its current form is not sufficient. In particular, it should be supported in a more effective manner by the results obtained during research, because the first part which is read by Journal's audience is Abstract and thus it should reflect the novelty and perform the main results.

3) The Introduction Section in its current form is not adequate. In this context, I recommend the authors to further analyze and discuss the results of Refs. [5-8], [9-11], [15-19], [20-22] and [23-27]. In addition, the differences/advantages of the present investigation compared to other literature works should be written out at the end of this Section in a much more thorough and comprehensive manner.

4) The materials, their applications, applied methods and especially the use of the investigated material are explained in detail. The composition, the origin of the material used, dimensions of specimens etc are all mentioned

5) Presentation of the experimental work is very thorough. Process and prerequisites of sample preparation are clearly mentioned. However, the authors are kindly recommended to provide some further technical details about the laboratory equipment that they used to carry out their experiments.
Report-report similarities show groups of reports almost identical

ijms23010134

Reviewer 2 Report

I read with great interest the manuscript, which falls within the aim of this Journal. In my honest opinion, the topic is interesting enough to attract the readers' attention. Nevertheless, authors should clarify some points and improve the discussion, as suggested below.

Authors should consider the following recommendations:

* Manuscript should be further revised in order to correct some typos and improve style.

* Accumulating evidence suggests that one of the most important mechanisms of PCOS pathogenesis is the insulin-resistance. For this reason, the use of insulin-sensitizers, such as inositol isoforms, gained increasing attention due to their safety profile and effectiveness. Authors may better discuss this point, taking to account these recent articles: PMID: 26927948; PMID: 27579037.

biomedicines10020456

Reviewer 2 Report

I read with great interest the manuscript, which falls within the aim of this Journal. In my honest opinion, the topic is interesting enough to attract the readers' attention. Nevertheless, authors should clarify some points and improve the discussion, as suggested below.

Authors should consider the following recommendations:

* Manuscript should be further revised in order to correct some typos and improve style.

* Accumulating evidence suggests that one of the most important mechanisms of PCOS pathogenesis is the insulin-resistance. For this reason, the use of insulin-sensitizers, such as inositol isoforms, gained increasing attention due to their safety profile and effectiveness. Authors may better discuss this point, taking to account these recent articles: PMID: 26927948; PMID: 27579037.

ijerph182111274

Reviewer 3 Report

I read with great interest the manuscript, which falls within the aim of this Journal. In my honest opinion, the topic is interesting enough to attract the readers' attention. Nevertheless, authors should clarify some points and improve the discussion, as suggested below.

Authors should consider the following recommendations:

* Manuscript should be further revised in order to correct some typos and improve style.

* Accumulating evidence suggests that one of the most important mechanisms of PCOS pathogenesis is the insulin-resistance. For this reason, the use of insulin-sensitizers, such as inositol isoforms, gained increasing attention due to their safety profile and effectiveness. Authors may better discuss this point, taking to account these recent articles: PMID: 26927948; PMID: 27579037.
# Result 2: Inter-textual similarities between reports

Regarding report-report similarities

* paper having 100% of reports similar to
  (> 90%) another report

<figure><img src="image_28.png" /></figure>
Result 3: Reports sharing long sequence of words (10+)

Regarding report sharing chunks of text

* 36,476 reports sharing at least one sequence of words (10+) with another report
  11.4 % of reports

<figure><img src="image_29.png" /></figure>

*Only first 300 nodes are displayed
Result 3: Reports sharing long sequence of words (10+)

<figure><img src="image_30.png" /></figure>
Conclusions & Future work

* Questionable practices exist in MDPI open reports... Even few compared to the processed volume...

* Study limitations mainly due to a not so clean and very incomplete dataset

* More research is needed to improve malpractice detection and assess its prevalence in (open) peer review reports

* Encourages open access to peer review reports and raises questions about all the review processes not made public

https://www.mdpi.com/about/announcements/1405

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>

<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

12 October 2018

**Open Peer Review for all MDPI Journals**

It is now a little over four years since MDPI first started to offer open peer review. Given the popularity of this feature in the 14 journals operating it until now, we have decided to extend options for open reports and open identities to all MDPI journals. This means that **authors have the option for the review reports and author responses to be published alongside their article, and reviewers have the option to add their name to the published review report.** For further background and information, see our **blog post**.

<figure><img src="image_35.png" /></figure>