# VIRUS: Visualization of Irregular Research Under Scrutiny

Fabrice Frank*

Lonni Besançon†

Independent researcher, Essaouira, Morocco

Linköping University, Sweden

<figure><img src="image_1.png" /><figcaption>Figure 1: Overview of the interactive visualization. Panel ① contains filters; Panel ② and Panel ③ show the two different visualizations available; Panel ③ the information about a selected paper. To provide an informative teaser image, the underlying data has been modified.</figcaption></figure>

## ABSTRACT

We present VIRUS, a visualization system to keep records of questionable papers being investigated and their impact on several scholarly measures. This effort, based on an identified set of potentially problematic papers, is meant to help researchers explore what interactive visualization systems may be needed in the future. As the number of retracted papers continues to soar, we believe that such systems will be essential for sleuthing efforts. We thus describe a prototype that we aim to iteratively evaluate with academic sleuths for improvement. All materials (poster, code, follow-up) for this submission are available at https://osf.io/83v2a/.

**Index Terms:** Research Ethics, Scientometrics, Visualization

## 1 INTRODUCTION

The ratio of retracted papers to articles published is rising at an alarming rate and towers, nowadays, 0.2% [12]. A paper is retracted when it is found to be either fraudulent or containing too severe errors to be corrected. Most of these retractions are due to the efforts of scientific sleuths who find, track, and report questionable papers on Pubpeer [1] and in correspondence with editors. Questionable or problematic papers can present in a variety of forms, ranging from plagiarism to fake papers written by ChatGPT, and include image or data manipulations [5], paper mill products [7], or the use

of sneaked references to hide in the metadata of articles undue references at when registering an accepted article [4]. The retraction watch database is essential to track the status of questionable papers. However, while it is publicly available, a tabular dataset does not scale well to follow the status of multiple specific papers. In particular, when hundreds of papers are flagged for the same set of concerns, it is difficult to follow their individual status (still published, expression of concern, retracted) or to assess their actual impact on the scholarly world. This is the case, for instance, of papers flagged with sneaked references (a few thousand), the thousands of papers containing tortured phrases [7] or the hundreds of COVID-19 papers peer-reviewed in less than 24 hours [3].

In most cases [2], reactions from editors and publishers, are particularly slow, if existent. Retractions are still only considered for extremely problematic papers and they may still take time. If editors and publishers are transparent enough, a paper will receive an expression of concern while it is being investigated. We hope to be able to build a dashboard that could help visualize the status of hundreds of papers and highlight their impact on the scholarly world in order to facilitate the sleuthing work done by whistleblowers, and 2/ perhaps accelerate editorial decisions. We thus present VIRUS, an online tool that helps visualize metrics and status of a specific set of questionable papers. To illustrate the tool, we use a dataset of 456 papers from one institute flagged for potential ethical issues in a scientific publication [10]. We completed it by adding papers that received concerns on Pubpeer to reach a total of 690 papers. The issues are ranging from lack of approval to run a clinical trial to recruitment starting before approval was obtained and including the questionable duplication of a single ethical approval number in 249 heterogeneous studies. We believe this convenience sample can be representative of manuscripts being flagged on Pubpeer.

<footnote>*e-mail: fabrice.frank@infostrategie.com</footnote>

<footnote>†e-mail: lonni.besancon@liu.se</footnote>
## 2 RELATED WORK

The VIS community has a keen interest to collect, understand, and visualize its own publishing metadata [11], and quite a few systems have been developed to look at either our own data, or academic careers and outputs as a whole (e.g., [13]). Particularly relevant to our work are approaches that propose to interactively analyze and display information about published academic papers. For instance, Paper 101 [8] helps exploring academic outputs and obtain metadata about potential papers of interest. Overall, most of the research work conducted by the visualization community focuses on providing citation relationship and network-based analysis or recommendations (e.g., [8]). In contrast, here, we mostly aim at displaying relevant metadata about specific groups of papers that have been identified as questionable in order to provide academic sleuths with a visual tool to help them explore the evolving status of these papers and the relevant metrics to present to editors and publishers. While the interactive visualization we propose is not, per se, novel, it tackles a problem that the visualization community has not yet focused on.

## 3 VIRUS

Our prototype is an online interactive visualization developed with Javascript and D3.js [6]. Its landing page and its different panels are detailed in Figure 1. The page first provides some information about the goal of the project, the tool, and some contextualization on research integrity and questionable research practices and sleuthing work. The main visualization is visible in Panel 2 of Figure 1. By default, when the page is opened, it proposes to display a stacked barchart that contains, on the x-axis the number of papers, and on the y-axis, the different journals that published those papers. For each stack of bars, the individual bars encode, through color, the status of the paper: papers that are *still published* and have not publicly received an expression of concerns, papers that have received a published *expression of concern* without being retracted, papers that have been *retracted* (see Panel ②). If, after filtering, the number of papers is small enough for each paper to be visible, the system will replace the bar with a circle for each individual paper in each specific journal (see Panel ②). The circle's size is then representative of a variable of interest about this set of papers: number of citations (gathered here from OpenCitation and Crossref, but other dashboards can be used such as Dimensions or Google Scholar), the *Altmetric* score of that paper (which mentions the attention a paper obtained outside of academic literature, e.g., News mentions or tweets, see [9]), or the number of self-citations contained in this paper. In this case, hovering over a specific data point creates a highlight around it and provides additional information about the paper in Panel ③. This information currently includes: title, authors, publishing journal, number of citations, *Altmetric* score, DOI, current status, and links to possible expressions of concerns or retractions. In addition, the *Altmetric* donut of the paper is displayed on the right of these fields to obtain a more detailed breakdown of the score. Other measures of interest can be added so long as the data is acquired. Panel ① contains all of the interactive filtering components that we believe are needed to find a specific subset of papers. Filtering options currently include: filtering by IRB approval numbers, filtering to only display journals with a specific number of papers or only including papers from a specific author only. The interaction panel also provides the possibility to switch views from the stacked barchart (Panel ②) to the view of individual papers (Panel ②).

VIRUS is not yet deployed on an publicly available website, pending agreement from collaborators at Dimensions who provided us with parts of the data. Code, images, and updates on deployment and dataset will be made public on our OSF repository available at https://osf.io/83v2a/.

## 4 DISCUSSION AND FUTURE WORK

Scientific sleuths working to detect, correct, or retract problematic papers have been instrumental in maintaining the integrity and correctness of academic records. However, most of us are doing this work on our free time and the tools to assist us in detecting or tracking our findings are almost nonexistent. Notable exceptions are, for instance, tools such as The Problematic Paper Screener [7], but most tools focus on providing solution to detect errors or questionable practices. We therefore developed VIRUS to help us track large sets of papers being reported. However, we have, so far, only focused a specific subset of papers with ethical concerns [10]. We hope to continue developing this tool with a user-centred design process in mind with regular discussions and evaluations with our sleuthing colleagues to extend it to other databases and perhaps integrate it directly within the Problematic Paper Screener [7].

## ACKNOWLEDGMENTS

We thank our sleuthing colleagues from the informal collective Collège Invisible for their inspiring contribution to scientific integrity. We thank Dimensions for access to their database and API. Code for VIRUS was cleaned and partially generated by ChatGPT.

## REFERENCES

[1] B. Barbour and B. M. Stell. Pubpeer: Scientific assessment without metrics. *Gaming the metrics: Misconduct and manipulation in academic research*, pp. 149–155, 2020. 1

[2] L. Besançon, E. Bik, J. Heathers, and G. Meyerowitz-Katz. Correction of scientific literature: Too little, too late! *PLoS Biology*, 20(3):e3001572, 2022. doi: 10.1371/journal.pbio.3001572 1

[3] L. Besançon, N. Peiffer-Smadja, C. Segalas, H. Jiang, P. Masuzzo, C. Smout, E. Billy, M. Deforet, and C. Leyrat. Open science saves lives: lessons from the covid-19 pandemic. *BMC Med. Res. Methodol.*, 21(1):117, 2021. doi: 10.1186/s12874-021-01304-y 1

[4] L. Besançon, G. Cabanac, C. Labbé, and A. Magazinov. Sneaked references: Fabricated reference metadata distort citation counts. *J. Assoc. Inf. Sci. Technol.*, n/a(n/a). doi: 10.1002/asi.24896 1

[5] E. M. Bik, A. Casadevall, and F. C. Fang. The prevalence of inappropriate image duplication in biomedical research publications. *MBio*, 7(3):10–1128, 2016. doi: 10.1128/mbio.000809-16 1

[6] M. Bostock, V. Ogievetsky, and J. Heer. D³ data-driven documents. *EEE Trans. Vis. Comput. Graph.*, 17(12):2301–2309, 2011. doi: 10.1109/TVCG.2011.185 2

[7] G. Cabanac, C. Labbé, and A. Magazinov. Tortured phrases: A dubious writing style emerging in science, evidence of critical issues affecting established journals. *preprint arXiv:2107.06751*, 2021. 1, 2

[8] K. Choe, S. Jung, S. Park, H. Hong, and J. Seo. Papers101: Supporting the discovery process in the literature review workflow for novice researchers. In *Proc. PacificVis*, pp. 176–180, 2021. doi: 10.1109/PacificVis.52677.2021.00037 2

[9] S. A. Elmore. The altmetric attention score: what does it mean and why should i care?, 2018. doi: 10.1177/0192623318758294 2

[10] F. Frank, N. Florens, G. Meyerowitz-Katz, J. Barriere, É. Billy, V. Saada, A. Samuel, J. Robert, and L. Besançon. Raising concerns on questionable ethics approvals—a case study of 456 trials from the institute hospitalo-universitaire méditerranée infection. *Research integrity and peer review*, 8(1):9, 2023. doi: 10.1186/s41073-023-00134-4 1, 2

[11] P. Isenberg, F. Heimerl, S. Koch, T. Isenberg, P. Xu, C. D. Stolper, M. Sedlmair, J. Chen, T. Möller, and J. Stasko. Vispubdata.org: A metadata collection about ieee visualization (vis) publications. *IEEE Trans. Vis. Comput. Graph.*, 23(9):2199–2206, 2017. doi: 10.1109/TVCG.2016.2615308 2

[12] R. Van Noorden. More than 10,000 research papers were retracted in 2023—a new record. *Nature*, 624(7992):479–481, 2023. doi: 10.1038/d41586-023-03974-8 1

[13] Y. Wang, T.-Q. Peng, H. Lu, H. Wang, X. Xie, H. Qu, and Y. Wu. Seek for success: A visualization approach for understanding the dynamics of academic careers. *EEE Trans. Vis. Comput. Graph.*, 28(1):475–485, 2022. doi: 10.1109/TVCG.2021.3114790 2