# Publishing Visualization Studies as Registered Reports: Expected Benefits and Researchers' Attitudes

Lonni Besançon, Anastasia Bezerianos, Pierre Dragicevic, Petra Isenberg,

Yvonne Jansen

► To cite this version:

Lonni Besançon, Anastasia Bezerianos, Pierre Dragicevic, Petra Isenberg, Yvonne Jansen. Publishing Visualization Studies as Registered Reports: Expected Benefits and Researchers' Attitudes. 2021. [https://arxiv.org/abs/2109.03441](https://arxiv.org/abs/2109.03441v2)

HAL Id: hal-03441049

https://inria.hal.science/hal-03441049v2

Preprint submitted on 22 Sep 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Publishing Visualization Studies as Registered Reports: Expected Benefits and Researchers' Attitudes

Lonni Besançon*
Monash University
Linköping University

Anastasia Bezerianos†
Université Paris-Saclay, CNRS, Inria, LISN

Pierre Dragicevic‡
Univ. Bordeaux, CNRS, Inria, LaBRI

Petra Isenberg§
Université Paris-Saclay, CNRS, Inria, LISN

Yvonne Jansen¶
Univ. Bordeaux, CNRS, Inria, LaBRI

## ABSTRACT

**Background:** Registered Reports are publications in which study proposals are peer reviewed and pre-accepted before the study is run. Their adoption in other disciplines has been found to promote research quality and save time and resources.

**Objectives:** We offer a brief introduction to Registered Reports and their expected benefits for visualization research. We then report on a survey on the attitudes held in the Visualization community towards Registered Reports.

**Method:** We ran an online survey that was open from October 7, 2021 until October 24, 2021. We received 105 responses from visualization researchers.

**Results:** Most respondents expressed interest in Registered Reports as an additional submission format. Junior researchers as well as researchers with prior experience of preregistration tended to be more inclined to adopt this format. However, qualitative feedback highlights several perceived drawbacks and worries about Registered Reports, such as low suitability for some types of research, and increased workload for reviewers and authors.

**Conclusion:** While many respondents expressed interest in (and sometimes enthusiasm for) Registered Reports, many also voiced concerns that should be considered and discussed in the community.

**Reproducibility:** All materials are available at osf.io/4nrma/.

**Index Terms:** Human-centered computing—Visualization—Visualization techniques—Treemaps; Human-centered computing—Visualization—Visualization design and evaluation methods

## 1 INTRODUCTION

Publishing scientific papers is the primary means of communicating results, ideas, and tools to the public and scientific community. The publication of new scientific communications in journals or conferences is subject to peer-review which eventually decides if

the communication is worth publishing. However, traditional peer-review has been found to be sub-optimal [10, 48, 50]: it is hardly objective and there are many biases hindering the process, including a bias for positive results [15]. To address some of these issues and after facing the replication crisis [35], many fields tried improving their research and publishing methodology. Out of these efforts, Registered Reports emerged. According to the Center for Open Science, a Registered Report is:

> "a form of empirical journal article in which methods and proposed analyses are preregistered and peer-reviewed prior to research being conducted. High quality protocols are then provisionally accepted for publication before data collection commences. This format of article is designed to reward best practice in adhering to the hypothetico-deductive model of the scientific method. It eliminates a variety of questionable research practices, including low statistical power, selective reporting of results, and publication bias, while allowing complete flexibility to conduct exploratory (unregistered) analyses and report serendipitous findings."

The benefits of Registered Reports have been discussed extensively in the literature (see Sect. 2), and this publication format has been advocated by empirical computer science [15]. However, to our knowledge, no visualization journal or conference offers Registered Reports as a publication format. Here, we (i) explain Registered Reports and their benefits to the visualization community, and (ii) report the results of a survey. This survey aims at better understanding the visualization community's general awareness of Registered Reports, in how far such a publication model fits into community members' habits and expectations, and in crowdsourcing potential issues which may need to be taken into account to adapt Registered Reports to the needs of the visualization community.

We refer to our own article as a *quasi Registered Report*, because it tries to be as close as possible to a Registered Report but it has not been submitted to a venue with a proper two-stage Reviewing Process already in place. Even then, to the best of our knowledge, this article is the closest to a Registered Report published in a visualization venue so far. The present document is the second-stage version, with all survey results and discussions. The first-stage version, which has been accepted and presented at the alt.VIS venue before we ran the survey, is available as a preprint online [11].

## 2 REGISTERED REPORTS

The concept of Registered Report has been conceived more than a decade ago. Before we summarize here the hypothesized benefits and the evidence found to support them, we first quickly explain the differences between Registered Reports and preregistrations, because the terms are quite similar and can be easily confused.

<footnote>*e-mail: lonni.besancon@gmail.com</footnote>

<footnote>†e-mail: anastasia.bezerianos@lri.fr</footnote>

<footnote>‡e-mail: pierre.dragicevic@inria.fr</footnote>

<footnote>§e-mail: petra.isenberg@inria.fr</footnote>

<footnote>¶e-mail: yvonne.jansen@cnrs.fr</footnote>
## 2.1 Registered Reports vs. Preregistrations

Preregistering a study consists of uploading a complete data collection and analysis plan to an online repository that time-stamps it and conserves it [37, 39]. This plan (called a preregistration or sometimes a registration) is submitted before the study is conducted. Its goal is to make research more transparent and more trustworthy by drawing a clear line between confirmatory and exploratory analyses, and by preventing unintentional inflation of false-positive findings through analytical flexibility [9, 21, 38]. After the study is conducted, the researcher includes a link to the online preregistration in their paper, as a proof that they have followed a pre-specified plan. Deviations from the plan are allowed as long as they are explained in the paper. Preregistered studies can, by default, be submitted as regular papers and thus get peer-review feedback only after the study is already conducted.

A Registered Report is similar to a preregistration in that researchers must commit to a complete data collection and analysis plan before they run their study. However, a Registered Report is not just an analysis plan: it takes the form of a paper with partial content, called *Stage 1 Registered Report*, that is submitted for peer review instead of being uploaded to an online registry. This Stage 1 submission usually contains an introduction, related work as well as the methodology, including analysis plan. Reviewers can make comments about the study plan and ask for changes before the study is conducted. In addition, a study submitted as a Registered Report is conditionally pre-accepted for publication before the study is conducted, which is not possible with a preregistered study submitted as a regular paper.

Because study preregistration does not require any change to standard reviewing procedures, it can be readily adopted and used in any venue. Authors in the Visualization community have indeed begun to publish preregistered studies (e.g., [7, 27, 28, 33, 42, 52]). In contrast, Registered Reports are not yet supported.

## 2.2 Benefits of Registered Reports

### 2.2.1 Mitigating the File-Drawer Effect

A Registered Report is inherently result-agnostic: decisions to accept or reject publications are solely based on the relevance of the research questions and the proposed methodology to study these. As such, Registered Reports are expected to mitigate the bias towards the publication of positive findings, also known as the file-drawer effect. Not only does this bias hide interventions that do not work, it negatively impacts the trustworthiness of published results and can harm the credibility of a field [23, 49]. Initial investigation in psychology [44] has found some evidence that Registered Reports can indeed mitigate publication bias: Registered Reports presented only 47% positive results while comparable classical submissions presented an outstanding 96% positive results.

### 2.2.2 Avoiding Questionable Research Practices

By design, both Registered Reports and registrations are meant to prevent questionable research practices. In particular, they render HARKing (hypothesizing after the results are known) [16] impossible, and help prevent selective reporting and p-hacking [15]. However, in addition to preregistrations, Registered Report formats involve peer review prior to data collection, which gives them an additional benefit: the experimental design and methods can be improved upon through the recommendations made by reviewers.

### 2.2.3 Improving Quality and Reducing Scientific Waste

The reviewing of Registered Reports focuses on the authors' research questions and their importance, as well as the proposed methodology. The iterations between authors and reviewers before the study is conducted are likely to improve experimental design [6], help correct a wrongfully chosen research outcome/question, and detect errors or imprecision in data analysis. These iterations all take place

before any data collection begins and thus before any scientific resource is used, be it participants that are enrolled, money spent, or devices/doses used. For this reason, Registered Reports have been said to help reduce waste of scientific resources [9, 13, 14]. In addition, they could help reduce the norm of submitting already completed but potentially flawed studies to many different journals until they get accepted [24, 36]. Even when only considering the time spent reformating scholarly articles from one journal template to another [30], Registered Reports would already save precious scientific time.

## 2.3 Perceived Quality and Novelty

While Registered Reports do seem to increase research quality and mitigate many of the biases of the publishing system, some have voiced concerns that they could lead to scientists pursuing less novel or interesting research questions [8, 17]. However, initial evidence would seem to suggest that such fears are unfounded, and that Registered Reports are perceived by scholars as being statistically indistinguishable from regular papers in terms of novelty or creativity, while being at the same time perceived as more robust and of higher quality [46].

# 3 REGISTERED REPORTS FOR VISUALIZATION RESEARCH: WHY AND HOW

Today, according to the Center for Open Science, more than 300 scientific venues offer the option to submit Registered Reports.<fnref n="1" /> However, to the best of our knowledge, no visualization or even human-computer interaction venue offers the possibility to submit or review Registered Reports. Cockburn et al. [15] argued that the adoption of Registered Reports at major human-computer interaction venues could be daunting, but that reviewing workflows could easily be adapted to allow it.

Visualization papers come in many different types [31, 34], and not all of them report a user study. Those that do not include a user study would not benefit from the addition of Registered Reports as a publication type since no experimental design is included in them. Such manuscripts could continue to be submitted as regular papers and reviewed accordingly. On the other hand, papers with any form of empirical evaluation could benefit from being reviewed as Registered Reports. This form of review could be optional, and it could be up to the authors to choose whether or not they want their work to be reviewed as a registered report.

As an example, consider an author who wants to publish a novel visualization technique with an evaluation. They would submit a first-stage Registered Report that contains an introduction explaining the problem the technique is intended to solve, an overview of related work, a full description of the novel system or technique (ideally with an implementation available for reviewers to try), and a final section detailing how the empirical evaluation will be conducted, what data will be collected and how the data will be analyzed. Discussions and conclusion sections are left out. As another example, if an author wants to compare existing techniques or conduct a study that does not involve any new technique, their Registered Report would contain all the sections previously mentioned except the section on the novel system or technique. As reviewers may ask for changes in the experimental design, it would be futile to commence data collection before the reviews are sent to the authors. Reviewers might even suggest small improvements to the techniques tested.

For reviewers to be able to evaluate the methodology proposed by authors, the empirical evaluation section needs to be described as precisely as possible, much like a study preregistration [39]. Authors should include: research questions and hypotheses, detailed experiment design with all independent and dependent variables, sample size, and data analysis plan. Adding drafts of plots and tables

<footnote n="1">¹https://www.cos.io/initiatives/registered-reports</footnote>
with fictional data can be very helpful to reviewers. As an example, the first-stage version of the present *quasi Registered Report* can be found at [osf.io/4nrma/](https://osf.io/4nrma/) [9].

# 4 SURVEY OF VISUALIZATION RESEARCHERS

We describe below our research questions and hypotheses as well as our survey questions, its administration and the data analysis.

## 4.1 Research Questions and Hypotheses

We wanted to understand visualization researchers' knowledge of and attitudes towards Registered Reports. We set out to answer if the visualization community knows about Registered Reports, feels confident about submitting and reviewing them, and is ultimately interested in adopting them for submissions. We anticipated that interest and confidence might be dependent on the respondent's level of adoption of open science practices (e.g., preregistrations [40]), and perhaps also on their years of expertise in the field.

## 4.2 Administration and Ethics Approval

The survey was presented as an online form, and participation was voluntary. Participants were informed about the purpose of the survey and its approximate completion time (based on pilot studies) before they started answering. No personal information was collected. No compensation was offered but participants were given the link to our OSF repository so they could later check the results of the survey.

## 4.3 Recruitment

We shared our survey on social media (Twitter, Slack,...) and by email to past and present collaborators. We invited colleagues through word of mouth and informal sharing. Based on a previous similar survey of the visualization community, we anticipated that our survey would gather at least 50 answers. The final number of participants was 105.

## 4.4 Timing

Once our first-stage *quasi Registered Report* had been reviewed by the alt.VIS workshop organizers, we iterated on our survey design and our data analysis plan. We then submitted our study protocol for ethics approval. We started data collection after we obtained ethics approval. On the day of the workshop, we presented our preliminary results. We stopped data collection the night of the alt.VIS workshop, at 23:59 AoE time (October, 24 2021). We then wrote this second-stage Registered Report, which includes an analysis and discussion of all our 105 responses. In order to remain as close as possible to a Registered Report, we have sent the last revision to the workshop organizers (see letter to workshop organizers submitted together with the stage 1 version of this article on our OSF repository). Two organizers provided a light second-stage review to approve of our final manuscript.

## 4.5 Survey and Analysis Plan

We describe below the content of our survey and our analysis plan.

### 4.5.1 Survey

The survey we used is available on our OSF repository ([osf.io/4nrma/](https://osf.io/4nrma/)). It consists of five pages. The first page details the aim of the survey and asks for the participant's informed consent. The second page aims to gather participants' research experience through Likert items, asking them, for example, how long they have been active in the visualization community or their knowledge and use of preregistrations. The third page asks participants if they know what a Registered Report is, before giving, on the fourth page, the definition of Registered Reports according to the Center for Open Science. After having been explained what Registered Reports are, participants were invited to tell us if this definition corresponds to

<figure><img src="image_2.png" /><figcaption>Figure 1: Likert Plots for the four attitude questions.</figcaption></figure>

what they thought a Registered Report was, through Likert Items. We then invited them to use four Likert items to assess:

Q1 how helpful they think it would be to have their study design reviewed before data collection,

Q2 how willing they would be to submit a Registered Report,

Q3 how willing they would be to review a Registered Report,

Q4 how interested they would be in having Registered Reports as a publication option in the visualization community.

All of the questions above are complemented with an optional text field where participants can explain their answers and make comments if they wish to.

### 4.5.2 Data Analysis Plan

The detailed analysis plan that we submitted as a first-stage Registered Report [11] is available on our OSF repository ([osf.io/4nrma/](https://osf.io/4nrma/)). It included a qualitative and quantitative analysis of simulated data. We also planned to report the results of the research experience questions on page 2 as descriptive statistics. Any analysis not included in the first-stage Registered Report is reported below as complementary analysis.

## 4.6 Changes to the Plan

During late pilot testing, we realized that some participants might wrongly report knowing what a registered report is, for example if they read it as a synonym of a preregistered study. Therefore, after explaining what the term means, we added a question asking how well the explanation captures what they thought a registered report was. Response analyses are reported in Sect. 5.2.

Concerning participant recruitment, our Stage 1 submission states that we planned to share our survey on social media (Twitter, Slack,...) and by email to past and present collaborators. We thus initially only invited colleagues through word of mouth and informal sharing. However, we realized that this approach would likely only reach a small subset of all potential participants considered eligible according to the criteria indicated in the survey. We therefore decided to additionally extracted 796 email addresses of from articles published at VIS and sent an email invitation to all past authors.

All other changes were trivial corrections to the wording of the survey and to the analysis code.

# 5 RESULTS

We received a total of 105 responses to our survey, which we analyze here. Our sample is comprised of 56 people with “10 years or more” of experience (53%), 28 with “5 to 10 years” (27%), and 21 with “less than 5 years” (20%).
<figure><img src="image_3.png" /><figcaption>Figure 2: Mean responses to the four attitude questions depending on years of experience in visualization research. Error bars: 95% CIs.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Figure 3: Mean responses to the four attitude questions depending on the number of preregistrations submitted. Error bars: 95% CIs.</figcaption></figure>

## 5.1 Planned Quantitative Analysis

We report a Likert plot showing responses to each of the four attitude questions in Fig. 1. With respect to our first question (Q1) on the usefulness of early reviews, only 5 respondents (5%) found early reviews as provided by Registered Reports "not helpful at all". Responses to Q2 and Q3 show that most respondents (75 respondents, 71%) are rather interested and willing to submit Registered Reports and that many of them (36 respondents, 34%) are willing to review them. The slightly less positive answers for Q2 (reviewing), might indicate that finding reviewers for Registered Reports could be, at least initially, slightly difficult if the visualization community adopted Registered Reports. That being said, the mismatch does not seem to be too large as a significant number of participants (50 respondents, 48%) seem to be neutral about reviewing Registered Reports, and the perception of this new submission format may change once they become more common. Finally, responses to the fourth question (Q4) show that a majority of our respondents are interested in having Registered Reports as an alternative publication model (66 participants, 63%). Note that although responses to Q1 appear more negative than responses to Q2, Q3, and Q4, the response options were different, with a more positive framing for Q1 (e.g., compare "Somewhat helpful" for Q1 with "Reluctant" for Q2).

To check whether attitude depends on seniority (operationalized as years of experiences in visualization research), or on current adoption of open science practices (operationalized as the number of preregistered studies [40] previously submitted), we report average responses to the four attitude questions as a function of seniority (Fig. 2) and number of preregistrations submitted (Fig. 3). In both figures, we report the mean response with 95% bootstrap confidence intervals. While the use of means to summarize ordinal responses is not universally accepted [43], it is a common and

methodologically-defensible practice [32, 43]. We interpret our results with an estimation approach [20].

In Fig. 2, we see that younger researchers are overall slightly more positive in their answers to the four attitude questions than more experienced researchers. The difference is very clear between junior researchers (< 5 years of experience) and the most senior ones (> 10 years), but evidence that there is a difference on average is weaker for adjacent categories. In Fig. 3 we have good evidence that people with no prior preregistration experience are slightly more negative in their answers than people with some experience (1 to 5 preregistrations submitted). For people with 6+ preregistrations (6 respondents out of 93), intervals are too wide to conclude due to the small sample size.

## 5.2 Additional Quantitative Analyses

Here we report responses to the non-attitude questions like knowledge and use of preregistration and registered reports (Fig. 4). First, concerning familiarity with preregistrations, the answers we obtained suggest that overall, knowledge of preregistration varies a lot among our respondents. Indeed, 39 participants (37%) responded "I have a good understanding of what it is", while 38 (36%) ticked "I have a vague notion of what it means", and 28 (27%) reported never having heard of it. Lack of familiarity could be explained by the breadth of visualization-related publications for which registrations are not useful. Concerning familiarity with Registered Reports, the majority of our sample reported having little to no knowledge. Indeed, 46 participants (44%) reported having "never heard of it", and 41 (39%) said they had "a vague notion of what it means." Only 14 (13%) reported having "a good understanding of what it is". Four participants (4%) answered that they had "written and submitted [one] before." Since no visualization venues offer Registered Reports, these answers could be explained either by the fact that they submitted one in another field of research, or that they mistook Registered Reports for preregistrations. Related to this point, we asked participants if the definition of a Registered Report corresponded to what they expected. The answers to this question were: 13 (12%) "Not at all", 12 (11%) "Not very well", 47 (45%) "Reasonably well", 24 (23%) "Very well", 9 (9%) "Perfectly". These answers are consistent with our hypothesis that some people mistook a preregistration for a Registered Report.

## 5.3 Qualitative Analysis

Our survey gave respondents the opportunity to provide additional explanations about the four attitude questions in the survey. We conducted a thematic analysis of these answers. Two coders of the author team analyzed responses independently to categorize them based on their themes. A third coder was involved to resolve non-matching categorization of answers.

Our complete list of codes is reported in Table 1, with frequencies of appearance. If a code appears multiple times in a respondent's answers, it is counted only once. The maximum count is 55, which is the number of respondents (out of the total 105) who provided qualitative data. The other respondents filled out the form but did not enter any explanatory text.

The comments we received on questions Q1–Q4 predominantly raised concerns regarding Registered Reports but also highlighted some of their benefits. Here we highlight the major themes we found and comment on the most important in the discussion section.

### 5.3.1 Comments on the Benefits of Registered Reports

As a major benefit, respondents highlighted the possibility of Registered Reports to improve research quality. Respondents mentioned that they would welcome early help and feedback (9× Q1, 4× Q2) and expect to see an increase in the quality of their own research (5× Q1, 4× Q2).
<table><caption>Table 1: Full list of codes from the qualitative analysis regarding our questions about registered reports (RR).</caption><thead><tr><th>Code</th><th># Responses (out of 55)</th></tr></thead><tbody><tr><td><strong>Comments on Q1:</strong><br>How helpful would it would be to have your study design reviewed before data collection?</td><td>46</td></tr><tr><td><strong>Reasons given why RRs would be useful</strong></td><td>19</td></tr><tr><td>One can receive help and insight from others (early)</td><td>9</td></tr><tr><td>RRs will increase research quality</td><td>5</td></tr><tr><td>RRs will reduce risk and wasted resources</td><td>4</td></tr><tr><td>RRs fit my work or a certain type of visualization work/process</td><td>2</td></tr><tr><td>RRs would help junior or researchers new to a field</td><td>2</td></tr><tr><td>Early review can increased confidence in own research</td><td>2</td></tr><tr><td>Early feedback & planning are already part of our research process</td><td>1</td></tr><tr><td>RRs allow to publish negative or inconclusive results</td><td>1</td></tr><tr><td>Early feedback reduces risk of unexpected reviewer feedback</td><td>1</td></tr><tr><td><strong>Reasons given why early RRs would not be helpful</strong></td><td>29</td></tr><tr><td>RRs are not suitable for certain types of vis / hci work</td><td>12</td></tr><tr><td>RRs increase workload for authors and risk delaying publication</td><td>8</td></tr><tr><td>RRs make adjustments to research more difficult</td><td>6</td></tr><tr><td>RRs increase workload for reviewers and organizational overhead</td><td>6</td></tr><tr><td>Current practices can already ensure sufficient research quality</td><td>3</td></tr><tr><td>There is no required expertise & guidelines yet in the community</td><td>2</td></tr><tr><td>RRs create the risk of reviewers stealing ideas</td><td>2</td></tr><tr><td>Potential for poor reviews remains with RRs</td><td>2</td></tr><tr><td>For reviewers it is hard to identify problems without seeing the final study</td><td>2</td></tr><tr><td>RRs may give a false sense of security. Issues may appear after early review</td><td>2</td></tr><tr><td>RR reviewers might steer research in a bad direction</td><td>1</td></tr><tr><td><strong>Comments that were not clearly for or against finding RR helpful</strong></td><td>1</td></tr><tr><td>Researchers get feedback informally now</td><td>1</td></tr><tr><td><strong>Comments regarding Q3: How willing would you be to review a RR?</strong></td><td>36</td></tr><tr><td><strong>Reason for being willing to review a RR</strong></td><td>9</td></tr><tr><td>I would like to help improve research quality</td><td>4</td></tr><tr><td>It is fulfilling to give advice early</td><td>3</td></tr><tr><td>Early review might be easier</td><td>2</td></tr><tr><td>Early review allows to focus reviews on other aspects</td><td>1</td></tr><tr><td><strong>Reason against being willing to review a RR</strong></td><td>25</td></tr><tr><td>It takes more time, increases workload</td><td>15</td></tr><tr><td>I do not have the expertise to assess study designs/statistical analyses</td><td>4</td></tr><tr><td>An early reviewer has a higher responsibility for others' research projects</td><td>2</td></tr><tr><td>I don't want to be involved with RR</td><td>2</td></tr><tr><td>I cannot see the benefit of RRs</td><td>2</td></tr><tr><td>I would not want to dictate my opinions on methodology to others</td><td>1</td></tr><tr><td>RRs might incentivise sloppier second round reviewing</td><td>1</td></tr><tr><td>Reviewing RRs is more difficult</td><td>1</td></tr><tr><td><strong>Comments on Q2: How willing are you to submit a RR</strong></td><td>31</td></tr><tr><td><strong>Reason for being willing to submit a RR</strong></td><td>8</td></tr><tr><td>To improve the quality of my research</td><td>4</td></tr><tr><td>To receive early feedback</td><td>4</td></tr><tr><td>To make it easier to publish the research I do</td><td>1</td></tr><tr><td>To open discussion of unpublished work</td><td>1</td></tr><tr><td>To save resources</td><td>1</td></tr><tr><td><strong>Reason for not being willing to submit a RR</strong></td><td>13</td></tr><tr><td>RR increase my workload</td><td>4</td></tr><tr><td>Review quality for RRs might be too low</td><td>4</td></tr><tr><td>RRs increase the time to get published</td><td>4</td></tr><tr><td>I would need trust reviewer confidentiality</td><td>2</td></tr><tr><td>Not clear if reviewer comments will be helpful</td><td>2</td></tr><tr><td>RRs are not a fit for the work I do</td><td>2</td></tr><tr><td>RRs do not seem useful for validating visualization designs</td><td>1</td></tr><tr><td>Already have different ways to accomplish the goals of RRs</td><td>1</td></tr><tr><td><strong>I would submit if...</strong></td><td>11</td></tr><tr><td>... the research I do matches</td><td>5</td></tr><tr><td>... review quality is good</td><td>3</td></tr><tr><td>... the timing works out</td><td>3</td></tr><tr><td>... it doesn't increase the amount of work</td><td>1</td></tr><tr><td>... I will not receive a random stage 2 rejection</td><td>1</td></tr><tr><td>... reviews are helpful</td><td>1</td></tr><tr><td>... there will be credit for reviewers</td><td>1</td></tr><tr><td><strong>Comments on Q4:</strong></td><td>20</td></tr><tr><td><strong>How interested are you in having RRs as a publication option in the visualization community?</strong></td><td>7</td></tr><tr><td><strong>Reasons for interest in RRs as submission category</strong></td><td>3</td></tr><tr><td>RRs would benefit the community</td><td>2</td></tr><tr><td>RRs would increase scientific rigor</td><td>1</td></tr><tr><td>RRs allow publishing negative results</td><td>1</td></tr><tr><td>RRs are the right thing to do</td><td>1</td></tr><tr><td>If RRs are introduced as a complementary publication model</td><td>1</td></tr><tr><td><strong>Reason against interest in RRs as submission category</strong></td><td>10</td></tr><tr><td>RRs can have a negative impact on some parts of Vis research</td><td>3</td></tr><tr><td>It is unclear if RRs are practically realizable</td><td>3</td></tr><tr><td>RRs increase the community's workload</td><td>2</td></tr><tr><td>RRs raise confidentiality concerns regarding "stealing" of ideas</td><td>2</td></tr><tr><td>It is unclear if there are enough qualified reviewers</td><td>1</td></tr><tr><td>There is a risk of breaking the current reviewing system</td><td>1</td></tr></tbody></table>
<figure><img src="image_5.png" /><figcaption>Figure 4: Responses to the definition questions.</figcaption></figure>

> "I'm supportive of the idea of registered reports: they provide a sanity check on the soundness of a proposed study and can lower barriers to engaging in experimental research to people just getting involved in such study."

Four respondents (Q2) mentioned that they would be happy to help improve research quality as a reviewer of registered reports and three that they would find it fulfilling to give advice early (Q2).

> "I rather review a registered review and give authors feedback early on than telling authors that their study is not "useful" afterwards and feeling bad because they probably have spend time, money, and resources in conducting a study which is leading nowhere."

Five responses similarly mentioned the potential for registered reports to save resources (time, money, ...) in the research process (4× Q1, 1× Q2).

### 5.3.2 Concerns Raised Regarding Registered Reports

Despite some respondents mentioning that they could save time and resources with Registered Reports, others saw time and workload as a risk. Specifically, responses mentioned an increased workload for authors and risk of delaying publication (8× Q1, 8× total Q2).

> "There are questions where the experimental methods are tricky (or even part of the contribution), in which case having additional input on the methods would be helpful. Other experiments are pretty straightforward, and having an expectation of prior methodological review would simply be yet another delay in the process of getting the results to the research community."

Others voiced concerns over increased workload for reviewers (6× Q1, 15× Q3).

> "My reviewing load is already high and I do not see how to reduce it. Thus, when I start a new line of reviewing activity, the question arises: How to get rid of some existing (reviewing) tasks."

Comments regarding added workload were sometimes coupled with concerns regarding reviewer expertise. Reviewer qualification concerns were evident in several responses (11× total Q1, 6× total Q2, 4× Q3, 3× total Q4). Authors feared that their work may be steered in the wrong direction and that the community has no guidelines and procedures in place to deal with Registered Reports.

> "While most reviewers are reasonable, there are often reviewers who believe the purpose of a review is to oppose one's own view onto the authors."

From the reviewer's perspective, several comments mentioned that it might be harder to review incomplete studies:

> "The direct acceptance condition worries me a bit as it is often hard to identify a problematic study just based on its description. Small variations in experimental instructions or in an experimental apparatus can make a huge difference. As a reviewer, I may only realize a huge problem by looking on the reported results (or the raw data)."

Several respondents (2× Q1, 2× Q2, 2× Q4) voiced fears over reviewers being able to steal ideas for research to be conducted (i.e., scooping) and asked questions about the early publication of Registered Reports to make them citable.

> "my question is regarding citing such a registered report and making sure others do not "steal" the idea - does a registered report have a DOI that can be cited later on, this is a bit unclear."

The most common worry regarding Registered Reports, however, concerned the applicability to only certain types of VIS work (12× Q1, 2× Q2, 3× Q4). In particular the applicability to qualitative research was mentioned by several respondents:

> "It is the very nature of qualitative research that the subject of interest can adapt and change as the study progresses. Therefore the whole notion of preregistration is out of alignment with this approach to research."

> "Furthermore, I am deeply concerned that efforts that make preregistration (and registered reports) an expectation in vis will further marginalize qualitative research studies, which are already marginalized."

Similar to these concerns, other responses expressed a worry that by writing a Registered Report, the research would be constrained by external reviewers, or that no subsequent changes could be made to adjust and improve studies.

> "This could stifle approaches and innovation by discouraging efforts early in the process."

> "A preregistration is almost guaranteed to overtly constrain the research to reinforce preconceptions."

A few respondents found registered reports simply unnecessary because they already used other mechanisms for collecting early feedback outside of a formal review process, for example through internal reviews (3× Q1, 1× Q2).

Several respondents made conditional statements regarding their willingness to adopt and review Registered Reports that echo the concerns listed above. Respondents mentioned, for example, that they would submit Registered Reports if their research matches, if they can expect a high review quality, or if Registered Reports would mean not losing time or wasting effort in the research process.
# 6 DISCUSSIONS

Our quantitative results highlight both interests but also worries about having Registered Reports available as a publication option for the visualization community. Some of the mentioned concerns are legitimate, while others may partly stem from a misconception of what a registered report ultimately is and means to achieve. We already discussed some concerns in Sect. 2 (e.g., that Registered Reports would be too time-consuming) and here we cover additional concerns that we consider important to address.

## 6.1 Constraining Research and Creativity

Some of the answers we obtained categorized Registered Reports as being constraining regarding innovation but also the research process itself. This parallels a perception from many researchers that study preregistration constrains research [18,37]. However, the idea of pre-committing to a data collection and analysis plan (both in Registered Reports and in preregistrations) is not to forbid any deviation from the initial plan, in particular if such a deviation would allow the protocol to eventually be more robust or the research to be more worthwhile, but rather to make sure that the deviations from the initial plan are transparently reported [37]. In fact, while the authors of this paper have preregistered their analysis plans in some of their visualization papers, they also have deviated from such plans on several occasions, but transparently reported deviations (see e.g., [45,51]). However, contrary to preregistration, Registered Reports get a conditional acceptance based on their initial plan, so while some deviation might be possible, the degree of deviation allowed may well be smaller. We discuss this issue further in Sect. 6.5.

Some respondents worried that Registered Reports could hinder the novelty or creativity of produced research. Although this is possible, Soderberg et al. [46] found that reviewers in psychology and neuroscience judged Registered Reports as no less novel or creative than traditional papers, while rating Registered Reports as more rigorous and of higher quality overall. As such, there is some evidence that creativity in study design might not be impacted as much as feared by some respondents. However, creative visualization design work (as opposed to experiment design work) would likely not benefit from being submitted as a Registered Report due to the evolving nature of creative work.

## 6.2 Risks of Stealth and Scooping

Echoing past findings around perception of preregistration [47], some of our respondents highlighted their fears of a breach of confidentiality: they worried that reviewers might steal their ideas and potentially run the study before them. Such behavior in any review process would be considered academic misconduct (see for example the following retraction statement [3]) and can happen also with a traditional reviewing process. Once uncovered, this behavior will usually lead to retractions [4]. However, a simple way to avoid this issue is to preprint first-stage Registered Reports when they are submitted. This is also what we have done with this quasi Registered Report [11]. Preprints are time-stamped and allow researchers to claim intellectual precedence [19, 22, 41].

## 6.3 Unreasonable Reviewer Requests

Some of our respondents mentioned that they feared unreasonable demands from their peers during the first stage of the reviewing process. There is indeed a risk that reviewers require changes to an experimental protocol that the authors consider unreasonable or misguided. In classical reviewing workflows, reviewers can influence or require changes to a statistical analysis, but not to an experiment protocol, since the study has already been conducted. If they do not like the way the study was done, reviewers can only ask the authors to highlight its limitations, or recommend rejection. However, the situation is not so different with Registered Reports, since authors are not obligated to make any changes they disagree with, and, if

their manuscript is pending major/minor revision, they are free to explain their disagreements to reviewers through the summary of revisions and responses to reviewers. Should they fail to convince them, their paper will be rejected just as they would have likely been rejected had the study been already conducted. The authors could then try to submit their Register Report to another venue (should this format become more broadly accepted in visualization) or conduct their study anyway and submit it as a regular paper. In such a case, the authors would have, in the worst case, just delayed the publication of their work, and will most likely have received at least some valuable feedback on their study design. Finally, opening up the reviewing process, by making the anonymous reviews public (and anonymous), might help mitigate the issue of unreasonable reviews [10], as it may increase scrutiny on both the Registered Report and its reviews, and therefore increase the chances that unreasonable demands are publicly flagged.

## 6.4 Registered Reports and Qualitative Research

In the answers to our survey, several researchers highlighted the potential incompatibility of Registered Reports with qualitative research. This is a legitimate worry considering that Registered Reports were initially designed, much like preregistration, for quantitative methods [12]. Given the relative novelty of Registered Reports, there is little literature on their possible benefits for qualitative research. However, there is some literature on the benefits and challenges of preregistration for qualitative research [25, 26, 29]. Similar to Registered Reports for quantitative research, Registered Reports of qualitative work could also provide early feedback on the study design, prevent publication bias, and by forcing researchers to describe their processes clearly they could promote transparency and increase the perception of rigor for qualitative work.

However, the depth to which the data analysis procedures can be described for qualitative work will necessarily be different compared to quantitative research. As the researcher is the one producing summary data and patterns—and not a specific statistical instrument—the processes of data analysis are not mechanical and the success of a data analysis hinges on the ability of the researcher and their invested time and effort to dig through emerging patterns. As such it is perhaps not a given (and this needs to be discussed further) that a Registered Report of qualitative work accepted in Stage 1 should be auto-accepted in Stage 2.

It thus seems that more discussions are needed on the topic of the Registered Report format and how it could work for qualitative research. Noteworthy, some venues already use two-step submission processes for contributions which are not quantitative research, for example, the *State of the Art Report (STAR)* track at Eurovis. While at those there is no conditional acceptance after the first step, the benefits for research mentioned above in the qualitative responses likely apply there as well, for instance, benefiting from early feedback and reducing the risk of wasted resources.

## 6.5 How much Deviation is Acceptable

Although it is considered acceptable to deviate from a preregistration [37], a Stage 1 registered report that is approved by reviewers is essentially a conditionally accepted paper. Because the reviewers only approved the version of the protocol that they have seen, it is unclear what kind of changes would be deemed acceptable for a Stage 2 submission. For instance, we do not know if the changes we have made to our own survey (Sect. 4.6) would have disqualified our paper in an actual Registered Report reviewing process. We scanned through some of the journals that implement Registered Reports, and found that they rarely provide clear guidelines on what changes are acceptable between a Stage 1 and Stage 2 manuscript [1, 2, 5]. It seems that journals usually advise authors who want to make changes to contact the editor and to potentially resubmit for a Stage 1 reviewing before proceeding to data collection. If
Registered Reports were to be adopted in visualization research, it would be helpful to have clear guidelines explaining what changes can be considered minor and what changes require a re-submission as Stage 1. Regardless, this type of decision is inherently subjective and in many cases, the final decision will likely depend on who is reviewing.

## 7 CONCLUSION

We introduced and motivated the use of Registered Reports as a publication model for visualization studies, and reported on a survey attempting to better understand visualization researchers' attitudes towards them. This article itself takes the form of a *quasi* Registered Report, which shares most of the characteristics of Registered Reports, except the alt.VIS reviewing workflow has not been designed with Registered Reports in mind. Despite the widespread adoption of this new publication model in a range of disciplines, and the tremendous benefits it can provide to all actors involved in certain types of research (authors, reviewers, journal editors), Registered Reports do not yet exist for visualization work. Our survey highlights that the community seems to be open to the idea of Registered Reports as an alternative publication model in visualization venues. The qualitative feedback we gathered also helped us understand some of the concerns around registration overall, some of which we have hopefully addressed in our discussions, and others that still need more in-depth work and discussion within our community. We hope this article will encourage journal editors, conference organizers, and authors in visualization to consider adopting Registered Reports as an additional publication format, and help them understand the opportunities and challenges in implementing them.

## ACKNOWLEDGMENTS

We wish to confirm our registered thanks to our participants for their time and valuable answers. Lonni Besançon's work has been supported by the Knut and Alice Wallenberg Foundation (grant KAW 2019.0024)

## REFERENCES

[1] Nature scientific reports - registered reports. https://www.nature.com/srep/journal-policies/registered-reports

[2] Registered reports - guidelines for authors and reviewers. https://www.cambridge.org/core/journals/journal-of-child-language/information/registered-reports-guidelines-for-authors-and-reviewers

[3] Retraction statement: Potentiality of neem (azadirachta indica) powder in rheology modification of oil-in-water emulsion. *Journal of Food Process Engineering*, 40(2):e12531, 2017. doi: 10.1111/jfpe.12531

[4] Scientific misconduct hurts. *Annals of Internal Medicine*, 166(2):148-149, 2017. PMID: 27951591. doi: 10.7326/M16-2550

[5] Bmj open - registered reports guidelines, May 2021. https://openscience.bmj.com/pages/registered-reports-guidelines, journal=BMJ Open Science.

[6] C. Allen and D. M. A. Mehler. Open science challenges, benefits and tips in early career and beyond. *PLOS Biology*, 17(5):1-14, 05 2019. doi: 10.1371/journal.pbio.3000246

[7] A. Batch, A. Cunningham, M. Cordeil, N. Elmqvist, T. Dwyer, B. H. Thomas, and K. Marriott. There is no spoon: Evaluating performance, space use, and presence with expert domain users in immersive analytics. *IEEE transactions on visualization and computer graphics*, 26(1):536-546, 2019. doi: 10.1109/TVCG.2019.2934803

[8] R. F. Baumeister. Charting the future of social psychology on stormy seas: Winners, losers, and recommendations. *Journal of Experimental Social Psychology*, 66:153-158, 2016. *Rigorous and Replicable Methods in Social Psychology*. doi: 10.1016/j.jesp.2016.02.003

[9] L. Besançon, N. Peiffer-Smadja, C. Segalas, H. Jiang, P. Masuzzo, C. Smout, E. Billy, M. Deforet, and C. Leyrat. Open science saves lives: lessons from the covid-19 pandemic. *BMC Medical Research Methodology*, 21(1):1-18, 2021. doi: 10.1186/s12874-021-01304-y

[10] L. Besançon, N. Rönnberg, J. Löwgren, J. P. Tennant, and M. Cooper. Open up: a survey on open and non-anonymized peer reviewing. *Research Integrity and Peer Review*, 5(1):1-11, 2020. doi: 10.1186/s41073-020-00094-z

[11] L. Besançon, A. Bezerianos, P. Dragicevic, P. Isenberg, and Y. Jansen. Publishing visualization studies as registered reports: Expected benefits and researchers' attitudes, Jul 2021. Stage 1 available under the first version of archived paper. doi: 10.31219/osf.io/3z7kx

[12] B. Bowers. Pre-registering qualitative research: Benefits, unintended consequences, and unanswered questions. *Innovation in Aging*, 3(Suppl 1):S400, 2019. doi: 10.1093%2Fgeroni%2Figz038.1483

[13] I. Chalmers and P. Glasziou. Avoidable waste in the production and reporting of research evidence. *The Lancet*, 374(9683):86-89, 2009. doi: 10.1016/S0140-6736(09)60329-9

[14] C. D. Chambers, E. Feredoes, S. D. Muthukumaraswamy, and P. Etchells. Instead of "playing the game" it is time to change the rules: Registered reports at aims neuroscience and beyond. *AIMS Neuroscience*, 1(1):4-17, 2014. doi: 10.3934/Neuroscience2014.1.4

[15] A. Cockburn, P. Dragicevic, L. Besançon, and C. Gutwin. Threats of a replication crisis in empirical computer science. *Commun. ACM*, 63(8):70-79, July 2020. doi: 10.1145/3360311

[16] A. Cockburn, C. Gutwin, and A. Dix. *HARK No More: On the Pre-registration of CHI Experiments*, p. 1-12. Association for Computing Machinery, New York, NY, USA, 2018. doi: 10.1145/3173574.3173715

[17] A. Cropley. Research as artisanship versus research as generation of novelty: The march to nowhere. *Creativity Research Journal*, 30(4):323-328, 2018. doi: 10.1080/10400419.2018.1500190

[18] A. DeHaven. Preregistration: A plan, not a prison, May 2017. https://www.cos.io/blog/preregistration-plan-not-prison

[19] P. Desjardins-Proulx, E. P. White, J. J. Adamson, K. Ram, T. Poisot, and D. Gravel. The case for open preprints in biology. *PLoS Biology*, 11(5):e1001563, 2013. doi: 10.1371/journal.pbio.1001563

[20] P. Dragicevic. Fair statistical communication in HCI. In J. Robertson and M. Kaptein, eds., *Modern Statistical Methods for HCI*, chap. 13, pp. 291-330. Springer International Publishing, Cham, Switzerland, 2016. doi: 10.1007/978-3-319-26633-6_13

[21] W. Forstmeier, E.-J. Wagenmakers, and T. H. Parker. Detecting and avoiding likely false-positive findings—a practical guide. *Biological Reviews*, 92(4):1941-1968, 2017. doi: 10.1111/brv.12315

[22] P. Ginsparg. Arxiv at 20. *Nature*, 476(7359):145-147, 2011. doi: 10.1038/476145a

[23] B. Goldacre. What doctors don't know about the drugs they prescribe [TED talk]. Online, 2012.

[24] J. A. Grand, S. G. Rogelberg, G. C. Banks, R. S. Landis, and S. Tonidandel. From outcome to process focus: Fostering a more robust psychological science through registered reports and results-blind reviewing. *Perspectives on Psychological Science*, 13(4):448-456, 2018. PMID: 29961411. doi: 10.1177/1745691618767883

[25] A. M. Jacobs. Pre-registration and results-free review in observational and qualitative research, Mar. 2020. doi: 10.1017/S9781108762519.009

[26] D. Kapiszewski and S. Karcher. Transparency in practice in qualitative research. *PS: Political Science & Politics*, 54(2):285-291, 2021. doi: 10.1017/S1049096520000955

[27] Y.-S. Kim, P. Kayongo, M. Grunde-McLaughlin, and J. Hullman. Bayesian-assisted inference from visualized data. *IEEE Transactions on Visualization and Computer Graphics*, 27(2):989-999, 2020. doi: 10.1109/TVCG.2020.3028984

[28] R. Kosara. Evidence for area as the primary visual cue in pie charts. In *2019 IEEE Visualization Conference (VIS)*, pp. 101-105. IEEE, 2019. doi: 10.1109/VISUAL.2019.8933547

[29] T. L. Haven and D. L. Van Grootel. Preregistering qualitative research. *Accountability in Research*, 26(3):229-244, 2019. doi: 10.1080/08989621.2019.1580147

[30] A. G. LeBlanc, J. D. Barnes, T. J. Saunders, M. S. Tremblay, and J.-P. Chaput. Scientific sinkhole: The pernicious price of formatting. *PLOS ONE*, 14(9):1-7, 09 2019. doi: 10.1371/journal.pone.0223116

[31] B. Lee, K. Isaacs, D. A. Szafir, G. E. Marai, C. Turkay, M. Tory, S. Carpendale, and A. Endert. Broadening intellectual diversity in vi-
sualization research papers. *IEEE computer graphics and applications*, 39(4):78–85, 2019. doi: 10.1109/MCG.2019.2914844

[32] J. R. Lewis. Multipoint scales: Mean and median differences and observed significance levels. *International Journal of Human-Computer Interaction*, 5(4):383–392, 1993. doi: 10.1080/10447319309526075

[33] L. Morais, Y. Jansen, N. Andrade, and P. Dragicevic. Can anthropographics promote prosociality? a review and large-sample study. In *Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems*, pp. 1–18, 2021. doi: 10.1145/3411764.3445637

[34] T. Munzner. Process and pitfalls in writing information visualization research papers. In *Information visualization*, pp. 134–153. Springer, 2008. doi: 10.1007/978-3-540-70956-5_6

[35] L. D. Nelson, J. Simmons, and U. Simonsohn. Psychology's renaissance. *Annual review of psychology*, 69:511–534, 2018.

[36] B. A. Nosek and Y. Bar-Anan. Scientific utopia: I. opening scientific communication. *Psychological Inquiry*, 23(3):217–243, 2012.

[37] B. A. Nosek, E. D. Beck, L. Campbell, J. K. Flake, T. E. Hardwicke, D. T. Mellor, A. E. van't Veer, and S. Vazire. Preregistration is hard, and worthwhile. *Trends in cognitive sciences*, 23(10):815–818, 2019. doi: 10.1016/j.tics.2019.07.009

[38] B. A. Nosek, C. R. Ebersole, A. C. DeHaven, and D. T. Mellor. The preregistration revolution. *Proceedings of the National Academy of Sciences*, 115(11):2600–2606, 2018. doi: 10.1073/pnas.1708274114

[39] J. P. Simmons, L. D. Nelson, and U. Simonsohn. Pre-registration: why and how. *Journal of Consumer Psychology*, 31(1):151–162, 2021. doi: 10.1002/jcpy.1208

[40] J. P. Simmons, L. D. Nelson, and U. Simonsohn. Pre-registration: Why and how. *Journal of Consumer Psychology*, 31(1):151–162, 2021.

[41] O. Pourret, D. E. Irawan, and J. P. Tennant. On the potential of preprints in geochemistry: The good, the bad, and the ugly. *Sustainability*, 12(8), 2020. doi: 10.3390/su12083360

[42] K. Reda and M. E. Papka. Evaluating gradient perception in color-coded scalar fields. In *2019 IEEE Visualization Conference (VIS)*, pp. 271–275. IEEE, 2019.

[43] J. Sauro. Can you take the mean of ordinal data? https://

measuringu.com/mean-ordinal/#. Accessed: 2019-06-15.

[44] A. M. Scheel, M. Schijen, and D. Lakens. An excess of positive results: Comparing the standard psychology literature with registered reports, Feb 2020. doi: 10.31234/osf.io/p6e9c

[45] M. Sereno, L. Besançon, and T. Isenberg. Point Specification in Collaborative Visualization for 3D Scalar Fields Using Augmented Reality. *Virtual Reality*, 2022. doi: 10.1007/s10055-021-00614-2

[46] C. K. Soderberg, T. M. Errington, S. R. Schiavone, J. Bottesini, F. S. Thorn, S. Vazire, K. M. Esterling, and B. A. Nosek. Initial evidence of research quality of registered reports compared with the standard publishing model. *Nature Human Behaviour*, pp. 1–8, 2021. doi: 10.1038/s41562-021-01142-4

[47] G. M. Tawfik, H. T. N. Giang, S. Ghozy, A. Altibi, H. Kandil, H.-H. Le, P. S. Eid, I. Radwan, O. M. Makram, T. T. T. Hien, M. Sherif, A.-S. Hossain, T. L. L. Thang, L. Puljak, H. Salem, T. Numair, K. Moji, and N. T. Huy. Current issues in protocol registration for systematic review and meta-analysis studies: a survey of global researchers, 2020. doi: 10.21203/rs.3.rs-18841/v1

[48] J. P. Tennant. The state of the art in peer review. *FEMS Microbiology Letters*, 365(19):fny204, 2018. doi: 10.1093/femsle/fny204

[49] The Economist. Unreliable research: Trouble at the lab. Online, 2013.

[50] R. Walker and P. Rocha da Silva. Emerging trends in peer review—a survey. *Frontiers in Neuroscience*, 9:169, 2015. doi: 10.3389/fnins.2015.00169

[51] X. Wang, L. Besançon, M. Ammi, and T. Isenberg. Understanding Differences between Combinations of 2D and 3D Input and Output Devices for 3D Data Visualization. *International Journal of Human-Computer Studies*, 163:102820, July 2022. doi: 10.1016/j.ijhcs.2022.102820

[52] G. Woodin, B. Winter, and L. Padilla. Conceptual metaphor and graphical convention influence the interpretation of line graphs. *IEEE transactions on visualization and computer graphics*, 28(2):1209–1221, 2021.