# Small Data - Visualizing Simple Datasets for Communication and Decision Making

Pierre Dragicevic

<figure><img src="image_1.png" /></figure>

## To cite this version:

Pierre Dragicevic. Small Data - Visualizing Simple Datasets for Communication and Decision Making. Human-Computer Interaction [cs.HC]. Université de Bordeaux (UB), 2023. tel-04352699

HAL Id: tel-04352699

https://inria.hal.science/tel-04352699

Submitted on 19 Dec 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Small Data – Visualizing Simple Datasets for
Communication and Decision Making

Habilitation à Diriger les Recherches

Pierre Dragicevic

Defended on 23 June 2023

Version: v1.1 – 6 April 2023,
with annotations added on 19 Dec 2023.

**Draft document.** This is the pre-defense document sent to reviewers. The final dissertation that takes into account reviewers' comments will be uploaded on HAL as a new version.
<figure><img src="image_3.png" /></figure>
Université Bordeaux 1

Habilitation à Diriger les Recherches

Defended on 23 June 2023

Small Data – Visualizing Simple Datasets for
Communication and Decision Making

Pierre Dragicevic

**Reviewer** Priti Shah
Full Professor, Department of Psychology
University of Michigan

**Reviewer** Jessica Hullman
Associate Professor, McCormick School of Engineering
Northwestern University

**Reviewer** Jeffrey Heer
Full Professor, Dept. of Computer Science & Engineering
University of Washington

**Examiner** Hélène Sauzéon
Full Professor, Psychology
Université de Bordeaux

**Examiner** Guy Melançon
Full Professor, Computer Science
Université de Bordeaux

**Examiner** Pascal Guitton
Professor Emeritus, Computer Science
Université de Bordeaux
Pierre Dragicevic

Small Data – Visualizing Simple Datasets for Communication and Decision Making

Habilitation à Diriger les Recherches, Defended on 23 June 2023

Reviewers: Priti Shah, Jessica Hullman, and Jeffrey Heer

Examiners: Hélène Sauzéon, Guy Melançon, and Pascal Guitton

Université Bordeaux 1

École Doctorale de Mathématiques et d'Informatique
Abstract

This habilitation thesis presents research projects that explore how data visualizations can be used to communicate quantitative facts and support decisions. The thesis identifies the umbrella topic of “small data” as the common denominator for these projects, as they all involve datasets that are smaller than what is typically found in the information visualization literature. Although much of this literature focuses on large datasets, this thesis shows that we often do not know how to best visualize even small datasets, and that interesting research questions can also arise from such datasets.

The first part of this thesis focuses on supporting rational judgments and decisions with data visualizations. It examines whether data visualization can improve reasoning with base rates, it looks at whether a cognitive bias called the attraction effect transfers to data visualizations, and discusses how to generally evaluate data visualizations for decision making.

The second part of this thesis discusses how to support effective communication with data visualizations. It begins by exploring different ways in which researchers can communicate their data to their peers: first through tabular visualizations, and then through multiverse analyses. It then reports on two studies examining how to communicate data to large audiences: a replication study on whether trivial charts can hinder truthful communication, and a study on how to convey data on humanitarian issues.

Finally, a concluding chapter brings together several of the research problems discussed here by offering perspectives on how data visualization can support rational decision-making and effective communication on humanitarian issues. It discusses how can the effective altruism movement inform research in this area, and how may immersive displays be used to connect people to invisible or distant suffering.
<figure><img src="image_4.png" /></figure>
# Résumé

Cette thèse de HDR présente des projets de recherche qui explorent comment les visualisations de données peuvent être utilisées pour communiquer des faits quantitatifs et aider à la prise de décision. La thèse identifie le thème général des "petites données" comme le dénominateur commun de ces projets, car ils impliquent tous des jeux de données plus petits que ce que l'on trouve généralement dans la littérature sur la visualisation d'information. Bien qu'une grande partie de cette littérature se concentre sur les grands jeux de données, cette thèse montre que nous ne savons souvent pas comment visualiser même les petits jeux de données, et que des questions de recherche intéressantes peuvent également être soulevées à partir de ces types de données.

La première partie de cette thèse se concentre sur l'aide aux jugements et aux décisions rationnels à l'aide de visualisations de données. Elle examine si la visualisation des données peut améliorer le raisonnement avec les fréquences de base, si un biais cognitif appelé effet d'attraction se transfère aux visualisations de données, et comment évaluer de manière générale les visualisations de données pour la prise de décision.

La deuxième partie de cette thèse s'intéresse à comment communiquer les données de manière efficace. Elle commence par explorer les différentes façons dont les chercheurs peuvent communiquer leurs données à leurs pairs : d'abord par des visualisations tabulaires, puis par des analyses multivers. Il présente ensuite deux études examinant comment communiquer des données au grand public : une étude de réplication visant à déterminer si des graphiques triviaux peuvent entraver une communication sincère, et une étude sur la manière de communiquer des données sur des questions humanitaires.

Enfin, un chapitre de perspectives rassemble plusieurs des problèmes de recherche discutés dans cette thèse en offrant des perspectives sur la façon dont la visualisation des données peut soutenir la prise de décision rationnelle et la communication efficace sur les questions humanitaires. Il examine comment le mouvement de l'altruisme efficace peut informer la recherche dans ce domaine, et comment les affichages immersifs peuvent être utilisés pour connecter les individus à des souffrances invisibles ou lointaines.
<figure><img src="image_5.png" /></figure>
# Acknowledgements

Many thanks to my students and collaborators for their joint work on the research reported here. I co-authored this research with (in alphabetical order):

**Students:** Evanthia Dimara, Luana Micallef, Luiz Morais, Charles Perin, and Abhraneel Sarma.

**Collaborators:** Nazareno Andrade, Anastasia Bezerianos, Fanny Chevalier, Jean-Daniel Fekete, Steven Franconeri, Yvonne Jansen, Matthew Kay, and Catherine Plaisant.

Also many thanks to the members of my HDR committee (again in alphabetical order):

**Reviewers:** Jeffrey Heer, Jessica Hullman, and Priti Shah.

**Examiners:** Pascal Guitton, Guy Melançon, and Hélène Sauzéon.

I am grateful to the members of Aviz and Potioc for the many discussions and for their support.

<figure><img src="image_6.png" /></figure>
<figure><img src="image_7.png" /></figure>
# Contents

<table><tr><td>1</td><td>Introduction</td><td>1</td></tr><tr><td>1.1</td><td>Purpose and Scope</td><td>1</td></tr><tr><td>1.1.1</td><td>Habilitation Theses</td><td>1</td></tr><tr><td>1.1.2</td><td>Data Visualization</td><td>2</td></tr><tr><td>1.1.3</td><td>Purposes of Data Visualization</td><td>3</td></tr><tr><td>1.1.4</td><td>Small Data</td><td>3</td></tr><tr><td>1.1.5</td><td>Other Work</td><td>4</td></tr><tr><td>1.2</td><td>Thesis Outline</td><td>6</td></tr><tr><td>2</td><td>Supporting Rational Judgments and Decisions</td><td>9</td></tr><tr><td>2.1</td><td>Reasoning with Base Rates</td><td>9</td></tr><tr><td>2.1.1</td><td>Introduction</td><td>10</td></tr><tr><td>2.1.2</td><td>First Experiment: Testing Six Visualizations</td><td>13</td></tr><tr><td>2.1.3</td><td>Second Experiment: Tweaking the Text</td><td>15</td></tr><tr><td>2.1.4</td><td>Conclusions</td><td>16</td></tr><tr><td>2.2</td><td>A Cognitive Bias Called The Attraction Effect</td><td>17</td></tr><tr><td>2.2.1</td><td>Introduction</td><td>17</td></tr><tr><td>2.2.2</td><td>First Experiment: Two to Three Alternatives</td><td>18</td></tr><tr><td>2.2.3</td><td>Second Experiment: More Alternatives</td><td>21</td></tr><tr><td>2.2.4</td><td>Conclusions</td><td>22</td></tr><tr><td>2.3</td><td>Evaluating Visualizations for Decision Making</td><td>23</td></tr><tr><td>2.3.1</td><td>Evaluating Elementary Multidimensional Visualizations</td><td>23</td></tr><tr><td>2.3.2</td><td>Using Narratives in Crowdsourcing Studies</td><td>28</td></tr><tr><td>2.3.3</td><td>A Taxonomy of Cognitive Biases to Study</td><td>33</td></tr><tr><td>2.4</td><td>General Conclusions</td><td>37</td></tr><tr><td>3</td><td>Supporting Effective Communication</td><td>39</td></tr><tr><td>3.1</td><td>Communicating Data with Tabular Visualizations</td><td>39</td></tr><tr><td>3.1.1</td><td>Introduction</td><td>40</td></tr></table>
<table><tbody><tr><td>3.1.2</td><td>Two examples from Jacques Bertin</td><td>42</td></tr><tr><td>3.1.3</td><td>Bertifier Example: Survey of Applications</td><td>44</td></tr><tr><td>3.1.4</td><td>Bertifier Example: User Study Data</td><td>45</td></tr><tr><td>3.1.5</td><td>Conclusion</td><td>45</td></tr><tr><td>3.2</td><td>Communicating Results from Multiverse Analyses</td><td>46</td></tr><tr><td>3.2.1</td><td>Introduction</td><td>47</td></tr><tr><td>3.2.2</td><td>Example 1 – FREQUENTIST</td><td>49</td></tr><tr><td>3.2.3</td><td>Example 2 – DATAVERSE</td><td>51</td></tr><tr><td>3.2.4</td><td>Example 3 – DANCE</td><td>52</td></tr><tr><td>3.2.5</td><td>Limitations and Possible Objections to EMARs</td><td>54</td></tr><tr><td>3.2.6</td><td>Conclusion</td><td>55</td></tr><tr><td>3.3</td><td>Do Trivial Charts Hinder Truthful Communication?</td><td>56</td></tr><tr><td>3.3.1</td><td>Introduction</td><td>56</td></tr><tr><td>3.3.2</td><td>First Replication of BwS' Second Experiment</td><td>59</td></tr><tr><td>3.3.3</td><td>Summary of All Experiments</td><td>62</td></tr><tr><td>3.3.4</td><td>Conclusion</td><td>62</td></tr><tr><td>3.4</td><td>Communicating Humanitarian Data</td><td>63</td></tr><tr><td>3.4.1</td><td>Introduction</td><td>64</td></tr><tr><td>3.4.2</td><td>Quick Overview of the Design Space</td><td>65</td></tr><tr><td>3.4.3</td><td>Overview of Previous Studies</td><td>68</td></tr><tr><td>3.4.4</td><td>Our Own Study</td><td>71</td></tr><tr><td>3.4.5</td><td>Conclusion</td><td>74</td></tr><tr><td>4</td><td>Perspectives – Supporting Rational Decisions and Effective Communication on Humanitarian Issues</td><td>77</td></tr><tr><td>4.1</td><td>Introduction</td><td>77</td></tr><tr><td>4.2</td><td>What is Effective Altruism?</td><td>80</td></tr><tr><td>4.3</td><td>How Can EA Inform Visualization</td><td>81</td></tr><tr><td>4.4</td><td>Using Visualization and Psychology to Support EA</td><td>83</td></tr><tr><td>4.5</td><td>Combining Experiences with Quantitative Facts</td><td>84</td></tr><tr><td>4.6</td><td>Emerging Technologies</td><td>86</td></tr><tr><td>4.7</td><td>Conclusion</td><td>89</td></tr><tr><td>5</td><td>Conclusion</td><td>91</td></tr><tr><td colspan="2">Bibliography</td><td>95</td></tr></tbody></table>
1

Introduction

We hear about big data everywhere. I am often tempted to say that before we tackle
big data, we should first solve small data. In reality, I think we should be working on
both in parallel. Issues with big data – such as how to store, access and visualize large
databases – are very important, pressing, and deserve attention. However, there are other
data visualization problems that are arise independently of dataset size – such as how to
communicate data to general audiences, or how to help people make decisions based on
data. These are also very important and have not received so much attention.

This habilitation thesis summarizes research projects I was involved in which focus on using data visualizations to communicate quantitative facts and support personal data-driven decisions. While trying to find a common denominator for this stream of work, I realized that many of these projects use datasets that are much smaller than what is generally found in the information visualization literature – the smallest one consists of only two numbers. Maybe this is because I like to simplify problems in order to make them more tractable and easier to comprehend. In any case, I decided that *small data* could be the umbrella topic for my thesis. However, I never consciously decided to work on small data, or identified small data as a research direction in and of itself.

I will now lay out the scope of this thesis in more detail.

1.1 Purpose and Scope

1.1.1 Habilitation Theses

Many people outside France may not know what an habilitation is. In the French academic
system, the Habilitation à Diriger les Recherches (HDR) is a degree given to experienced
academic researchers that allows them to officially supervise PhD students and apply for
major job promotions. It is not a tenure - in France, academic researchers get tenure as
soon as they are hired (although this seems to be changing). Getting the HDR requires
that the researcher writes a document summarizing their past work or a subset thereof (the thesis) and defends it in front of a jury. Researchers can still – and do – advise PhD students before that, but their supervising role is not fully recognized by the University. Researchers typically go through the HDR process about five to ten years after being hired – sometime later (it took me 16 years), and some never do it.

1.1.2 Data Visualization

The term *data visualization* is ubiquitous but academic researchers tend to prefer *information visualization*. Although several definitions of information visualization have been proposed, an especially popular one is the one by Card, Mackinlay and Shneiderman in the introductory chapter of their seminal book [KMS99]:

"Information visualization: the use of computer-supported, interactive, visual representations of data to amplify cognition." [KMS99]

*Statistical graphics, scientific visualization and visual analytics are closely related academic terms. Although they historically refer to somewhat distinct lines of research, the distinctions are subtle [Mun08; Kei+08; Kos12], and all terms are compatible with the definition above. Therefore, my co-authors and I have sometimes used visualization as an umbrella term for all research areas involving computer-supported visualization of data [Jan+15; DJV20]. This is consistent with many authors using visualization as a shorthand for information visualization, scientific visualization, or data visualization.*

However, for people outside those academic areas, visualization as an isolated word can
be confusing. For example, a Google Scholar search of “visualization in psychology” yields
several articles about topics at the intersection of data visualization and psychology, but
also completely unrelated papers about mental imagery. Before you turn this page, can
you guess which of the following papers are about data visualization and which ones are
about mental imagery?

1. *The Psychology of Visualization* [Mar+93]

2. Research on Visualization in Learning and Teaching Mathematics: Emergence from Psychology [Pre06].

3. *A History of Visualization in Psychology and Science* [PNM10]
The answers are: 1) data visualization; 2) mental imagery; 3) mental imagery. Even researchers in human-computer interaction are confused, and many think that if what you're doing is highly visual, then it's visualization. Therefore, I came to prefer the term *data visualization*. It is also a good thing that this term is broadly used outside academic research, because there seems to be no reason to use different terms within and outside academia to refer to the same concepts.

### 1.1.3 Purposes of Data Visualization

Bremer and Munzner [BM13] listed three reasons why people use data visualizations:

1. *To present*: to communicate data to an audience;

2. *To discover*: to learn by exploring data;

3. *To enjoy*: to consume data for personal satisfaction and pleasure.

Dimara and Stasko [DS21] recently argued for adding a fourth category:

4. *To decide*: to use data to make decisions.

This fourth category is not fundamentally separate from the second one, as people often need to learn from data in order to make data-driven decisions. However, people can learn from data without using their newly acquired knowledge to drive decisions, and conversely, visualizations that are optimized for learning and understanding are not necessarily optimized for decision making [DS21].

Over the past decades of research in data visualization, there has been a considerable amount of research on supporting data-driven exploration and discovery (2. to discover), but comparatively less so on supporting the other motivations for using data visualizations. During my career, I have been especially interested in two of them: communication (1. to present) and decision making (4. to decide). I will summarize my contributions to those two lines of research in this thesis.

### 1.1.4 Small Data

As I already wrote, another way in which the work reported here differs from mainstream data visualization research is by its focus on small datasets. A lot of data visualization
research focuses on visualizing large and complex datasets (e.g., large networks and large
multidimensional datasets), not to mention addressing big data challenges involving
heterogeneous and rapidly-growing datasets [DGG15].

Again, this line of research is really important and the focus on large datasets is largely justified. However, it is often wrongly assumed that interesting research problems in information visualization only arise when datasets are becoming large. There are two versions of this claim; One is that we already know how to visualize small datasets – just use the standard charts available in any statistical package or spreadsheet software. Another version is that very small datasets do not need to be visualized at all. In our work on trivial charts [DJ18a] (which I will summarize in section 3.3), we quoted advice from visualizations experts that is consistent with this belief:

Tufte [Tuf01] deplores trivial charts and affirms that “tables are preferable to graphics for many small data sets”. For Duklan and Martin [DM02], “it makes no sense to encode only a few numbers into an overblown graphic”. For Kelly et al [KJW05], “graphs take up a lot of space if showing only a few data points. Hence they are best not used if there are only a few numbers to present.”. Gillan [Gil+98] similarly suggests that “few data points might best be presented in the body of the text” due to the cost of “processing the data display and integrating the information from the display and text”.

There no empirical evidence to support these recommendations, and I also think they
are misguided. Over my career I have collected several examples where visualizing a
single number is useful. I call them *single-datum visualizations* – see Figure 1.1. More
examples can be found in the article by Chevalier et al. [CVG13] on concrete scales and
at http://dataphys.org/list/tags/single-datum/.

1.1.5 Other Work

The research reported here covers about 25% of my work over the past 10 years, and about 15% in my whole research career. Other major topics I have worked on in the past 10 years are *data visualizations beyond the desktop* (i.e., visualizations that are physical, situated, immersive, or wall-sized), and *research transparency and statistical reporting*. Before that, I have worked on animated transitions and visual tracking, new interaction and visualization techniques, input devices and motor control, and user interface prototyping tools. I will not discuss these topics, except in the last chapter on future perspectives where I will mention beyond-desktop visualizations.
<figure><img src="image_8.png" /><figcaption>**Fig. 1.1:** Physical and graphical representations of single quantities: a) Bundles of wire representing the distance traveled by electricity in a nanosecond, and used by Grace Hopper to help programmers understand “*just what they’re throwing away when they throw away a millisecond*” [Plo21]; b) Number of plastic bottles purchased in a day across the world [SH19]; c) Hundreds of thousands of ceramic poppies representing WWI British military fatalities [Jan14]; d) Amount of Earth’s water [Sch19].</figcaption></figure>
## 1.2 Thesis Outline

This report primarily consists of extended summaries of some my past articles – generally, one article by section. At many places, I have reworked the explanations to make them easier to understand, and often added new explanations. I tried to be generous with figures. To make the thesis lighter, I don’t cover related work extensively, and so my contributions are sometimes not very well positioned with respect to the past literature. More comprehensive information about how each piece of work fits within the literature can be obtained by referring to the corresponding paper.

The thesis outline is as follows:

* Chapter 1. **Introduction**. The introduction you are reading now.

* Chapter 2. **Supporting Rational Judgments and Decisions**.

1. **Reasoning with Base Rates.** We look at whether data visualization can improve how we reason with base rates, such as when we need to estimate the probability of having a disease after having received a positive test [MDF12].

2. **A Cognitive Bias Called the Attraction Effect.** We look at whether the attraction effect – a bias whereby our choice between two options is influenced by an irrelevant (clearly undesirable) option – persists with data visualizations [DBD17c].

3. **Evaluating Visualizations for Decision Making.** How should we evaluate data visualizations’ support for judgment and decision making? We look at three different facets of this question [DBD17a; DBD17b; Dim+18].

* Chapter 3. **Supporting Effective Communication**.

1. **Communicating Data with Tabular Visualizations.** We look at a powerful and versatile method that was used decades ago by researchers to communicate data to their peers, and explain how we adapted it to modern technology [PDF14; PFD19].

2. **Communicating Results from Multiverse Analyses.** We look at the concept of interactive research paper where readers can change statistical analysis options and look at how fragile or robust the findings are [Dra+19].
3. **Do Trivial Charts Hinder Truthful Communication?** A 2016 study found that charts can persuade by their mere presence, simply because they look scientific. With a replication study, we look at whether this claim is justified [DJ18a].

4. **Communicating Humanitarian Data.** We look at different ways of communicating data about human tragedies that may help the audience relate to the victims, and whether they actually work [Mor+20; Mor+21].

* Chapter 4. **Perspectives — Supporting Rational Decisions and Effective Communication on Humanitarian Issues.** We look at avenues of research that target the goals of Chapters 2 and 3, with applications to the broad area of humanitarian visualization [Dra22a]. I discuss effective altruism and immersive displays, among other things.

* Chapter 5. **Conclusion.** We go through 17 key takeaways from this HDR thesis.
<figure><img src="image_9.png" /></figure>
# Supporting Rational Judgments and Decisions

Rationality is complicated and controversial. It is certainly wrong to think that people or their decisions can be easily categorized as rational or irrational. But I think it is also a mistake to reject the concept of rationality entirely, just because it causes a range of conceptual difficulties and can be abused. It is important that we as humans try to improve how we think, and rationality is a useful concept to that end. I will briefly get back to this issue in the conclusion of this chapter, and say more about difficulties with the concept of rationality. For now, I will refrain from even attempting to define rationality, and will instead directly dive into concrete examples.

This chapter covers two concrete examples looking at whether visualization can help people make better judgments and decisions. The first one involves reasoning with base rates, while the second one involves choosing between multiple options that have been set up in order to elicit a cognitive bias called the attraction effect (also known as the decoy effect). This chapter will conclude with a third section looking at different aspects to consider when evaluating visualizations for their ability to support rational judgment and decision making.

Almost all of the work reported here is by Evanthia Dimara, a PhD student I co-advised between 2014 and 2017. The coming section is work by Luana Micallef, a visiting PhD student I co-advised in 2011.

## 2.1 Reasoning with Base Rates

People have difficulty reasoning with probabilities and are unaware of their wrong judgments, particularly in problems involving base rates. This issue has been extensively studied in psychology, but rarely in a data visualization context. In this section, we look at whether visualizations can improve reasoning with base rates. This section is based on a 2012 paper authored by Luana Micallef, myself, and Jean-Daniel Fekete [MDF12].
Supplementary material (data, code, analyses) is available at https://osf.io/68ntk/ and https://aviz.fr/bayes.

2.1.1 Introduction

Both laymen and professionals have difficulty making inferences and decisions based on
uncertain information [CT96; Gig+07; Hof+00]. This can have severe consequences in
many domains. Physicians need to diagnose diseases based on the outcome of unreliable
medical tests. Patients need to decide whether they should undertake heavy medical
treatment. Wrong judgments are common and often result in overdiagnosis [WB10a],
e.g., up to two thirds of breast cancers detected by mammography can be overdiagnosed
[ZM04]. In other cases, patients with a positive HIV test result attempted or committed
suicide before further tests turned out negative [Chi93; Gig98; Sti96]. In this domain, a
crucial piece of information for effective decision making is the probability that a patient
has a disease given that a test is positive.

In legal trials, juries have to convict or acquit defendants based on unreliable evidence
and here too, wrong judgments abound [Koe97]. A respected professor and advisor to
defense lawyers claimed on U.S. television that since only 0.1% of wife batterers murder
their wives, evidence of battering should be ignored in murder trials [Goo95]. This
reasoning is however fallacious, since the only important information is the probability
that a husband was the murderer given that he battered his wife and she was killed.

All such scenarios involve Bayesian inference, more specifically, reasoning with base rates.
Base rates are known to be counter-intuitive and subject to fallacious reasoning. As a
concrete illustration, consider the following classic problem [Edd82]:

The probability that a woman at age 40 has breast cancer is 1%. According to the literature, the probability that the disease is detected by a mammography is 80%. The probability that the test misdetects the disease although the patient does not have it is 9.6%.

If a woman at age 40 is tested as positive, what is the probability that she indeed has breast cancer?

Out of 100 medical doctors, 95 estimated this probability to be between 70% and 80%, while the correct probability is only 7.8% [Edd82]. The probability is low because the
prevalence of the disease in the population, i.e., the *base rate*, is low. When making Baysian inference, this information is often ignored [GH95; Hof+00], thus leading to the so-called base rate fallacy [BS07a; BS07b]. Using natural frequencies<fnref n="1" /> instead of probabilities reduces the fallacy [CT96; GH95; HG04]. However, it is still difficult to comprehend how the different numerical quantities relate to each other<fnref n="2" />.

Previously proposed solutions to improve base-rate reasoning involve educational inter-
ventions, some of which use visual representations. For example, a study has suggested
that when Bayes' theorem is introduced to students through visualizations, students
learn better than without a visualization [SG01]. However, prior training is not always
possible. A few studies were conducted to assess the immediate benefits of visualiza-
tions, with mixed results. From them, we learn that two different but complementary
types of representations may help people reason with base rates: *Euler diagrams* can
convey critical information on the nested-set relations in such problems [GG01; Slo+03],
while *frequency visualizations* like icon arrays may facilitate probabilistic reasoning more
broadly [CT96]. In our work, we were interested in visualization designs that combine
both approaches, as in Figure 2.1.

The underlying intuition that motivated this study was that hybrid visualizations showing
both nested-set relations and frequencies must help people more than either design. We
only knew of one study prior to 2012 which evaluated such hybrid visualizations: the
study by Gary Brase [Bra09]. Brase's study did not find benefits over simpler diagrams,
but the visualization designs used in this study suffer from a number of issues, making
the results difficult to interpret (full discussion in [MDF12]). In addition, like most
other studies, Brase's study involved a population of university students who had to
participate in the study as part of their psychology course requirement, making it difficult
to generalize their findings to a more diverse population. In addition, only one medical
diagnosis problem was evaluated, making it difficult to generalize the findings to other
problems. In our study, we addressed most of these limitations.

<footnote n="1">¹Using *natural frequencies* would mean stating “10 out of every 1,000 women” instead of giving a 1% probability. This frequency format is said to be *natural* as the denominator corresponds to the number of actual observations.</footnote>

<footnote n="2">²In our example, 1% is the *base rate*, 80% is the *hit rate*, and 9.6% is the *false alarm rate*.</footnote>
<figure><img src="image_10.png" /><figcaption>**Fig. 2.1:** A hybrid Euler/frequency visualization of the classic mammography problem by David M. Eddy [Edd82], described in section 2.1. Each dot is a woman; Pink dots are women with breast cancer (unbeknownst to them) while blue dots are women without breast cancer. All women undergo a mammography. The dots with a black outline are women whose test came back positive, while all others were negative. As we saw in section 2.1, the test is such that 80% of women with breast cancer get a positive test (the black dots in the pink group). Meanwhile, among the women who do *not* have breast cancer, 9.6% get a positive test nonetheless (the black dots in the blue group). Now suppose you are a woman who just got a positive test. The only thing you learned is that you are somewhere in the black ellipse, and therefore, that the probability that you do have breast cancer is the proportion of pink dots in that group, i.e., 7.8%.</figcaption></figure>
2.1.2 First Experiment: Testing Six Visualizations

We designed a set of six visualizations for base rate problems (V1–V6) which include
Euler-based representations, frequency representations, and different combinations of
the two (Figure 2.2).

To test these visualizations, we recruited 168 participants in a crowdsourcing platform (24 per condition). Each of them saw three base rate problems used in prior work: the mammography problem [Edd82] we already discussed, as well as the cab problem [Bar80], and the choosing a course in economics problem [Ajz77]. The three problems were presented on separate pages, and used the same presentation format: either text alone (V0) or text followed by a visualization (V1–V6), depending on the participant. The ordering of the problems was randomized.

Our main dependent variables were *bias* and *error*. We operationalized *bias* as $log_{10}(p/p_e)$, with $p_e$ being the true probability and *p* the participant's estimated probability. To illustrate, in the mammography problem, if a participant answers that the probability of having breast cancer is 6 out of 10, then $p = 6/10 = 0.6$, and the bias is therefore $log_{10}(0.6/0.078) = 0.886$. The strictly positive value indicates an overestimated probability. In contrast, a negative value would have indicated an underestimation, while a value of zero corresponds to a perfectly correct answer. We operationalized *error* as the absolute value of bias, i.e., in our example, $|0.886| = 0.886$. This continuous measure of error stands in contrast with previous studies, which simply counted the proportion of perfectly correct answers, thus wasting statistical power and ignoring the extent to which people's answers are wrong.

Results for bias are in Figure 2.3. Each of the three columns show the distribution of biases for a specific base rate problem, for each of the visualization conditions (rows). Visually, the distributions differ across base rate problems but seem very similar across visualization conditions. In particular, there is no obvious sign of V1–V6 outperforming V0 (text only). This is confirmed by looking at errors: differences in mean errors are not statistically measurable, even after collapsing V1–V6 into a single condition.

Therefore, we failed to find a facilitating effect of visualization on base rate reasoning.
In addition, our participants' accuracy was remarkably lower compared to many of the
previous studies. For instance, considering V0 (text only), previous studies reported 35%
to 72% perfectly correct answers [Bra09; CT96; GH95; Slo+03; CT96], while we only
had 6% of them. We were not able to fully explain this discrepancy, but part of it might
<figure><img src="image_11.png" /><figcaption>**Fig. 2.2:** The six visualizations evaluated in our first experiment, showing the mammography problem [Edd82].</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>**Fig. 2.3:** Distributions of bias in answers per base rate problem and visualization condition ($N = 24$ each). Black bars are perfectly correct answers. A bias of -1 means an answer 10× lower and a bias of 1 means 10× higher.</figcaption></figure>
be due to our population being more diverse than in previous studies, which generally involve highly-focused university students.

Open-text comments from participants revealed that the lack of benefit of visualizations may be partly due to people ignoring them. Various participants commented that they did not fully understand the visualization, that having two representations was confusing, or that they ignored it and that the information provided in the text was sufficient. Some doubted the credibility of the visualization and chose to trust the text. This blind focus on the numbers seems to have led many participants to miscalculate and arrive at answers that are off by an order of magnitude.

### 2.1.3 Second Experiment: Tweaking the Text

We ran a second experiment to confirm with a larger sample that simply adding a visualization to a base-rate problem is of little help, and to investigate alternative solutions. Since the answer to the base-rate problem is in the visualization itself (see again Figure 2.1), it should be possible to increase the chances that people find it, either by i) helping them make the link between the text and the visualization [GG01]; ii) encouraging them to search for the solution in the visualization; or iii) forcing them to search for the solution in the visualization.

We tested again V0 and V4, and added two alternative presentation techniques that only involved moderate modifications to the text. In the condition V4a, we added short instructions in the text that refer to the visualization. For example, after the sentence “8 of every 10 women with breast cancer will get a positive mammography”, we added “(compare the red dots that have a black border with the total number of red dots)”. This was meant to support strategy i) and possibly ii). In the condition V4b, we removed all numerical quantities from the text. For example, the previous sentence became “A large proportion of women with breast cancer will get a positive mammography”. This was meant to support strategy iii), and was based on the notion that an imprecise estimation can be better than a plain wrong calculation.

We tested those four conditions with 480 new participants (120 per condition), using only the mammography problem. The results for bias are reported in Figure 2.4. The distributions are similar to those of the previous experiment, except for V4b which is closer to normal. Consistent with our expectations, V4b rarely yielded a perfectly correct answer (only 1 out of 120 – a participant who reported taking a screenshot and counting
Mammography Bias

<figure><img src="image_13.png" /><figcaption>**Fig. 2.4:** Distributions of biases in answers to the mammography problem (N = 120 each) per presentation type. Black bars are perfectly correct answers.</figcaption></figure>

the dots using MS Paint), but it yielded more accurate answers on average. The median error was 0.41, which is much lower than with V0, V4 and V4a (all 0.89). In addition, V4b yielded almost no average bias, whereas with other visual presentations subjects tended to underestimate probabilities.

2.1.4 Conclusions

In this study, we used crowdsourcing to assess the effect of six visualizations (based on
Euler diagrams, glyphs and combinations of both) on base-rate reasoning using three
classic problems in psychology. To our surprise, participants’ accuracy was remarkably
low and did not substantially improve when a visualization was provided with the text. A
follow-up experiment with higher sample size confirmed that simply adding a visualization
to a textual base-rate problem is of little help for crowdsourcing participants. It however
revealed that removing all numbers from the text in order to force participants to perform
visual estimation with the visualization substantially reduces probability estimation errors.
Thus, novel representations that holistically combine text and visualizations in a way that
promotes the use of visual estimation rather than numerical calculation deserve to be
investigated.
<figure><img src="image_14.png" /><figcaption>**Fig. 2.5:** Illustration of the attraction effect in a hypothetical election: Bob has an excellent education plan, while Alice is very strong in crime control. The addition of Eve, a candidate similar but slightly inferior to Alice, raises Alice's attractiveness as a candidate. This irrelevant option is called a decoy.</figcaption></figure>

## 2.2 A Cognitive Bias Called The Attraction Effect

The attraction effect is a well-studied cognitive bias in decision making research, where people's choice between two alternatives is influenced by the presence of an irrelevant (dominated) third alternative. In this section, we examine whether this cognitive bias, so far only tested with three alternatives and simple presentation formats such as numerical tables, text and pictures, also appears in visualizations. This section is based on a 2016 paper authored by Evanthia Dimara, Anastasia Bezerianos, and myself [DBD17c]. Supplementary material (data, code, analyses) is available at https://osf.io/kfhme/ and https://aviz.fr/decoy.

### 2.2.1 Introduction

Suppose you are voting for primary elections and need to choose between candidates **Bob** and **Alice** (Figure 2.5). Bob has a solid education plan, while Alice has an excellent strategy for crime control. There is no obviously wrong choice: it all depends on how much you care about education and safety. But now suppose there is a third candidate, **Eve**. Like Alice, Eve is stronger on crime control, but her crime control plan is not as good as Alice's. Should her presence influence your decision? Normally not, as Eve is inferior in all respects and therefore irrelevant to your choice [HPP82]. And yet countless studies have shown that adding Eve make people more likely to chose Alice in similar decision-making tasks [OP95].
This shift in preference is called the *attraction effect* (also known as the *decoy effect* and the *asymmetric dominance effect*). Studies suggest the attraction effect is quite general and robust, e.g., it occurs when people choose consumer products like beers, cars, or films [HPP82], when they gamble [Wed91], select candidates to hire [Hig96], choose a meal in a menu [HJW12], decide which suspect committed a crime [Tru12], or vote [OP95]. Even animals like hummingbirds [BHH02], bees [SWS02], and amoebae [LB11] appear to be subject to the attraction effect when selecting their food.

Like other cognitive biases, the attraction effect leads to suboptimal decisions that can have important implications in many areas such as politics and advertising. Our goal in this study was to find out whether the attraction effect also has implications for information visualization design. If someone uses a visualization to choose among several alternatives (e.g., when buying an apartment [WS92]), will the presence of inferior alternatives affect their decision? In other words, does the attraction effect transfer to visualizations?

### 2.2.2 First Experiment: Two to Three Alternatives

The purpose of our first experiment was to replicate the design of a standard attraction effect experiment (two alternatives plus an optional decoy, all presented in a numerical table), and test if the effect persists when alternatives are shown using a scatterplot visualization.

We reproduced the first experiment by Malkoc et al. [MHH13], which is representative of a classical attraction effect experiment. In this experiment, participants were presented a fictional scenario where they had to pick a fitness club. Each fitness club was defined by its variety and its cleanliness, both rated from -10 to +10. Some participants were given the choice between two fitness clubs (equivalent to showing only Bob and Alice in Figure 2.5), while others had to choose among three fitness clubs, one of which was a decoy (equivalent to choosing between Bob, Alice, and Eve). The attraction effect is measured by looking at the proportion of people choosing Alice when the decoy is present vs. absent. As many studies before, Malkoc et al.'s experiment used a numerical table to convey the different alternatives and their attributes (see top of Figure 2.6). We did the same, but added conditions where the same information was conveyed with scatterplots (bottom of Figure 2.6).
<figure><img src="image_15.png" /><figcaption>**Fig. 2.6:** Examples of experimental stimuli for the table (a,b) and the scatterplot (c,d) conditions. The left decision task (a,c) has no decoy, while the right decision task (b,d) includes a decoy on the *variety* attribute.</figcaption></figure>

In Figure 2.6-right, the decoy C is similar to (and dominated by) B. Because it is expected to make B more attractive, B is commonly referred to as *the target*. In addition, because C presumably makes B more attractive by making variety look more important as an attribute, it is conventionally referred to as a *decoy on variety*. Some participants saw a *decoy on cleanliness* instead, with A being the target (stimuli not shown on the figure); We will call this other decoy D. To balance out possible layout effects, we varied the ordering of the alternatives and the attributes in the table (an improvement over Malkoc et al.), as well as the mapping of attributes to the *x* and *y* axes of the scatterplot. In total, we used 18 stimuli, which we randomly assigned to participants (each participant only saw one stimulus). However, we only considered two independent variables in our analysis: the *decision task*, defined by the set of gyms to choose from: {A, B} (no decoy), {A, B, C} (decoy on variety), or {A, B, D} (decoy on cleanliness); and the *presentation format*: table or scatterplot.
<figure><img src="image_16.png" /><figcaption>**Fig. 2.7:** Point estimates and 95% confidence intervals for the attraction effects in Malkoc et al. [MHH13], and in our two experiments.</figcaption></figure>

We collected data from 305 participants in a crowdsourcing platform. The results are in Figure 2.7, in the middle area labeled *Gyms*. The top area labeled *Malkoc '13* reports the results from the original experiment, for comparison. The bottom area labeled *Bets* will be discussed in the next section. Back to the *Gyms* area, the top three error bars report are results for numerical tables; The top error bar indicates the difference in the percentage of participants who chose the target when there was a decoy on cleanliness vs. no decoy, with its 95% confidence interval. A positive value (to the right of the dashed line) indicates an attraction effect. Since the CI clearly crosses 0%, the result is inconclusive.<fnref n="3" /> The next error bar reports the results for the decoy on variety: here, there is evidence of an attraction effect. Finally, the purple error bar reports the *combined attraction effect*, which directly compares the two decoy conditions with each other. There is again some evidence of an attraction effect. The next three rows show the same results, but for the scatterplot condition. Again, we only see evidence for one of the two decision tasks, but when we combine both, we see evidence of an attraction effect.

Overall, we replicated the attraction effect that has been previously observed with numerical tables (although our effect is smaller than in Malkoc et al. [MHH13]), and we also found evidence of an attraction effect with scatterplots. However, using a scatterplot to visualize only three alternatives does not capture most situations where visualizations are used for decision support.

<footnote n="3">³I do not report any p-value, here or anywhere in this report, for reasons I explain in [Dra16]. I draw my inferences from confidence intervals, which convey the same information as p-values, and more.</footnote>
### 2.2.3 Second Experiment: More Alternatives

In this second experiment, we looked at whether the attraction effect persists in more realistic situations where more than two or three alternatives are visualized.<fnref n="4" /> We also attempted to increase statistical power in order to get stronger evidence.

To increase statistical power, we used a within-subjects design. Such designs are less common in previous studies but have been occasionally used. Wedell [Wed91], for example, measured a clear attraction effect with numerical tables using a within-subjects procedure, where participants were given multiple decision tasks. He further increased statistical power by excluding no-decoy conditions and only measuring the combined decoy effect. We therefore decided to replicate Wedell's design.

As in Wedell's experiment, our tasks involved choosing among three lottery tickets, each defined by two attributes: the *probability* of winning, and the amount that can be won (the *prize*). Participants were presented with twenty such decision tasks in sequence. Contrary to Wedell, our experiment was incentivized for higher external validity: upon completion of all tasks, the twenty lotteries were ran by a computer and the results were used to determine a bonus reward for each participant.

We only tested scatterplots. Figure 2.8 shows two of our experimental stimuli (floating boxes added by us). In contrast to Wedell and to our previous experiment, the number of alternatives to choose from was substantially higher than three. On the left plot, the participant is asked to choose among approximately twenty lottery tickets defined by their probability to win (x-axis) and their prize (y-axis). Two tickets dominate others in terms of their expected value (A and C), so participants were expected to choose either one of them. However, both A and C have the same expected value so there is no right answer, just as with A and B in Figure 2.6 or Bob and Alice in Figure 2.5. The other dots are either decoys dominated by A or other dominated alternatives we call distractors. This stimulus was paired with another stimulus presented at another time during the sequence of 20 decision tasks (Figure 2.8-right), and where decoys were dominated by C. If participants tend to pick ticket A more often with the left stimulus than with the right one, this is evidence for an attraction effect.

To make it more difficult for participants to infer patterns in the sequence of decision tasks, we inserted irrelevant decision tasks at various positions, which were not used in

<footnote n="4">⁴Prior to this experiment, we ran another (inconclusive) experiment that is reported in full in a research report [DBD16].</footnote>
<figure><img src="image_17.png" /><figcaption>**Fig. 2.8:** Experimental stimuli for the two matched decision tasks *AC* and *CA* (black-and-white background images), and explanatory annotations (box overlays). Each dot is a lottery ticket.</figcaption></figure>

our analyses. These tasks differed in that they had either one or three non-dominated alternatives (instead of two), and they did not exhibit an imbalance in the number of asymmetrically dominated alternatives.

We used an aggregate measure of attraction effect that uses participant's responses to all twenty stimuli (10 paired stimuli). The result is reported at the bottom of Figure 2.7, in the area labeled *Bets*. Although the effect size is likely not larger than in the previous experiment, this time the statistical evidence is remarkably strong.

### 2.2.4 Conclusions

In this study, we showed that the attraction effect generalizes to data visualizations. In other words, when people use scatterplots to visualize multiple alternatives to choose from, the number and position of irrelevant (dominated) alternatives may influence their choice. This shift in preferences violates basic axioms of rational choice theory [HPP82]. In addition to being the first study on the attraction effect in information visualization, our study contributes to ongoing debates in decision-making research on whether the effect generalizes to non-numerical formats [FLB14; HPP14; Sim14; YL14], and whether it should disappear as more alternatives are added [BLP98]. However, more realistic
datasets remain to be tested, and our inconclusive results with real datasets (reported separately [DBD16]) suggest that the effects with real-world datasets may be small and hard to detect.

## 2.3 Evaluating Visualizations for Decision Making

So far we have summarized two studies addressing specific questions related to judgment and decision making with visualizations. But how should we evaluate visualizations for judgment and decision making more generally? Although this is a vast question, this section summarizes three contributions that bring us slightly closer to a response. The first one looks at what to measure when we want to compare elementary multidimensional visualizations for their ability to support multi-attribute choice tasks. The second one looks at whether or not we should use narratives in task instructions when we evaluate visualizations, both for analytical tasks and decision tasks. Finally, the third contribution helps us identify which cognitive biases could be interesting to study in a data visualization context.

### 2.3.1 Evaluating Elementary Multidimensional Visualizations

This subsection is based on a 2017 paper authored by Evanthia Dimara, Anastasia Bezerianos, and myself [DBD17a]. Supplementary material (data, code, analyses) is available at https://aviz.fr/dm.

Suppose we want to book a hotel, and we have access to a dataset with about one hundred hotels with a dozen attributes such as price, room size, bed size, or user ratings. We refer to tasks like these as *multi-attribute choice tasks*: tasks that consists of finding the best alternative among a fixed set, where alternatives are characterized across several attributes with known values, but where the objective function is unknown to the system (it is inside the user's head). What are the best visualizations for tasks like these?

On the one hand, a dataset like the hotel dataset is trivial to visualize: it is well-structured (a table with columns having a well-defined type), it is fairly small in visualization standards, and there is no missing or uncertain data. Many general-purpose visualization techniques exist that can visualize such a dataset, including scatterplot matrices, parallel coordinates, and tabular visualizations [KK96; OL03; PDF14]. There are also visualization
systems specifically designed for multi-attribute choice tasks, many of which use general-
purpose visualizations with extra features [Paj+17; CL04; Gra+13].

But surprisingly, there is very little empirical data to help us decide which tool to use.
Most existing studies are either qualitative studies without a comparison baseline [Yi+05;
Gra+13; ATS95; Paj+17; PF00; AAJ03; AA03], or use elementary analytic tasks such as
value retrieval [Kua+12; WS92] or correlation estimation [LMV10; Yi+05]. Low-level
analytic tasks like these are important but they are not decision tasks. How do we
evaluate visualization tools for their ability to support multi-attribute choice?

To answer this, we identified a range of subjective and objective metrics and used them to compare three elementary multidimensional visualizations in a within-subject study (see Figure 2.9). We first explained the techniques to the participants and gave them low-level analytic tasks (value retrieval, range, correlation). Such tasks made sure that participants were fully trained and allowed us to ensure that they properly understood the techniques when they performed the choice tasks, thus eliminating potential confounds (e.g., a technique yielding poor decisions because participants did not know how to use it). A dual evaluation may also uncover potentially interesting interactions between a technique's ability to support analytic tasks and its ability to support decision tasks.

After performing all analytic tasks with all techniques, participants were told they would now use the techniques to make a personal choice. They were asked to imagine planning their vacations and looking for the ideal holiday package. We used three synthetically generated datasets containing 100 data cases each (holiday packages) and 8 dimensions: price per person (euro/day), hotel quality, archaeological interest, landscape interest, night life interest, security level, sport activity level, and kids friendly. Except for price, all dimensions were ratings from 0 to 100. The meaning of each attribute was explained to the participants. Participants performed three such decision tasks, each time with a different technique and a different dataset (fixed ordering for datasets, counterbalanced ordering for techniques).

There is no simple way to define the accuracy of a multi-attribute choice task, given its
subjective nature. Although Pareto dominance is one such measure [DBD17c], partici-
pants are unlikely to select a dominated alternative given the number of alternatives and
attributes. We thus decided to use as an indicative measure of accuracy the *consistency*
*between the choice made by a participant and their self-reported choice criteria.*
<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /><figcaption>**Fig. 2.9:** The three visualizations we evaluated: parallel coordinates (PC), scatterplot matrix (SM), and tabular visualization (TV). All three visualizations supported basic interactions for highlighting individual data cases, selecting ranges of attribute values, and reordering dimensions. Here we see a training dataset of country indicators, with 29 data cases and 6 dimensions. The datasets used in the experimental tasks (both analytic and decision) had 100 data cases and 8 dimensions.</figcaption></figure>
Thus before the first decision task, participants were asked to rate how *important* each of the 8 holiday package attributes was to them, on a scale from 0 to 10. They were also asked to indicate the *direction* of their preference, i.e., whether they prefer the attribute to be high or low. For example, a holiday package with lots of physical activity can be perceived as desirable by an athletic person but undesirable by someone with limited mobility. As preferences may evolve during the session, the questionnaire was administered before and after each decision task (4× total).

We used participants' preference ratings to roughly estimate how desirable each alternative *should be* using a normalized weighted sum approach [Tri13], resulting in a desirability score of $d=1$ for the “best” alternative in the dataset and $d=0$ for the worst alternative (details in our paper [DBD17a]). Desirability scores can be computed using the preferences elicited either before the decision task ($d_{pre}$), or after the task ($d_{post}$). Since preferences can evolve while exploring options, $d_{post}$ may seem more indicative of the “true” desirability. However, a participant may also update their preferences *after* the choice was made, e.g., as a way of rationalizing their choice. Thus, we consider both $d_{pre}$ and $d_{post}$ and define the *accuracy* of a decision task as $a = \max(d_{pre}, d_{post})$, with $d_{pre}$ and $d_{post}$ being the desirability scores of the chosen alternative.

This decision accuracy score is an approximation and is not meant to capture decision quality perfectly. The elicited criteria may not be completely reliable, and cannot fully capture the complexity of real choice criteria (i.e., someone may want an attribute to be neither too high nor too low). However, if a visualization happens to be misleading or particularly hard to use, we should expect participants to make choices that are clearly inconsistent with their criteria, thus yielding an abnormally low decision accuracy.

After each decision, participants were also asked to evaluate the choice they just made on four subjective dimensions: *satisfaction* with their choice, *confidence* in their choice, the *easiness* with which they made their choice, and their *attachment* to their choice (measured by asking them if they would switch to the choice proposed by a hypothetical recommender system). Finally, at the end of the experiment, participants were asked which technique they prefer overall for a) analytic tasks, and b) decision tasks. Participants were timed on all tasks.

We ran a lab experiment with 21 volunteers having different levels of reported expertise in data visualization (range 2–9 on a 0–10 scale, average 6). The experiment lasted 1 hour and a half on average. As we analyze multiple dependent variables, we are subject to statistical multiplicity issues and any finding should be taken as tentative. With that
said, our paper explicitly distinguishes between planned and unplanned analyses, and so
in contrast to typical exploratory analyses, there is no hidden multiplicity [Ber07].

Overall, we found the three visualizations to be comparable on most metrics, with perhaps a slight advantage for tabular visualizations. More specifically, all three techniques yielded close-to-perfect accuracy in analytic tasks, confirming that participants were sufficiently trained. There were however large differences in completion times: the scatterplot matrix was slowest for value retrieval and range tasks, but by far the fastest in correlation tasks. This result matches previous intuitions and findings on visualization for analytic tasks.

In terms of actual decision-making tasks, we found our techniques to be comparable across most metrics, with a slight speed advantage for the tabular visualization. Participants also preferred the tabular visualization over the scatterplot matrix overall. However, they reported being more attached to choices they made with the scatterplot matrix on average, a result that needs to be confirmed by further studies. If the effect is real, one explanation could be that the scatterplot matrix's better support for overview tasks (confirmed by our results with the correlation task) made participants more confident that they did not miss a particularly interesting alternative.

Our accuracy metrics for decision support showed a much larger variability in responses than analytic tasks. Perhaps as a result, many of our metrics were not sensitive enough to capture differences across conditions that likely exist [Coh94]. Additional work is needed on establishing more sensitive metrics of choice quality. Nevertheless, it is interesting that tabular visualizations allowed participants to reach decisions of similar accuracy faster. Thus, although decision time is rarely central when assessing decision support, it could be used as a tie-breaker when visualizations achieve similar decision accuracy.

Again, all these findings are tentative and need to be confirmed in separate experiments. In the meantime, with the data we have so far, tabular visualizations seem to be a compelling choice for supporting multi-attribute choice tasks, despite the low attention they have received in the literature on multidimensional visualization [PDF14]. Nevertheless, several decision-support visualization tools use a tabular layout [CL04; Gra+13; Paj+17], and our experiment provides preliminary empirical justification for this design choice.
### 2.3.2 Using Narratives in Crowdsourcing Studies

This subsection is based on a 2017 paper authored by Evanthia Dimara, Anastasia Bezerianos, and myself [DBD17b]. Supplementary material (data, code, analyses) is available at https://aviz.fr/narratives.

Crowdsourcing enables rapid evaluation of visualizations [HB10; KCS08; Bou+12; Boy+14], including for judgment and decision making tasks [MDF12; DBD17c]. However, engaging contributors and obtaining high-quality responses is hard [Hul11; ECD14]. To address this, visualization studies often use task instructions with narrative elements in order to engage participants, help simulate the real use of a system, and elicit a more representative user behavior. A minimalist form of narrative consists in attributing a meaning to the datasets – for example, to evaluate HomeFinder, Williamson et al. [WS92] used questions such as “*what neighborhood has the most expensive houses?*”. Narratives can also invite participants to imagine themselves in a hypothetical decision situation. For example, Yi et al. [Yi+05] evaluated their Dust & Magnet system by asking participants to choose a cereal brand.

Unsurprisingly, hypothetical decision scenarios are common when evaluating visualization tools for decision support. For example, Aseniero et al. [Ase+15] evaluated a tool designed for software release plans by instructing participants to imagine taking the role of a project manager identifying an optimal plan. In order to evaluate a tool for preferential choices, Bautista and Carenini [BC08] immersed participants in shopping scenarios and put them in a situation of finding a hotel. Similarly, Daradkeh et al. [DCM13] asked participants to make hypothetical investments. Finally, the very study presented in the previous subsection used a hypothetical scenario where participants had to choose a holiday package.

Narratives are commonly used when evaluating visualizations, but the real effects of adding narrative elements to task instructions are unclear, in particular in crowdsourcing settings where incentives vary across people, and attention and motivation are hard to control for. For example, what would be the difference between instructing participants to identify the data point with the minimum X value, and instructing them to imagine they are trying to find the cheapest available house? Both versions are equivalent at the task level and consist of finding an extremum on a particular data dimension. The second version is possibly more salient and engaging, and with a context that is easy to understand, characteristics linked to good crowdsourcing performance [Hul11; MW10].
<figure><img src="image_21.png" /><figcaption>**Fig. 2.10:** The three tasks used in our experiment, with correct answers in blue (annotations ours). Plot labels depended on the condition – for example, scatterplot axes were labeled [X, Y] for ABS, and [size (m²), price ($)]] in the narrative conditions. In the Ext and Com tasks, the scatterplots supported basic interactions: hovering over a data point highlighted it, displayed horizontal and vertical projection lines, and overlaid the data point’s X and Y values on the axes.</figcaption></figure>

However, the first version is more succinct and less demanding in terms of time and patience, aspects that have also been emphasized in crowdsourcing guidelines [ECD14].

To help address this knowledge gap, we ran a study to find out if crowdsourcing participants perform differently in simple visualization tasks when task instructions are enhanced with narrative components. We gave them three basic visualization tasks inspired from previous task taxonomies [AES05; WL90; RM90], that were analytical in nature but could be mapped to decision tasks:

* An **extremum** task (Ext), where participants had to find the data point with highest value according to the *X* dimension (leftmost scatterplot in Figure 2.10).

* A **correlation** task (Cor), where participants had to find the scatterplot with the highest correlation among four different ones (second panel in Figure 2.10).

* A **comparison** task (Com), where participants had to compare data points across their two dimensions simultaneously (third panel in Figure 2.10). The task required finding a data point without any “competitor”, a competitor being defined as a data point that has both larger *X* and smaller *Y*. There were four possible correct answers.

All participants did all three tasks in sequence, but saw different instructions depending on the condition they were randomly assigned to. Each task came with five different sets of instructions, broken down into two pages (see Table 2.1 for an overview of all textual instructions):
<table><caption>Tab. 2.1: The text instructions in each experiment condition. The condition O-NAR (not listed here) had the same page 1 as AN-NAR, and the same page 2 as SEM.</caption><thead><tr><th>Condition</th><th>Task</th><th>Page 1</th><th>Page 2</th></tr></thead><tbody><tr><td rowspan="3">ABS</td><td>Ext</td><td>You will be asked to answer a few questions about data. In the next page you will see many data points displayed in a diagram.</td><td>Which is the data point with the largest value of X?</td></tr><tr><td>Cor</td><td>Now you will see four diagrams with data points. You will be asked to compare them.</td><td>In which of these four diagrams is Y most related to X?</td></tr><tr><td>Com</td><td>Now you will see one of the previous diagrams again. You will be asked a question that requires identifying "competitors". In our case, a data point is a competitor of another data point if it has both larger X and smaller Y.</td><td>Select a data point that has no competitor.</td></tr><tr><td rowspan="3">SEM</td><td>Ext</td><td>You will be asked to answer a few questions about houses. In the next page you will see many houses displayed in a diagram.</td><td>Which is the biggest house?</td></tr><tr><td>Cor</td><td>Now you will see four diagrams with houses. You will be asked to compare them.</td><td>In which of these four diagrams is price most related to size?</td></tr><tr><td>Com</td><td>Now you will see one of the previous diagrams again. You will be asked a question that requires identifying "competitors". A house is a competitor of another house if it is both bigger and cheaper.</td><td>Select a house that has no competitor.</td></tr><tr><td rowspan="3">AN-NAR</td><td>Ext</td><td>You will be asked to answer a few questions about houses. Imagine that you are a real estate analyst and you need to understand the house market. You focus on extremely rich customers who seek to buy a house that is as big as possible. In the next page you will see the houses currently on the market, displayed in a diagram.</td><td>Given what you read, which house would be the most attractive to your customers?</td></tr><tr><td>Cor</td><td>Now you want to focus on regular customers who are not necessarily very rich. You want to investigate how reliable some real estate agencies are. You will see four diagrams with houses. Each diagram shows the houses proposed by a different agency. An agency that sets arbitrary prices is NOT reliable. While in a reliable agency, price is very related to size.</td><td>Given what you read, which of these four real estate agencies is the most reliable?</td></tr><tr><td>Com</td><td>Now you will see one of the previous diagrams again. It shows the houses offered by the best agency. You need to report on their best deals. A good deal is a house that has no "competitor". A house is a competitor of another house if it is both bigger and cheaper.</td><td>Given what you read, select a house that is a good deal.</td></tr><tr><td rowspan="3">DM-NAR</td><td>Ext</td><td>You will be asked to make a few decisions about houses. Imagine you are moving to a new city and you need to buy a house. You are extremely rich and you want your house to be as big as possible. In the next page you will see the houses currently on the market, displayed in a diagram.</td><td>Given what you read, which house would you buy?</td></tr><tr><td>Cor</td><td>You don't have as much money as you initially thought. So before buying a house, you need to find a reliable real estate agency. You will see four diagrams with houses. Each diagram shows the houses proposed by a different agency. An agency that sets arbitrary prices is NOT reliable. While in a reliable agency, price is very related to size.</td><td>Given what you read, which of these four real estate agencies would you choose?</td></tr><tr><td>Com</td><td>Now you will see one of the previous diagrams again. It shows the houses offered by the best agency. You will finally get to choose your house. A good choice is a house that has no "competitor". A house is a competitor of another house if it is both bigger and cheaper.</td><td>Given what you read, which house would you buy?</td></tr></tbody></table>
* In the **abstract** (ABS) condition, the dataset had no specific meaning. Both page 1 and page 2 used abstract wordings. An example of question was "*Which is the data point with the largest value of X?*".

* In the **simple semantics** (SEM) condition, data points were houses and dimensions were price and size. Questions were of the type "*Which is the biggest house?*".

* In the **analytic narrative** (AN-NAR) condition, page 1 contained a narrative that asked participants to put themselves in the situation of a real estate analyst and to find answers to analytical questions. An example of question on page 2 was "*Given what you read, which house would be the most attractive to your customers?*".

* In the **decision-making narrative** (DM-NAR) condition, page 1 had a narrative that asked participants to put themselves in the situation of a house buyer and, given some criteria and constraints, to make choices. Questions were of the type "*Given what you read, which house would you buy?*".

* In the **optional narrative** (O-NAR) condition, page 1 was identical to AN-NAR and page 2 was identical to SEM. An example of question was "*Which is the biggest house?*". In contrast with AN-NAR, participants did not have to read the narrative on page 1 to be able to answer the question.

To assess the merits of the different instructions, we used both objective performance metrics and self-reported impressions. In terms of objective metrics, we used a normalized measure of *accuracy* ranging from 0 to 1 for each of the three tasks. To see if task instructions affect the degree to which participants pay attention to the tasks, we also measured *in-task attention* by testing their ability to recall the options presented to them in the correlation task, after all tasks were completed. To further assess attention fatigue, we measured participant's *post-task attention* through an instructional manipulation check [OMD09] at the end of the experiment. As for the subjective metrics, we asked participants to report their *confidence* in their answer and the *easiness* with which they arrived at their answer for each of the three tasks, and after all three tasks were completed, the overall *enjoyability* of the experiment and the *usefulness* of the visualizations for completing the tasks.

We recruited 405 crowdsourcing workers (about 80 per condition). As with the previous study, the strength of our findings is somehow weakened by the use of multiple dependent variables, but there is no hidden statistical multiplicity in our planned analyses.

We expected that providing task context in the form of data semantics or backstory narratives would improve the overall quality of responses. However, adding semantics to the data (i.e., using SEM instead of ABS) did not seem to help much, and some of
the narratives we used might have even harmed performance. In particular, it appears
that participants who were given the decision-making narrative (DM-NAR) performed
less accurately on average than those who were only given minimal context (SEM) or no
context at all (ABS). As for in-task attention, participants exhibited a better recall of the
correlation task when given minimal semantics (SEM) than no context (ABS), suggesting
they were paying more attention. However, adding a backstory narrative on top of that
(AN-NAR or DM-NAR) decreased their recall, bringing it to the same level as ABS. For
post-task attention, the results were mostly inconclusive.

Why do narratives sometimes seem to harm performance? As we mentioned, crowdworkers generally appreciate succinct instructions [ECD14]. An otherwise simple task can appear more demanding in attention and time if it requires reading a long (in crowdsourcing standards) piece of text beforehand. Also, experienced contributors are generally used to performing abstract and mechanical tasks since these abound on crowdsourcing platforms. The fairly good performances we observed for ABS do suggest contributors were overall able to understand the context-less tasks and willing to carry them out.

Despite our results on performance, we have strong evidence that adding data semantics
improves subjective experience on a range of metrics (confidence, perceived easiness,
enjoyability, and perceived usefulness of the visualization). With that said, our backstory
narratives did not yield measurable subjective benefits compared to data semantics
alone (SEM). Thus, even though crowdsourcing contributors appreciate working with
meaningful data, they may not be particularly interested in more elaborate narratives
and may prefer to focus on carrying out their task.

Overall, our study provides compelling reasons for incorporating data semantics in
crowdsourced evaluations of visualizations, i.e., stating what the datasets and their
dimensions mean. However, it also uncovered what could be termed a “double-edged
sword effect” of narratives – thus it seems safer to use elaborate narratives parsimoniously,
unless there are clear reasons to do so. When evaluating domain-specific and decision-
support visualization systems, avoiding narratives is difficult; For example, if we want
to evaluate a system meant to help customers choose a car, plain instructions such as
“select the best car” may lack external validity, and it may seem more suitable to provide a
backstory narrative to help participants imagine a real car purchase situation. Our study
suggests that such narratives will lead to lower performance, but if efforts are made to
keep the narrative short, the decrease should be relatively small and an acceptable price
to pay for better external validity. Much more empirical work is needed to find out how
to best design narratives, a question this study does not address.
### 2.3.3 A Taxonomy of Cognitive Biases to Study

Despite a growing interest in cognitive biases in information visualization, studies remain rare. Meanwhile, most experimental tasks studied in the cognitive bias research use textual representations. In order to bridge this gap between cognitive psychology and visualization research, we did an extensive review and analysis of the cognitive bias literature and derived a taxonomy of cognitive biases targeted to information visualization researchers. In this taxonomy of 156 cognitive biases, biases are classified by user task, instead of by proposals for psychological explanations of why biases occur, as is commonly the case for cognitive bias taxonomies.

This subsection is based on a 2018 paper by Evanthia Dimara, Steven Franconeri, Catherine Plaisant, Anastasia Bezerianos, and myself [Dim+18].

Our taxonomy is reproduced in Table 2.3. Each row lists a particular cognitive bias. Biases are organized into six different types of tasks, plus a category “other”. These tasks correspond to experimental tasks used in cognitive bias experiments, but they also map to real-world judgment and decision tasks:

* **Estimation:** in estimation tasks, people are asked to assess the value of a quantity. For example, a person may need to estimate the likelihood of theft to decide whether to insure their car, or their future financial needs in order to choose a retirement plan.

* **Decision:** decision tasks are any task involving the selection of one over several alternative options – for example, choosing a car to buy or a university to apply to.

* **Hypothesis assessment:** this category refers to any task involving an investigation of whether one or more propositions are true or false.

* **Causal attribution:** this category refers to any task involving an assessment of causality. For example, someone may need to decide whether a person had a car accident because the road was in bad condition, or because that person is not a good driver.

* **Recall:** this category includes all systematic biases that have been experimentally observed when participants were asked to recall or recognize previous material.

* **Opinion reporting:** this category includes all systematic biases that have been experimentally observed when participants were asked to answer questions regarding their beliefs or opinions on political, moral, or social issues.

To help group biases by similarity within each task category, we additionally categorized biases into *flavors* (second column in Table 2.3), which roughly capture the kind of cognitive phenomenon behind each bias. The flavors we identified are: **Association,**
where cognition is biased by associative connections between information items; **Baseline**, where cognition is biased by a comparison with (what is perceived as) a baseline; **Inertia**, where cognition is biased by the prospect of changing the current state; **Outcome**, where cognition is biased by how well something fits an expected or desired outcome; and **Self perspective**, where cognition is biased by a self-oriented view point.

In the taxonomy table, each bias is additionally described by one representative peer-reviewed paper (column *Ref*), the extent to which the bias might be relevant to visualization research (column *Relevance to InfoVis*), and a short description (column *Short description*). The color code for relevance to visualization is explained in Table 2.2:

<table><caption>**Tab. 2.2:** Color code for relevance to visualization.</caption><tr><td>8</td><td>Evidence for the alleviation of the cognitive bias in visualization</td></tr><tr><td>7</td><td>Evidence for the existence of the cognitive bias in visualization</td></tr><tr><td>6</td><td>Studied in visualization, but no clear evidence of existence or alleviation</td></tr><tr><td>5</td><td>Discussed in visualization research as relevant, but not yet studied</td></tr><tr><td>4</td><td>Not discussed in visualization but likely relevant</td></tr><tr><td>3</td><td>Probably relevant to visualization</td></tr><tr><td>2</td><td>Potentially relevant to visualization</td></tr><tr><td>1</td><td>Relevance to visualization currently unclear</td></tr></table>

This taxonomy is not meant to be authoritative and is not claiming to be perfect, if only because many of its aspects (e.g., categorization in tasks and flavors, relevance to visualization) are the result of the authors' subjective judgment. The goal is only to give a convenient overview of the rich set of known cognitive biases for visualization researchers who may be interested in studying them, and some representative references as a starting point. We hope this taxonomy will inspire future studies and ultimately help visualization designers anticipate – and possibly alleviate – limitations in human judgment. Please note that this taxonomy is from 2018 and has not been updated – a few relevant studies have been published in information visualization since then.
<table><caption>Tab. 2.3: Our task-based taxonomy of cognitive biases.</caption><thead><tr><th>#</th><th>Flavor</th><th>Cognitive bias</th><th>Ref</th><th>Relevance to InfoVis</th><th>Short description</th></tr></thead><tbody><tr><td colspan="6"><strong>TASK: ESTIMATION</strong></td></tr><tr><td>1</td><td rowspan="5">Association</td><td>Availability bias</td><td>[TK73]</td><td>5 [DDB14; CH17]</td><td>Events more probable if easy to remember</td></tr><tr><td>2</td><td>Conjunction fallacy</td><td>[TK83]</td><td>5 [ZC07]</td><td>Specific outcomes more probable than general</td></tr><tr><td>3</td><td>Empathy gap</td><td>[Loe05]</td><td>1</td><td>Estimations affected by not recognizing the role of current emotional state</td></tr><tr><td>4</td><td>Time-saving bias</td><td>[Sve08]</td><td>4</td><td>Overestimate time saved when increasing speed</td></tr><tr><td>5</td><td>Baseline</td><td>Anchoring effect</td><td>[FB11]</td><td>7 [Cho+18; VZS17b]</td><td>Estimation affected by first piece of information</td></tr><tr><td>6</td><td rowspan="10">Inertia</td><td>Base rate fallacy</td><td>[BS07a]</td><td>6 [MDF12; Kha+15]</td><td>Ignore base rate probability of general population</td></tr><tr><td>7</td><td>Dunning-Kruger effect</td><td>[KD99]</td><td>5 [VZS17a]</td><td>Low-ability people overestimate their performance (opposite for high-ability)</td></tr><tr><td>8</td><td>Gambler's fallacy</td><td>[TK75]</td><td>4</td><td>Current outcome that is more frequent will be less frequent in future</td></tr><tr><td>9</td><td>Hard-easy effect</td><td>[LF77]</td><td>3</td><td>Overconfidence for hard tasks, underconfidence for easy</td></tr><tr><td>10</td><td>Hot-hand fallacy</td><td>[GVT85]</td><td>5 [CH17]</td><td>Current outcome that is more frequent will be more frequent in future</td></tr><tr><td>11</td><td>Insensitivity to sample size</td><td>[TK75]</td><td>5 [ZC07; ED15]</td><td>Estimate probability ignoring sample size</td></tr><tr><td>12</td><td>Regressive bias</td><td>[Att53]</td><td>4</td><td>Overestimate high probabilities, underestimate low ones</td></tr><tr><td>13</td><td>Subadditivity effect</td><td>[TK94]</td><td>4</td><td>Overall probability less than the probabilities of the parts</td></tr><tr><td>14</td><td>Weber-Fechner law</td><td>[CF14]</td><td>6 [Har+14]</td><td>Failure to perceive small differences in large quantities</td></tr><tr><td>15</td><td>Conservatism</td><td>[PE66]</td><td>7 [ZBK15]</td><td>New information insufficiently updates probability estimates</td></tr><tr><td>16</td><td rowspan="10">Outcome</td><td>Exaggerated expectation</td><td>[WK85]</td><td>4</td><td>Exaggerating evidence to fit a conclusion</td></tr><tr><td>17</td><td>Illusion of validity</td><td>[TK75]</td><td>5 [ZC07]</td><td>Overconfidence in judgment based on intuition and anecdotes</td></tr><tr><td>18</td><td>Impact bias</td><td>[SS04]</td><td>1</td><td>Predict future emotional reactions as more intense</td></tr><tr><td>19</td><td>Outcome bias</td><td>[BH88]</td><td>2</td><td>Evaluate decision maker only by choice outcome</td></tr><tr><td>20</td><td>Planning fallacy</td><td>[BGP10]</td><td>5 [DJ14]</td><td>Overoptimistic task completion predictions, especially for self</td></tr><tr><td>21</td><td>Restraint bias</td><td>[NHP09]</td><td>1</td><td>Overestimate of ability to resist temptation</td></tr><tr><td>22</td><td>Sexual overperception bias</td><td>[Has03]</td><td>1</td><td>Over or underestimate of romantic interest from others</td></tr><tr><td>23</td><td>Curse of knowledge</td><td>[KH02]</td><td>7 [XWF17]</td><td>Experts assume that novices have same knowledge</td></tr><tr><td>24</td><td>Extrinsic incentives bias</td><td>[Hea99]</td><td>1</td><td>Others have extrinsic motivations (e.g.money), self are intrinsic (e.g. learning)</td></tr><tr><td>25</td><td>False consensus effect</td><td>[RGH77]</td><td>2</td><td>Overestimate the agreement of others with own opinions</td></tr><tr><td>26</td><td>Illusion of control</td><td>[Tho99]</td><td>3</td><td>Overestimation of one's influence on an external event</td></tr><tr><td>27</td><td>Illusion of transparency</td><td>[SG03]</td><td>1</td><td>Overestimate insight of others into own mental state, and vice versa</td></tr><tr><td>28</td><td rowspan="10">Self-perspective</td><td>Naive cynicism</td><td>[KG99]</td><td>2</td><td>Predict that others will be more egocentrically biased</td></tr><tr><td>29</td><td>Optimism bias</td><td>[Wei80]</td><td>4</td><td>Positive outcomes more probable for oneself than others</td></tr><tr><td>30</td><td>Out-group homogeneity bias</td><td>[PR82]</td><td>4</td><td>Estimate out-group will be more homogeneous than in-group members</td></tr><tr><td>31</td><td>Pessimism bias</td><td>[Sha+07]</td><td>4</td><td>Positive outcomes less probable for oneself than others</td></tr><tr><td>32</td><td>Spotlight effect</td><td>[GMS00]</td><td>1</td><td>Overestimate probability that people notice one's appearance/behavior</td></tr><tr><td>33</td><td>Worse-than-average effect</td><td>[Kru99]</td><td>3</td><td>Underestimate own achievements relative to others in difficult tasks</td></tr><tr><td colspan="6"><strong>TASK: DECISION</strong></td></tr><tr><td>34</td><td rowspan="15">Association</td><td>Ambiguity effect</td><td>[RB90]</td><td>4</td><td>Choices affected by their association with unknown outcomes</td></tr><tr><td>35</td><td>Authority bias</td><td>[Mil63]</td><td>1</td><td>Choices affected by their association with authority</td></tr><tr><td>36</td><td>Automation bias</td><td>[MRO12]</td><td>5 [Sac+16]</td><td>Choices affected by their association with an automated system</td></tr><tr><td>37</td><td>Framing effect</td><td>[TK81]</td><td>5 [ZC07; ED15]</td><td>Choices affected by whether alternatives are presented as gains or losses</td></tr><tr><td>38</td><td>Hyperbolic discounting</td><td>[Tha81]</td><td>4</td><td>Choices affected by small short-term rewards</td></tr><tr><td>39</td><td>Identifiable victim effect</td><td>[JL97]</td><td>6 [Boy+17]</td><td>Donation choices affected by whether victims are identifiable</td></tr><tr><td>40</td><td>Loss aversion</td><td>[Tha+97]</td><td>7 [ZBK15; Das+15]</td><td>Choices affected by whether alternatives are gains or losses</td></tr><tr><td>41</td><td>Neglect of probability</td><td>[Sun02]</td><td>4</td><td>Choices affected by disregard of probability</td></tr><tr><td>42</td><td>Pseudocertainty effect</td><td>[TK86]</td><td>5 [ZC07]</td><td>Choices affected by whether some alternatives are framed as certain</td></tr><tr><td>43</td><td>Zero-risk bias</td><td>[BGK93]</td><td>4</td><td>Choices affected by alternatives with complete risk elimination</td></tr><tr><td>44</td><td>Attraction effect</td><td>[HPP82]</td><td>8 [Dim+19; DBD17c]</td><td>Choices affected by irrelevant dominated alternatives</td></tr><tr><td>45</td><td>Ballot names bias</td><td>[Woo+11]</td><td>6 [Woo+11]</td><td>Voting choices affected by the order of candidate names</td></tr><tr><td>46</td><td>Cheerleader effect</td><td>[WV14]</td><td>1</td><td>Choices affected by whether people are in a group</td></tr><tr><td>47</td><td>Compromise effect</td><td>[Sim89]</td><td>5 [DBD17c]</td><td>Choices affected if presented as extreme or average alternatives</td></tr><tr><td>48</td><td>Denomination effect</td><td>[RS09]</td><td>4</td><td>Choices affected by whether the total amount comes for smaller currency bills</td></tr><tr><td>49</td><td>Disposition effect</td><td>[WC98]</td><td>4</td><td>Selling choices affected by initial and not current value</td></tr><tr><td>50</td><td>Distinction bias</td><td>[HZ04]</td><td>4</td><td>Choices affected by how many are the alternatives</td></tr><tr><td>51</td><td>Less is better effect</td><td>[Hse98]</td><td>4</td><td>Choices affected if presented separately or juxtaposed</td></tr><tr><td>52</td><td>Money illusion</td><td>[SDT97]</td><td>4</td><td>Choices affected by nominal monetary values</td></tr><tr><td>53</td><td>Phantom effect</td><td>[PW00]</td><td>4</td><td>Choices affected by dominant but unavailable alternatives</td></tr><tr><td>54</td><td rowspan="7">Inertia</td><td>Endowment effect</td><td>[MG15]</td><td>4</td><td>Choices affected by ownership of alternatives</td></tr><tr><td>55</td><td>Escalation of commitment</td><td>[Sta76]</td><td>4</td><td>Choices affected by continued commitment to suboptimal outcome</td></tr><tr><td>56</td><td>Functional fixedness</td><td>[GB05]</td><td>4</td><td>Choices affected by object use affected by the traditional way of use</td></tr><tr><td>57</td><td>Mere-exposure effect</td><td>[Zaj01]</td><td>7 [Das+17; ED15; Kap+16]</td><td>Choices affected by familiarity (repeated exposure)</td></tr><tr><td>58</td><td>Semmelweis reflex</td><td>[Mor+13]</td><td>3</td><td>Choices of medical practices affected by former established norms</td></tr><tr><td>59</td><td>Shared information bias</td><td>[Bak10]</td><td>4</td><td>Group choices affected by sharing only known information</td></tr><tr><td>60</td><td>Status quo bias</td><td>[SZ88]</td><td>5 [OPR16]</td><td>Choices affected by the urge to avoid a change, even when better expected value</td></tr><tr><td>61</td><td rowspan="4">Outcome</td><td>Well traveled road effect</td><td>[JJ82]</td><td>4</td><td>Travel route choices affected by road familiarity</td></tr><tr><td>62</td><td>Reactance</td><td>[Bre81]</td><td>4</td><td>Choices affected by the urge to do the opposite of what someone wants you to do</td></tr><tr><td>63</td><td>IKEA effect</td><td>[NMA12]</td><td>5 [SB17]</td><td>Choices affected by whether alternatives involved self-effort</td></tr><tr><td>64</td><td>Not invented here</td><td>[AP15]</td><td>4</td><td>Choices affected by alternatives of origin external to an organization</td></tr><tr><td>65</td><td rowspan="2">Self-perspective</td><td>Reactive devaluation</td><td>[RS91]</td><td>4</td><td>Choices affected by whether alternatives allegedly originated with an antagonist</td></tr><tr><td>66</td><td>Social comparison bias</td><td>[GST10]</td><td>1</td><td>Hiring choices affected by own competences</td></tr><tr><td colspan="6"><strong>TASK: HYPOTHESIS ASSESSMENT</strong></td></tr><tr><td>67</td><td rowspan="2">Association</td><td>Illusory truth effect</td><td>[HGT77]</td><td>4</td><td>Statement considered true after repeated exposure to it</td></tr><tr><td>68</td><td>Rhyme as reason effect</td><td>[MT00]</td><td>1</td><td>Statement more likely true if it rhymes</td></tr><tr><td>69</td><td rowspan="8">Outcome</td><td>Barnum effect</td><td>[Tob+88]</td><td>2</td><td>High accuracy ratings for vague and general statements</td></tr><tr><td>70</td><td>Belief bias</td><td>[EBP83]</td><td>3</td><td>Hypothesis true if conclusion is believable</td></tr><tr><td>71</td><td>Clustering illusion</td><td>[KT72]</td><td>5 [VZS17a; ED15]</td><td>Seeing patterns in noise, e.g. clusters in a dot field</td></tr><tr><td>72</td><td>Confirmation bias</td><td>[Mah77]</td><td>5 [ED15]</td><td>Favor reasoning or information that confirms preferred hypothesis</td></tr><tr><td>73</td><td>Congruence bias</td><td>[Was60]</td><td>4</td><td>Seeking confirmation of preferred hypothesis, but not for alternatives</td></tr><tr><td>74</td><td>Experimenter effect</td><td>[RR97]</td><td>3</td><td>Subconsciously influence study participants to confirm a hypothesis</td></tr><tr><td>75</td><td>Illusory correlation</td><td>[CC69]</td><td>5 [ZC07]</td><td>Perceived relationship between variables that does not exist</td></tr><tr><td>76</td><td>Information bias</td><td>[BBH88]</td><td>4</td><td>Seek additional information irrelevant to a hypothesis or action</td></tr><tr><td>77</td><td>Pareidolia</td><td>[VFP11]</td><td>4</td><td>Seeing faces in noise, e.g. in your toast</td></tr></tbody></table>
<table><thead><tr><th>#</th><th>Flavor</th><th>Cognitive bias</th><th>Ref</th><th>Relevance to InfoVis</th><th>Short description</th></tr></thead><tbody><tr><td colspan="6"><strong>TASK: CAUSAL ATTRIBUTION</strong></td></tr><tr><td>78</td><td></td><td>Group attribution error</td><td>[AM85]</td><td>2</td><td>Group traits extrapolated to an individual member</td></tr><tr><td>79</td><td></td><td>Hostile attribution bias</td><td>[GHW92]</td><td>1</td><td>Ambiguous intents read as hostile</td></tr><tr><td>80</td><td></td><td>Illusion of external agency</td><td>[Gil+00]</td><td>2</td><td>Positive outcomes attributed to mysterious external agents</td></tr><tr><td>81</td><td></td><td>Just-world hypothesis</td><td>[HB05]</td><td>2</td><td>What goes around comes around</td></tr><tr><td>82</td><td></td><td>System justification</td><td>[JPC02]</td><td>3</td><td>Inertial bias for unfair systems (e.g. slavery)</td></tr><tr><td>83</td><td></td><td>Actor-observer asymmetry</td><td>[MN75]</td><td>3</td><td>Failures of others due to behavior or personality, own failures due to situation</td></tr><tr><td>84</td><td></td><td>Defensive attribution hypothesis</td><td>[Sha70]</td><td>1</td><td>Failure or mishap of others judged by own similarity with the actor</td></tr><tr><td>85</td><td></td><td>Egocentric bias</td><td>[RS+79]</td><td>3</td><td>Own contribution overestimated</td></tr><tr><td>86</td><td rowspan="4">Self-perspective</td><td>Fundamental attribution error</td><td>[GM95]</td><td>1</td><td>Failures of others due to behavior or personality, own failures due to situation</td></tr><tr><td>87</td><td>In-group favoritism</td><td>[TD81]</td><td>1</td><td>Success and positive traits for ingroup members over outgroup</td></tr><tr><td>88</td><td>Self-serving bias</td><td>[CS99]</td><td>2</td><td>Own achievement attributed to behavior or personality, failures to situation</td></tr><tr><td>89</td><td>Ultimate attribution error</td><td>[Pet79]</td><td>3</td><td>Failures of outgroup due to behavior or personality, ingroup failures due to situation</td></tr></tbody></table>

<table><tbody><tr><td colspan="6"><strong>TASK: RECALL</strong></td></tr><tr><td>90</td><td></td><td>Childhood amnesia</td><td>[UN93]</td><td>1</td><td>Harder to recall event details before certain age</td></tr><tr><td>91</td><td></td><td>Cryptomnesia</td><td>[BM89]</td><td>3</td><td>Memory mistaken for imagination, inspiration (e.g. unintentional plagiarism)</td></tr><tr><td>92</td><td></td><td>Cue-dependent forgetting</td><td>[Tul74]</td><td>4</td><td>Failure to recall information without memory cues</td></tr><tr><td>93</td><td></td><td>Digital amnesia</td><td>[SLW11]</td><td>3</td><td>Less likely to remember easily searchable information</td></tr><tr><td>94</td><td></td><td>Duration neglect</td><td>[FK93]</td><td>2</td><td>Recall unpleasant experiences according to intensity, ignoring duration</td></tr><tr><td>95</td><td></td><td>Fading affect bias</td><td>[WS09]</td><td>1</td><td>Emotion of unpleasant events fades, but pleasant does not</td></tr><tr><td>96</td><td></td><td>False memory</td><td>[LP74]</td><td>1</td><td>Imagination mistaken for a memory</td></tr><tr><td>97</td><td></td><td>Humor effect</td><td>[SLH10]</td><td>3</td><td>Easier to recall humorous items</td></tr><tr><td>98</td><td></td><td>Leveling and sharpening</td><td>[KGP00]</td><td>4</td><td>Recall sharpens some features, weakens others</td></tr><tr><td>99</td><td></td><td>Levels-of-processing effect</td><td>[CT75]</td><td>4</td><td>Easier to recall result of deep level analysis</td></tr><tr><td>100</td><td></td><td>Misinformation effect</td><td>[PKF09]</td><td>3</td><td>Recall colored by new information</td></tr><tr><td>101</td><td></td><td>Modality effect</td><td>[GVP08]</td><td>4</td><td>Easier to recall items presented auditorily than visually</td></tr><tr><td>102</td><td></td><td>Mood-congruent memory</td><td>[MMS95]</td><td>1</td><td>Recall biased toward mood-congruent memories</td></tr><tr><td>103</td><td rowspan="10">Association</td><td>Next-in-line effect</td><td>[Bre73]</td><td>2</td><td>Failure to recall words of previous speaker in turns speaking</td></tr><tr><td>104</td><td>Part-list cueing effect</td><td>[Sla68]</td><td>4</td><td>Harder to recall material after reexposure to subset</td></tr><tr><td>105</td><td>Picture superiority effect</td><td>[MD02]</td><td>4</td><td>Easier to recall images (symbolic representations) than words</td></tr><tr><td>106</td><td>Positivity effect</td><td>[MC05]</td><td>1</td><td>Easier to recall positive events than negative</td></tr><tr><td>107</td><td>Processing difficulty effect</td><td>[OM85]</td><td>4</td><td>Easier to recall information which was hard to comprehend</td></tr><tr><td>108</td><td>Reminiscence bump</td><td>[JP96]</td><td>1</td><td>Easier to recall events from adolescence and early adulthood</td></tr><tr><td>109</td><td>Source confusion</td><td>[HC04]</td><td>2</td><td>Memory distorted after hearing people speak about a situation</td></tr><tr><td>110</td><td>Spacing effect</td><td>[Gre89]</td><td>3</td><td>Easier to recall information from spaced than massed exposures</td></tr><tr><td>111</td><td>Suffix effect</td><td>[MCP71]</td><td>2</td><td>Recency effect diminished by an irrelevant sound at list end</td></tr><tr><td>112</td><td>Suggestibility</td><td>[LJ89]</td><td>1</td><td>Ideas suggested by a questioner mistaken for memory</td></tr><tr><td>113</td><td>Telescoping effect</td><td>[JCM06]</td><td>3</td><td>Temporal displacement of an event</td></tr><tr><td>114</td><td>Testing effect</td><td>[McD+07]</td><td>3</td><td>Recall tests lead to better memory than recognition tests</td></tr><tr><td>115</td><td></td><td>Tip of the tongue phenomenon</td><td>[Bro91]</td><td>1</td><td>Recall parts of an item but not the whole</td></tr><tr><td>116</td><td></td><td>Verbatim effect</td><td>[Pop+08]</td><td>1</td><td>Easier to recall gist than verbatim wording</td></tr><tr><td>117</td><td></td><td>Zeigarnik effect</td><td>[Ein+03]</td><td>1</td><td>Easier to recall interrupted tasks than completed</td></tr><tr><td>118</td><td rowspan="4">Baseline</td><td>Bizarreness effect</td><td>[McD86]</td><td>4</td><td>Easier to recall bizarre items</td></tr><tr><td>119</td><td>List-length effect</td><td>[CR03]</td><td>4</td><td>Harder to recall items from longer lists</td></tr><tr><td>120</td><td>Serial-positioning effect</td><td>[Mur62]</td><td>4</td><td>Best recall first (primacy) and last (recency) items in a series</td></tr><tr><td>121</td><td>Von Restorff effect</td><td>[PWA98]</td><td>4</td><td>Distinct items are better remembered</td></tr><tr><td>122</td><td rowspan="3">Inertia</td><td>Continued influence effect</td><td>[JS94]</td><td>5 [Wal+17]</td><td>Recall only first information even after correction</td></tr><tr><td>123</td><td>Choice-supportive bias</td><td>[MJ00]</td><td>2</td><td>Recall past choices as better than they were</td></tr><tr><td>124</td><td>Hindsight bias</td><td>[FB75]</td><td>5 [ZC07]</td><td>Recall past predictions as more accurate after seeing the outcome</td></tr><tr><td>125</td><td rowspan="3">Outcome</td><td>Rosy retrospection</td><td>[Mit+97]</td><td>1</td><td>Remember past overly positively</td></tr><tr><td>126</td><td>Cross-race effect</td><td>[TKB04]</td><td>1</td><td>More difficulty distinguishing people of outgroup race</td></tr><tr><td>127</td><td>Self-generation effect</td><td>[CH89]</td><td>4</td><td>Self-generated content is easier to recall than if simply read</td></tr><tr><td>128</td><td>Self-perspective</td><td>Self-reference effect</td><td>[RKK77]</td><td>1</td><td>Easier to recall self-related information</td></tr></tbody></table>

<table><tbody><tr><td colspan="6"><strong>TASK: OPINION REPORTING</strong></td></tr><tr><td>129</td><td></td><td>Halo effect</td><td>[NW77]</td><td>2</td><td>Personality trait ascription affected by overall attractiveness</td></tr><tr><td>130</td><td rowspan="2">Association</td><td>Moral credential effect</td><td>[MM01]</td><td>1</td><td>Non-prejudice credentials allow prejudicial statements</td></tr><tr><td>131</td><td>Negativity bias</td><td>[Fis80]</td><td>3</td><td>Social judgments affected more by negative than positive information</td></tr><tr><td>132</td><td rowspan="2">Baseline</td><td>Focusing effect</td><td>[SK98]</td><td>4</td><td>Beliefs based on the most pronounced part of given information</td></tr><tr><td>133</td><td>Backfire effect</td><td>[NR10]</td><td>3</td><td>Prior beliefs stronger when correction attempted</td></tr><tr><td>134</td><td rowspan="2">Inertia</td><td>Omission bias</td><td>[SMB91]</td><td>3</td><td>Moral blame affected by whether the harm was due to inaction</td></tr><tr><td>135</td><td>Bandwagon effect</td><td>[NCG93]</td><td>2</td><td>Beliefs affected by opinions of others</td></tr><tr><td>136</td><td rowspan="5">Outcome</td><td>Moral luck</td><td>[Cus08]</td><td>1</td><td>Moral blame depends on event outcome, not just intent and action</td></tr><tr><td>137</td><td>Social desirability bias</td><td>[CM60]</td><td>2</td><td>Respond in questionnaires in a socially approved manner</td></tr><tr><td>138</td><td>Stereotyping</td><td>[Cor+02]</td><td>3</td><td>Assuming characteristics of group member</td></tr><tr><td>139</td><td>Women are wonderful effect</td><td>[RG04]</td><td>1</td><td>Associate more positive characteristics to women</td></tr><tr><td>140</td><td>Anthropocentric thinking</td><td>[Col+17]</td><td>1</td><td>Humans are the center of the universe</td></tr><tr><td>141</td><td rowspan="10">Self-perspective</td><td>Anthropomorphism</td><td>[WCE10]</td><td>1 [Boy+17]</td><td>Humans as analogical base for reasoning about non-human life and processes</td></tr><tr><td>142</td><td>Ben Franklin effect</td><td>[JL69]</td><td>5</td><td>Opinion of others is affected by one's behavior towards them</td></tr><tr><td>143</td><td>Bias blind spot</td><td>[PK07]</td><td>2</td><td>Belief that biases are more prevalent in others than oneself</td></tr><tr><td>144</td><td>Illusion of asymmetric insight</td><td>[Pro+01]</td><td>1</td><td>Belief that one knows more about others than others know about oneself</td></tr><tr><td>145</td><td>Illusory superiority</td><td>[DMH89]</td><td>2</td><td>Personality traits favorable to oneself over others</td></tr><tr><td>146</td><td>Naive realism</td><td>[HC54]</td><td>1</td><td>Belief we experience objects in our world objectively</td></tr><tr><td>147</td><td>Third-person effect</td><td>[Ant+15]</td><td>2</td><td>Others more vulnerable to mass media messages than oneself</td></tr><tr><td>148</td><td>Trait ascription bias</td><td>[PR06]</td><td>2</td><td>Own traits are variable, others are predictable</td></tr><tr><td>149</td><td>Zero-sum bias</td><td>[RBW15]</td><td>1</td><td>Belief that one's gain is another one's loss</td></tr></tbody></table>

<table><tbody><tr><td colspan="6"><strong>TASK: OTHER</strong></td></tr><tr><td>150</td><td rowspan="3">Association</td><td>Attentional bias</td><td>[Jan+10]</td><td>3</td><td>People extract or process information in a weighted manner</td></tr><tr><td>151</td><td>Risk compensation</td><td>[Hed00]</td><td>3</td><td>Risk tolerance based on constant risk, not minimization</td></tr><tr><td>152</td><td>Surrogation</td><td>[CHT13]</td><td>3</td><td>Metrics or models overtake what they were constructed to measure</td></tr><tr><td>153</td><td rowspan="2">Baseline</td><td>Unit bias</td><td>[GRD06]</td><td>1</td><td>People eat more food from bigger containers</td></tr><tr><td>154</td><td>Ostrich effect</td><td>[KLS09]</td><td>4</td><td>Avoiding negative information</td></tr></tbody></table>
## 2.4 General Conclusions

Although judgment and decision making is a huge topic in psychology, behavioral economics and marketing research, for a long time there has been very little work on it in visualization. This chapter summarized five research papers I was involved in that advanced the research area a tiny bit. These papers date back from 2012–2018 and we see more work on this now, some of which has been perhaps inspired by the work we did. For example, our study on base-rate reasoning [MDF12] has inspired follow-up studies in visualization [Ott+15; Kha+15; KBH18], and several papers exploring new biases in visualization [HWS21; Wal+19; Law+21; Xio+22] cite our bias taxonomy [Dim+18].

A lot remains to be done, both in terms of understanding how cognitive biases interact with visualizations, and knowing how to best evaluate visualizations for their ability to support rational judgment and decision making. This chapter already mentioned a few elements to consider in evaluations, including what to measure. I have not discussed a key aspect of measurement, which is the choice of *ground truth*. For example, in our study on base-rate reasoning, we (and previous work before us) consider that there is one correct answer to the mammography problem, and it is the one obtained by applying Bayes' rule: this answer, 7.8%, was our ground truth, and it allowed us to define an error metric. Ground truth relates to the notion of rationality which, as I mentioned in this chapter's introduction, is complicated. I will take up this discussion here, since this is an important issue that has not really been discussed in the papers I summarized.

Defining a ground truth can be easy in math-oriented problems like the mammography problem, but is often harder in tasks that more closely capture real-life judgments and choices. For example, in our study of multidimensional visualizations, the tasks we studied (multi-attribute choice tasks) have a subjective element that makes it hard to define a ground truth: there seems to be no right or wrong answer to a question asking participants to report which holiday package they personally like the best. However, it remains possible to identify choices that are more reasonable than others, especially if we have information about participants' choice criteria. Similarly, in our experiment on the attraction effect, choosing between the cleaner gym and the gym that has more variety is a matter of personal preference (Figure 2.6), and there is no ground truth. However, if we know that a choice was influenced by irrelevant information (the presence of a third, undesirable gym), it can be considered as sub-optimal, in part because it opens people to manipulation. In those cases as in many other cases in cognitive bias research, the
ground truth is not a correct answer but a *normative theory*, i.e., a set of rules or axioms that constrain what can be considered a rational judgment or choice.

In an experiment, the choice of a normative theory can seem perfectly reasonable at first, but turn out to be flawed upon further examination. For example, a classic framing study puts participants in a hypothetical scenario requiring them to choose whether to undergo a high-risk surgery [Mox+03]. To some participants, the doctor says that 90% of patients are alive after five years, and to others, the doctor says that 10% are dead after five years. The ground truth is that participants should be insensitive to the way information is framed, because it is the same information. However, Gigerenzer [Gig18] argued that in real-life communication, framing is often used to convey extra information, that intelligent listeners intuitively pick up. For example, U.S. doctors fear making direct recommendations due to a law that facilitates malpractice suits, and indirect recommendations can take place through framing [Gig18]. Thus, in the classic experiment I mentioned, it could be that participants' answers deviate from the ground truth not because participants are irrational, but because the ground truth is wrong.

If we think back about the attraction effect study, it is possible that the presence of a dom-
inated alternative (and more generally, the way dominated alternatives are distributed)
can give useful information. For example, if a real estate investor sees a cluster of similar
houses that are all dominated, it could mean that this type of house represents a larger
market, and therefore that the non-dominated house that is similar to them (the target)
is truly a better choice. However, situations also exist where the number and position of
dominated alternatives are clearly irrelevant and where a preference for the target would
be irrational. This was the case for our experiment involving real bets, and for which we
could argue that any deviation from the normative theory was irrational. Our qualitative
data does indicate that the vast majority of our participants were unable to rationalize
their choices based on where the dominated alternatives were located. Nevertheless, in
future studies on judgment and decision making, and particularly studies of cognitive
biases, it seems important to carefully justify the choice of normative theory, and discuss
possible cases where this normative theory might be wrong in ecological situations.
# Supporting Effective Communication

3

While visualizing data can help people make decisions for themselves, it is also important that insights gained from data can be efficiently conveyed to other people, to inform their own decisions. In most of the use case scenarios I have covered in the previous chapter, people explore data by themselves (e.g., a list of houses to buy, or a list of holiday plans) and draw conclusions or take action based on what they saw. Here, I focus on cases where someone (for example, a scientist or a journalist) wants to visualize data they are familiar with to convey it to others: this is the area of communicative visualization.

In this chapter, I present four research projects in which I was involved that have to do with communicative visualization (and also with small data). The first two contribute new *tools and techniques* to help researchers communicate data to *other researchers*. The other two projects are *studies* that examine how to communicate data to a *general audience*.

## 3.1 Communicating Data with Tabular Visualizations

A tabular visualization is a table whose cells contain colors and shapes instead of (or in addition to) numerals. Tabular visualizations are quite versatile – for example, we saw that they can be used to support data-driven decision making (subsection 2.3.1), but they can also be effective communication tools. In this section, I discuss a particular family of tabular visualizations designed for scientific communication that date back to the 1970s, and describe the work I and my co-authors did to bring them up-to-date and to make them known to and available to a larger audience.

This section is based on a 2014 paper authored by Charles Périn, Jean-Daniel Fekete, and myself [PDF14]. It also borrows from a 2019 paper [PFD19] and photo material (https://www.aviz.fr/slowdata) by the same authors. Supplementary material is available at https://aviz.fr/bertifier, and the Bertifier tool is available as a web app at https://aviz.fr/bertifier_app/.
<figure><img src="image_22.png" /><figcaption>**Fig. 3.1:** Part of a Bertin matrix handcrafted by the Laboratoire de Graphique (c. 1970–1980, *Archives Nationales*). This matrix is like a regular table where numbers have been replaced by visual shapes, and rows and columns were ordered in a way that reveals patterns. A common approach was to draw an initial matrix from numerical data, photocopy it, cut it along either rows or columns, reorder it, photocopy it again, and start over. Once the result was deemed satisfactory, a clean and annotated version of the matrix was produced for publication. The process could take weeks.</figcaption></figure>

### 3.1.1 Introduction

Jacques Bertin was a French cartographer known for his theoretical contributions to data visualization. His most influential work is his 1967 monograph “*La Sémiologie Graphique*” [Ber67], where he lays out a theoretical framework for visualizing data that influenced areas such as cartography, social sciences and information visualization. Much less known but equally important is his later work from 1975, “*La Graphique et le traitement graphique de l'information*” [Ber75; Ber81]. This book, which is our focus, details a method for visually processing and communicating numerical tables that was meant to be effective, generic and accessible to any scientist and researcher [Pal03]. It was based on four simple steps: i) compile empirical data into a two-dimensional table (called *matrix*), ii) represent cell values visually, iii) group similar rows and columns to reveal patterns, and iv) annotate the end result for communication purposes.

Bertin devised and refined his matrix method after years of work with researchers such as geographers, agricultural economists, ethnologists, and historians [Pal03]. In the 1970s, he founded a “Laboratoire de Graphique” whose main purpose was to help social scientists from his institution turn their data into insightful figures to be used in scientific publications. However, all figures were hand-crafted, and his method required
<figure><img src="image_23.png" /><figcaption>**Fig. 3.2:** A reorderable matrix (c. 1970–1980, *photo Serge Bonin, Archives Nationales*). Reordering Bertin matrices was a crucial but also time-consuming part of the process. In 1968, Bertin and his colleagues designed a mechanical device to facilitate reordering. It consisted in separate plastic cells, and a rod mechanism for reordering them across either rows or columns. The device underwent successive refinements up to the 80s, was sold to other French labs, and even used in schools [PFD19].</figcaption></figure>

using physical material and devices that involved tedious manipulations—often weeks of work (see Figures 3.1 and 3.2). There has been several computer adaptations of Bertin's method [BC94; Sii99; Saw96], but they are limited in functionality, feature only rudimentary interactions, and the end results are not optimized for communication. In our paper [PDF14], we addressed this gap by contributing:

* A list of requirements for a faithful computer adaptation of Bertin's method derived from an examination of his writings and artifacts created by his lab,

* An extensive review of existing computer systems for tabular visualization with their features and limitations,

* BERTIFIER (www.bertifier.com), a web-based tabular visualization authoring system that implements Bertin's method,

* *Crosssets*, a novel interaction technique for creating, manipulating and fine-tuning tabular visualizations,

* Algorithm adaptations to support user-driven matrix ordering,

* A user study suggesting that users from different backgrounds can use BERTIFIER to help them visualize and understand their own tabular data.

This section focuses on reporting examples of Bertin matrices made with and without BERTIFIER, in order to illustrate their usefulness and their expressive power.
<figure><img src="image_24.png" /><figcaption>**Fig. 3.3:** Meat production per country shown as pie charts (left), matrix (middle), and reordered matrix (right). Drafts for the book *La Graphique* [Ber75], Archives Nationales.</figcaption></figure>

### 3.1.2 Two examples from Jacques Bertin

In Figure 3.3, taken from *La Graphique* [Ber75], Bertin uses a simple example to illustrate why matrices are an effective representation for data. On the left, pie charts show the contribution of different countries in the production of different types of meat. Each pie is a meat type. While common, this type of design makes it difficult to understand the data, leading Bertin to qualify it as “useless”. The middle chart is a matrix visualization. With such a visualization, the way production is distributed across countries (rows) and meat types (columns) becomes easier to read. However, since countries and meat types do not have a natural order, many other matrices can be produced, including the one on the right, which is a better choice because it better shows general trends and patterns.

Figure 3.4 shows a more complex matrix, with a larger and more heterogeneous dataset. It is one of the earliest applications of Bertin's method to research data [VLB67]. The visualization lists all war ships (228 in total) owned by France and Great Britain in 1947. Each column is a ship. The top row shows whether each ship is French (black rectangle) or British (white rectangle). The bottom row shows the number of cannons for each ship – the taller the black bar, the more cannons. The middle area encodes each ship's year of construction – the lower the small black rectangle the more recent. This particular matrix is sorted by year of construction. It can be seen that the rate at which ships were built increased between 1700 and 1747, both for French and British ships (since the proportion of black/white rectangles on the first row doesn't clearly change over time).
<figure><img src="image_25.png" /><figcaption>**Fig. 3.4:** Bertin matrix from 1967, showing information about the 227 ships (each ship is a column) owned by the French and British navy in 1947 [VLB67]. The original figure has a vertical aspect ratio but has been rotated by 90° here to save space.</figcaption></figure>

The paper [VLB67] contains three other figures where the matrix is sorted differently to reveal other insights. This visualization manages to convey a fair amount of raw data in a small space – Bertin described his procedure as “*simplifying without destroying*”: raw data must be shown as is, without aggregation or complex processing.

The four navy figures were among the case studies we used when we developed and tested our tool BERTIFIER. We successfully reproduced them with the tool (see https://aviz.fr/Bertifier/Examples). Since the navy dataset was not available, we reconstructed it from Wikipedia articles.
<figure><img src="image_26.png" /><figcaption>**Fig. 3.5:** Computer applications implementing Bertin's matrix method. Each row is a computer application. Crosses indicate unavailable data.</figcaption></figure>

### 3.1.3 Bertifier Example: Survey of Applications

Our next example (Figure 3.5) was also made with BERTIFIER, but shows a recent dataset – a survey of computer applications. The figure visually summarizes the results of a survey we did of computer adaptations of Bertin's method, as part of our work on BERTIFIER. We used BERTIFIER to produce the matrix in Figure 3.5, which we included as a figure in our paper on BERTIFIER [PDF14]. Computer applications are in rows and their characteristics are in columns, such as the year the application was developed, the type of user interface it used, and the features it supported from Bertin's method. The name of each application is indicated on the second column (TMC, Cartax, etc.).

Overall, some early attempts – especially CHART [Bk77] in 1977 – were rich in terms of functionality, but poor in terms of interactivity. On the other end of the spectrum, a few recent implementations – Like Rubin's implementation in d3 [Rub] – provide consistent, direct and easily-discoverable interactions, but at the cost of only providing a few features. There is a marked trade-off between the level of functionality on one side, and the level of interactivity and discoverability on the other side. Nevertheless, it is possible for a computer adaption of Bertin's method to be both highly functional and highly interactive, as illustrated by the last row showing the features of BERTIFIER.
<figure><img src="image_27.png" /><figcaption>**Fig. 3.6:** Results of our usability study of BERTIFIER. Each row is a participant (N=8), with a visual summary of their demographics and of study data such as their responses to Likert items (the thicker a black rectangle the more positive the response on the scale). Crosses indicate unused features. Dashed gray lines indicate means.</figcaption></figure>

### 3.1.4 Bertifier Example: User Study Data

Our next and last example (Figure 3.6) was also made with BERTIFIER, again with the goal of producing a figure to be included in our paper on BERTIFIER. This matrix summarizes the results of our usability study of BERTIFIER involving 8 participants (as rows). Columns are divided in six groups: 1) *Background and expertise*, indicating the participant's basic demographics and level of expertise in different areas; 2) *Table used*, giving info about the numerical table used by them (they were each asked to bring a personal spreadsheet); 3) *Time spent using* BERTIFIER; 4) *Features*, i.e., responses to questions asking how useful and usable each BERTIFIER feature is; 5) *Overall design* – responses to more usability questions; and finally 6) *Outcomes* – responses to overall satisfaction and attitude questions. Those results are discussed in detail in [PDF14].

### 3.1.5 Conclusion

Bertin devised his matrix method and visual language to facilitate the communication of tabular data in research publications. Although visually encoding cell values was not a new idea at the time, he refined his method over many years. We presented examples of Bertin matrices, made with and without BERTIFIER. Although we did not cover the interactions aspects here, BERTIFIER implements novel interaction techniques for formatting and ordering tables that make it possible to easily and rapidly convert numerical tables into Bertin matrices (see video demo at https://www.youtube.com/
`watch?v=tJxAF_a_yBQ`). Our user study suggests BERTIFIER can help both researchers and a wider audience understand and communicate their data visually.

BERTIFIER can be especially useful to human-subject researchers who need to report responses to multiple Likert-type questions. Summary statistics and inferential statistics are commonly used to report such data, but they remove lots of potentially useful information. Bertin matrices allow researchers to report their raw data in a way that can be understood, and they can therefore usefully complement statistics, especially when inferential statistics are not applicable (e.g., for very small N). BERTIFIER has been indeed used in this manner in subsequent work [HJC14; RMM15; Thu+16; PBV16; Bot+16; Hur+16; Per+17; Bes+20]. BERTIFIER has been also used to summarize literature surveys [Hal+16; Raj+17; Koy+17; Fle18; ATD19]. However, most of these papers were co-authored by one of the authors of BERTIFIER or close collaborators, so we cannot claim that the tool has been widely adopted. Our goal was to design a tool that would finally popularize Bertin matrices, but for a tool to be widely adopted it needs to be 100% functional. Our tool was only 95% functional due to lacking features such as undo/redo and save, which likely prevented its broad adoption.

## 3.2 Communicating Results from Multiverse Analyses

In the previous section, I outlined a method researchers can use to communicate their data to their peers that is inspired by practices from 50 years ago. Here, I discuss another method researchers can use to communicate their data to their peers, but this time, it is inspired by a practice that has only recently emerged: that of reporting multiverse analyses. *Multiverse analysis* is a philosophy of statistical reporting where researchers report the outcomes of many different statistical analyses in their paper in order to show how fragile or robust their findings are [Ste+16]. In this section, I explore why it is interesting to combine this idea with the concept of *explorable explanations*, i.e., narratives that can be read as normal explanations but where the reader can also become active by dynamically changing some elements of the explanation [Vic11].

This section is based on a 2019 paper authored by myself, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, and Fanny Chevalier [Dra+19]. Interactive examples are available at https://explorablemultiverse.github.io/.
<figure><img src="image_28.png" /><figcaption>**Fig. 3.7:** Three reporting strategies, from the least transparent to the most transparent: a) traditional analysis with undisclosed flexibility; b) planned analysis; c) multiverse analysis. Each branching represents a choice between different analysis options [NT07].</figcaption></figure>

### 3.2.1 Introduction

The recent replication crisis in psychology and other disciplines has greatly affected the credibility of human-subject research and prompted a movement of methodological reform [NSS18]. Much of this movement calls for more transparency in the way statistics are reported, so that findings become more trustworthy, more likely to be interpreted correctly, and easier to verify and replicate [Nos+15; Mun+17; Cum14]. Many solutions have been proposed to increase transparency. In this work, we seek to understand how we can augment research papers to support and promote transparent statistics reporting.

While there are many ways a statistical report can lack transparency, a common and damaging form of opacity is *undisclosed flexibility* (see Figure 3.7a), i.e., not reporting the different options that have been tried during the analysis [SNS11; Wic+16], or the options that would have been chosen had the data been different [GL13]. Undisclosed flexibility is damaging because it substantially increases the chances of reporting erroneous findings, while being invisible to the reader.

One response to the issue of undisclosed flexibility has been to encourage researchers to commit to a single statistical analysis that has been planned [Dra16; Ber12] and ideally registered [CGD18] ahead of time (Figure 3.7b). Although planning eliminates the problem of undisclosed flexibility, some statisticians and methodologists are starting to argue that more transparency can be achieved by letting researchers try many analyses and report all of them in their paper [SSN15; Ste+16] (Figure 3.7c). This is partly motivated by evidence that different researchers who analyze the same data will make different choices and will thus get slightly—and sometimes widely—different results [Sil+17].

In a *multiverse analysis*, researchers identify sets of defensible analysis choices (e.g., different ways of excluding outliers, different data transformations), implement them
all, and then report the outcomes of all analyses resulting from all possible choice combinations. This approach increases transparency because readers can appreciate the “fragility or robustness of a claimed effect” [SSN15] by checking whether the findings are dependent on arbitrary analysis choices.

Multiverse analysis promises an unprecedented level of transparency for research papers, but the idea is still very much in its infancy. Writing papers with a multiverse analysis is difficult and there is currently very little guidance. An important part of the difficulty lies in reporting the outcomes of potentially hundreds or thousands of analyses in a single research paper (past examples contain between 120 and 1728 analyses [Ste+16; SSN15]), causing challenges for authors, reviewers, and readers. There are currently two ways to approach this problem.

A first option consists of sharing multiverse analyses as supplementary material, letting readers look under the hood of a default analysis and try alternative analysis options in a different environment. This approach has long been promoted by the *reproducible research* movement, and has been the subject of a vast body of work [BD95; GT07; Nüs+17]. Although supplementary material is crucial for reproducibility and reuse, casual readers are very unlikely to engage with it. Thus proponents of multiverse analysis argue for acknowledging, reporting and discussing the multiverse in the research paper itself [SSN15; Ste+16].

A second option consists of summarizing the multiverse in the research paper itself using tables and custom plots. This is the approach put forward by the two papers introducing multiverse analysis [SSN15; Ste+16], which use as examples histograms of *p*-values [Ste+16], grids of *p*-values [Ste+16], and sorted dot-plots of point estimates where color encodes statistical significance [SSN15]. Although such visualizations are very useful as overviews, they force the paper author to collapse the results of each analysis into a single point estimate, *p*-value or statistical significance verdict. In doing so, a lot of the richness and nuance of well-crafted statistical reports is lost.

A third approach, which we propose here, consists of making the results section of the research paper interactive. More specifically, we introduce and explore the concept of *explorable multiverse analysis report (EMAR)*. An EMAR is a multiverse analysis report that is presented in the form of an *explorable explanation* in Bret Victor's sense [Vic11]. Explorable explanations are explanations that “enable and encourage truly active reading” and allow “the reader to play with the author's assumptions and analyses, and see the consequences” [Vic11]. At the same time, they can be read like normal prose: “the reader
is not forced to interact in order to learn” [Vic11]. Consistent with this design philosophy, an EMAR looks like a regular results section, but readers are given the possibility to change analysis options and immediately see the results within the research paper itself.

We posit that EMARS can be a useful and compelling complement to existing multiverse reporting approaches. In this section, we will explore part of their design space through three examples of short interactive papers which we wrote to demonstrate the concept. More extensive discussions (e.g., positioning with respect to previous work on adjacent topics, a systematic description of the design space of EMARS, trade-offs between different EMAR reporting strategies, benefits and challenges raised by EMARS compared to alternative reporting strategies such as the use of supplemental material or static multiverse analysis reports) can be found in our EMAR paper [Dra+19].

We wrote five mini-papers, all of which are re-analyses of previously published studies for which data and R scripts were publicly available. They are self-contained and can be read online at https://explorablemultiverse.github.io or by clicking directly on a code name below. We present three of them here.

### 3.2.2 Example 1 – FREQUENTIST

The FREQUENTIST example [DJ18b] is a reanalysis of a CHI study evaluating physical visualizations [JDF13]. It is meant to illustrate a few basic multiverse analysis ideas for a typical frequentist analysis with confidence intervals (CIs). The results of the analysis are initially identical to the original paper, including the two figures reporting mean task completion time per technique and pairwise comparisons, with 95% CIs. Four aspects of the analysis can be changed by the reader, which has the effect of immediately updating the two plots and some text elements such as explanations and figure captions. Changes are made by clicking or dragging the elements of the text in blue as in Bret Victor’s explorable explanations [Vic11] (see Figure 3.8).

First, horizontally dragging the “95%” text has the effect of changing the confidence level (7 levels are provided from 50% to 99.9%) and updating the length of error bars in the two figures. This allows the reader to appreciate that the 95% level is arbitrary [McE18] and thus that CIs should not be interpreted in a strictly dichotomous manner [Cum14]. Meanwhile, readers who insist on interpreting effects as significant or non-significant have the option of changing the customary cutoff of $\alpha=.05$ (95% CIs), for example to the $\alpha=.005$ (99.5% CIs) criterion now advocated by some methodologists [Ben+18].
<figure><img src="image_29.png" /><figcaption>**Fig. 3.8:** Excerpt from the mini-paper FREQUENTIST, showing widgets embedded in the text in Bret Victor's [Vic11] style. Operating a widget changes one aspect of the analysis and immediately updates the figure.</figcaption></figure>

Clicking the “transformed data” text toggles the text to “untransformed data” and updates the two figures with results from the corresponding analysis. Although some researchers recommend that completion times be log-transformed [SL10], other researchers may be suspicious of, or unfamiliar with data transformations—this option reassures them that the results hold for untransformed data. Similarly, clicking on “t-distribution” switches the text to “BCa bootstrap” and shows the results of the analysis using non-parametric bootstrap CIs, which tend to be liberal (i.e., too narrow) with small samples but do not require distributional assumptions [KG13].

Finally, the plot with the three planned pairwise comparisons (not shown in Figure 3.8) shows uncorrected CIs, but the reader can apply a Bonferroni correction by clicking on the text “not corrected for multiplicity”. Many recommend such a correction but it is not without drawbacks, and there is a complex literature on the topic [Dra16]. To help the reader interpret the CIs correctly, the mini-paper contains a paragraph that gives the individual and the family-wise CI coverage and false positive rates, which are updated whenever Bonferroni correction is turned on or off, or whenever the confidence level is changed. More details can be found in the mini-paper itself [DJ18b].

The FREQUENTIST mini-paper covers a total of 7×2×2×2 = 56 unique analyses. The paper concludes that the findings from the original study (i.e., good evidence of a difference for
Fertility

The classification of women into a high or low fertility
group based on cycle day can be done in several ways:

☒ Participants with cycle days ranging from 7 to 14 are assigned to the high fertility group, whereas participants with cycle days ranging from 17 to 25 are assigned to the low fertility group [2],

☐ days 6–14 are used for high fertility, whereas days 17–27 are used for low fertility [4],

☐ days 9–17 for high fertility and 18–25 for low fertility [5],

☐ days 8–14 for high fertility and 1–7 and 15–28 for low fertility [6], and

☐ days 9–17 for high fertility and 1–8 and 18–28 for low fertility [7].

**Fig. 3.9:** Excerpt from the mini-paper *DATAVERSE*, listing five different ways of dichotomizing a dependent variable. Elsewhere in the mini-paper, an interaction plot gets updated each time an option is chosen.

the first two comparisons, inconclusive results for the third one) are reasonably robust,
as they hold across the sub-multiverse where the confidence level is at 95% or less.

All mini-papers offer the option of animating the multiverse by repeatedly drawing an analysis at random, which turns the paper's plots into hypothetical-outcome plots [HRA15; Kal+18]. This feature gives an overview of the multiverse and permits the reader to observe which parts of the article change substantially across the multiverse.

3.2.3 Example 2 – *DATAVERSE*

The *DATAVERSE* example [KD18] reproduces part of the multiverse analysis reported in Steegen et al. [Ste+16], which is itself a re-analysis of a famous and controversial study on the effect of ovulatory cycles on voting behavior [DRG13]. The *DATAVERSE* example is meant to illustrate alternative ways of processing experimental data (e.g., dichotomizing responses, excluding participants), and the use of interactive choice lists.

The “Constructing the data multiverse” section in Steegen et al. [Ste+16] goes through each data processing choice made in the original study [DRG13] and describes alternative choices that could have been reasonably made. The *DATAVERSE* mini-paper essentially
reproduces this section with the difference that the reader can select particular choices. The mini-paper first lists five ways of dichotomizing a particular dependent variable, and lets the reader choose one of them (Figure 3.9). Four other data processing operations are described afterwards, each with two to three options to choose from. The mini-paper ends with a figure showing the result of the selected analysis in the form of an interaction plot, which is updated each time a different option is chosen in the text.

The *DATAVERSE* mini-paper covers $5×2×3×3×2 = 180$ unique analyses. Steegen et al. [Ste+16] summarizes the multiverse by plotting the 180 corresponding *p*-values. While this summary provides an extremely useful overview clearly showing that the original findings are not robust, it does not allow the reader to examine detailed outcomes of specific analyses of interest. By making it possible to select any particular analysis and see the resulting effect sizes, the *DATAVERSE* mini-paper conveys more complete results than a simple summary of *p*-values. As in the *FREQUENTIST* mini-paper the multiverse can be animated, giving a striking demonstration of the variability of effect sizes across the multiverse that can usefully complement the *p*-value summary.

### 3.2.4 Example 3 – DANCE

The *DANCE* example [Dra18] is a reanalysis of a previous InfoVis study on the perception of correlations [Har+14]. It is meant to illustrate the use of simulated datasets to convey inferential information that can be missing from plots.

Here, the analysis procedure is the same across the multiverse but the raw dataset varies. More specifically, 100 alternative datasets were created from the original study's dataset using bootstrapping [Woo05]. A bootstrap dataset is created by sampling the original dataset with replacement. A remarkable property of bootstrapping is that the distribution of bootstrap samples tends to resemble the true sampling distribution, and thus bootstrap samples can be used to compute CIs [Efr92]. In the *DANCE* example, we use bootstrapping to derive a set of datasets that could have reasonably come up if the study was replicated with different participants. We then subject all datasets to the same statistical analysis and plotting procedure.

The mini-paper reproduces the analysis from the original study, with its four plots. It also lets readers replace the original dataset with any of the 100 bootstrap datasets. When the dataset changes, each of the 4 plots changes slightly. More interestingly, animating the multiverse yields hypothetical-outcome plots [HRA15; Kal+18] similar to Cumming's
<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /><figcaption>**Fig. 3.10:** Top: plot showing a ranking of visualizations in their ability to convey correlation [Har+14]. Bottom: an alternative plot that could have reasonably come up in an exact replication, created by bootstrapping the experimental dataset. Some results hold (e.g., the bottom of the ranking) while some do not (e.g., the top and middle of the ranking). The mini-paper DANCE allows to animate between 100 of those plots.</figcaption></figure>

dance of *p*-values [Cum09] and other statistical dances [Dra17], with the difference that the sampling distribution is estimated from data rather than simulated.

animating the multiverse of bootstrap datasets allows the reader to appreciate the reliability of the different quantities, trends and patterns depicted by each plot and to carry out “inference by eye” [CF05]: a pattern that is stable across the multiverse is a good indication that it is reliable. Compared to static representations of inferential information such as error bars, this technique has the advantage of being applicable to any plot. It is especially useful for revealing statistical uncertainty that is hidden in some plots, such as the ranking plot reproduced in Figure 3.10. More examples can be found in the mini-paper.
### 3.2.5 Limitations and Possible Objections to EMARS

There are several limits to our work. First of all, our examples are meant to capture elementary EMAR techniques, but many other more sophisticated techniques are possible to further enhance EMARS, such as coordinated views and linking between figures [Rob07], and reverse direct manipulation [Dra+08] to let readers manipulate plots and observe which analysis options lead to certain plots (e.g., “which options yield the largest effect?”). Our examples also cover relatively simple statistical analyses. More complex analyses would probably need to expose less parameters to preserve usability: but as pointed out by Steegen et al. [Ste+16], multiverses do not need to be large to be useful.

Importantly, our mini-papers are only proofs of concepts and for EMARS to be adopted, it is crucial that usable toolkits are developed. A future toolkit could build on modern interactive document authoring tools such as Idyll [CH18] and capitalize on initiatives to integrate academic writing with statistics environments such as the recently released Radix [Mis18], which combines the Distill framework with R Markdown. But even if tools are developed to support the authoring of EMARS, some objections to their widespread use will likely remain. We examine four potential objections:

**Objection 1: writing EMARS will remain hard.** With the proper tools, it is unclear whether writing EMARS will be harder than writing static multiverse analysis reports as in [Ste+16; SSN15]. It is clear however that writing EMARS will always be harder than writing single-universe analyses, and as with providing supplemental material, the extra effort may not come with tangible rewards [BVH]. The statistical analysis itself is more work-intensive and even if new libraries can be developed to facilitate it, it is unlikely that the job can be automated [Ste+16]. Similarly, it will never be possible to ensure that a multiverse analysis is complete or even well-chosen. But again, even a small multiverse analysis is superior to a single-universe analysis in terms of transparency [Ste+16]. So for researchers who want to signal or promote transparency, the option of writing EMARS can still be attractive, and may become more and more attractive as signals of research integrity and transparency get more and more rewarded [Gin12a; Gin12b].

**Objection 2: reading EMARS will remain hard.** In the full paper [Dra+19], we pointed out potential usability issues with EMARS, including difficulties with following changes that occur outside the viewport and predicting where changes will occur. It would be interesting in the future to investigate whether these issues can be mitigated by improvements to the paper reading UI. For example, the interface could present two views of the same article that can be scrolled independently, allowing readers to monitor
targets that are located far from the operated controls. Alternatively, changes outside
the viewport could be highlighted [Che+10; BR03], or replayed back when brought into
view [Bau+06; BDB06]. Regardless, engaging with EMARS will always require extra effort
from the reader. Because of this, it is again crucial that EMARS can be read at two levels
and understood without interacting [Vic11] by readers who quickly want to learn about
the authors' conclusions and how they arrived at them.

*Objection 3: reviewing EMARS will remain hard.* At first sight, EMARS seem to pose a challenge to reviewing policies. Publishers typically impose strict cut-offs in terms of page length, and enabling interactive content implies that authors can provide a theoretically unbounded amount of content. EMARS also break the linear structure reviewers are used to. The problem is however not new, and already arose with the introduction of supplemental materials [Mar09]. Publishers and reviewers can simply consider all non-default analyses as supplemental material: in case reviewers are not required to review supplemental material (e.g., as in many venues), only the default analysis would need to be reviewed. Reviewers are still free to scrutinize other analyses and demand analysis options to be added [SSN15].

*Objection 4: preregistration will remain preferable.* Multiverse analyses are compatible with preregistration [Ste+16; SSN15]. One approach is to preregister the entire multiverse analysis [SSN15], which would require specifying the default analysis in the EMAR (if any), as choosing which analysis to emphasize is a major researcher's degree of freedom. A weaker but easier form of preregistration could include the default analysis only.

3.2.6 Conclusion

We presented *explorable multiverse analysis reports (EMARS)*, a new approach to statistical reporting where readers of research papers can explore alternative analysis options by interacting with the paper itself. We hope our work will inspire more HCI research where the academic paper is treated as a user interface whose purpose is to convey scientific knowledge in an accurate and transparent manner. For now, this idea has not been implemented in a tool as far as we know, but it is relatively recent work so there is hope that it will inspire future tools. To further help researchers develop visualization tools and techniques to support multiverse analysis and its reporting, we (some of the same co-authors as well as Brian Hall and Yang Liu) subsequently published a survey of multiverse analysis visualizations that researchers use in their academic papers [Hal+22].
## 3.3 Do Trivial Charts Hinder Truthful Communication?

We have seen two (more or less) novel ways of conveying data to scientific audiences. Now I focus on communicating data to the general public. This topic spans a huge research space and I will only report two studies on very specific research questions.

In this section, I focus on whether charts can be counter-productive when the amount of data to be conveyed is extremely small. This question was prompted by a new study that garnered attention at the time [TW16], and according to which the answer is yes. I had strong reservations about what I thought was the paper's implied message – that charts hinder clear thinking and should be used sparingly in serious and honest communication. I found this advice misguided and the study's conclusions implausible, because I think visual encodings are always better (easier to process and more informative) than numerals, unless exact values are needed. A quick examination revealed serious flaws in the study. So with Yvonne Jansen, we decided to refute the study's conclusions with a new study.

This section is based on a 2017 paper authored by myself and Yvonne Jansen [DJ18a]. Supplementary material (stimuli, data, analyses and experimental software) is available at https://aviz.fr/blinded.

### 3.3.1 Introduction

In 2014, Tal and Wansink [TW16] published a study entitled "Blinded with science: Trivial graphs and formulas increase ad persuasiveness and belief in product efficacy" in the journal *Public Understanding of Science*. The study claims that adding a trivial chart to a description of a new drug increases people's belief in the drug's efficacy. Figure 3.11 shows the stimuli and question from their second experiment: when a chart was included, more people thought the drug was effective. The authors argued that the chart was redundant but persuasive merely because of its scientific aura. The article concludes by advising "caution when encountering communications hinting at scientific credibility", and advising consumers to "ignore spurious cues to a scientific basis". These findings were publicized by the media and cited in the scientific literature, including in information visualization [Pan+15; Sta+16; CG14].

The article summarized in this section is a reappraisal of this study, which from now on we refer to as the *BwS study* (as in “Blinded with Science”). Our goal was to examine an
<figure><img src="image_32.png" /></figure>

a

<figure><img src="image_33.png" /></figure>

b

<caption>**Fig. 3.11:** Stimuli used in our replication of experiment 2 from Tal and Wansink [TW16].
(a) no-chart condition; (b) chart condition.</caption>
alternative explanation for the results obtained in the BwS study, namely, that the charts allowed participants to better grasp the magnitude of the stated drug effect. Though the charts were informationally redundant, it is well known in information visualization that quantitative facts are often better understood if presented visually [KMS99; Fek+08; Mun14]. In the BwS study, the data only consisted of two quantities so the added value of the chart is unclear. Nevertheless, for the BwS study's conclusions to hold, it is important to ascertain that the chart did not give participants a better sense of the drug's efficacy.

Some chart experts advise against using charts for conveying very few numbers (as in Figure 3.11-b), but in reality, very little is known on whether such charts are actually helpful or harmful (for a review of the evidence, see our full paper [TW16]). In the BwS study, it could well be that participants found it hard to intuitively grasp the magnitude of the drug's effect when conveyed in textual form, and that the bar chart made it easier to see. The BwS study did attempt to establish that the chart had no positive effect on understanding, but the evidence is insufficient for several reasons: (1) the claim is based on accepting the null hypothesis with a single low-powered experiment, (2) the question did not test understanding but instead tested the retention of a specific number provided in the text (the percent reduction in illness), and (3) percent changes are known to be confusing, and are not the most intuitive measure of a drug's efficacy.

We conducted a series of four replications of the BwS study, which kept the stimuli and questions unchanged, and assessed people's understanding of the data provided about the drug. This allowed us to test whether the increased belief in drug's effectiveness could be explained by an improved understanding of its actual effectiveness.

The BwS study consisted of three experiments, all with positive results. Experiments 1 and 2 tested whether the addition of a simple bar chart affects people's belief in medication efficacy. Experiment 3 tested whether the addition of a chemical formula produced a similar effect. Since our focus is on charts, we did not consider experiment 3 and focused on replicating experiments 1 and 2. BwS' experiment 2 is an improved version of their experiment 1, and addresses a few issues with its design. Therefore, to save space here, we will not describe our replication of their experiment 1, and will move directly to our first replication of their experiment 2.
<figure><img src="image_34.png" /><figcaption>Fig. 3.12: Data comprehension test included in our first replication of BwS' second experiment.</figcaption></figure>

### 3.3.2 First Replication of BwS' Second Experiment

The replicated stimuli and questions are shown in Figure 3.11. All participants were presented with a short text stating that a new drug was developed that was found to reduce the occurrence of the common cold by twenty percent, i.e., from eighty three percent without the drug to sixty three percent with the drug (the numbers were fully spelled out like this). For half of the participants, the text was followed by a restatement of the last two quantities using numerals (illness drops from 83% to 63%). Meanwhile, for the other half, the same two numbers were conveyed using a bar chart. All participants were then asked to report the extent to which they believed that the new drug was effective, on a 1–9 scale.

In the original experiment, a post-test was administered to rule out the possibility that the chart helped participants process the information – and thus to confirm that the chart was truly superfluous: about 30 minutes later, participants were asked to report “*the percent by which the medication reduced illness*” (the experiment was also part of a longer session and the question was asked at the very end). The correct answer was 20%. No statistical difference was found between the two conditions, leading the authors to conclude that “*the effects of graphs [...] is not moderated by increased understanding or retention of information*” [TW16]. This claim is however unsupported for a number of reasons, which are discussed in full in our paper [DJ18a].
To address these limitations, we designed our own comprehension test, so we can more reliably measure participants’ intuitive understanding of the magnitude of the drug’s effect in the fictional story. Our test, shown in Figure 3.12, followed best practices in risk communication for medical decision making [Gig+07; MDF12].

The best possible answer to the test can be derived by computing $20 \times \frac{83}{63} \approx 15.2$, which rounds to 15. The reasoning is as follows: let us first consider the entire population — not the people in the comprehension test. Out of 100 people there, 83 would have gotten sick (and 16 remained healthy) if no one took the drug (world A), while 63 would have gotten sick (and 27 remained healthy) if everyone took the drug (world B). It is fair to assume that the 17 people who were healthy in world A would also be healthy in world B (i.e., the drug does not cause some people to get sick). This means that there are $37 - 17 = 20$ people in world B whose cold was prevented by the drug. The drug thus prevents 20 colds out of 83, i.e., fails to prevent 63 colds out of 83. Therefore, among the 20 people in our comprehension test, $20 \times \frac{63}{83} \approx 15$ should be expected to get sick.

To be clear, this is not a trivial exercise and we did not expect that most participants would answer 15, especially since we instructed them not to calculate and to give us their best guess instead. Our assumption was that answers would be wrong overall, but if the chart did help understand the drug’s true effect, answers would be on average less wrong with the chart. Accordingly, we used as dependent variable a continuous error metric (distance to the best answer, 15) instead of a binary measure of correctness.

We recruited 164 participants on a crowdsourcing platform, for a planned sample size of $N = 160$ (the data collection procedures and the analyses reported here were planned before data was collected).

Raw responses to the question “I believe the new drug is effective” are shown as histograms on the left side of Figure 3.13. Although responses vary, most participants thought the drug was relatively effective, with 7 being the most common answer in both conditions. The mean response in each condition is reported in the middle of the figure: the two blue dots are the point estimates, and the two error bars are 95% confidence intervals. Thus there seems to be no evidence for a positive effect of the chart, as confirmed by the difference in means and its 95% CI, shown on the right side of Figure 3.13. Point estimates from the original BwS experiment are shown in red for reference, with the 95% CI for the difference derived from the reported *p*-value [AB11].

Raw responses to the comprehension test are shown on Figure 3.14. The accuracy of participants was rather poor: while the best answer was 15, responses were widely
<figure><img src="image_35.png" /><figcaption>**Fig. 3.13:** Results from our replication of the second BwS experiment. **Left:** distribution of responses to the question “I believe the new drug is effective”, for the no-chart condition (top) and the chart condition (bottom). **Right:** means and difference in means (in blue). Red dots and error bars are the results from the original BwS experiment. Error bars are 95% CIs.</figcaption></figure>

<figure><img src="image_36.png" /><figcaption>**Fig. 3.14:** Results for the comprehension test. **Left:** distribution of responses to the comprehension question, for the no-chart condition (top) and the chart condition (bottom); **Right:** mean response errors and their ratio. Error bars are 95% CIs.</figcaption></figure>

distributed between 0 and 20. In addition, there is no evidence for a difference in mean
comprehension error between the no-chart and the chart conditions.

Thus in this experiment, we were not able to replicate the original finding that the chart
is better at **persuading** people that the drug is effective, despite our sample size being
almost three times as large (N = 164 for us, vs. N = 56 for BwS). As we will see, this
failure to replicate was consistent across our four experiments. In the experiment reported
here, we were also unable to find that the chart increases participants’ **understanding**
of the drug’s efficacy. As we will see, some of our other replications do provide weak
evidence however, and combined together, there is some evidence for an effect.
<figure><img src="image_37.png" /><figcaption>**Fig. 3.15:** Cohen's *d* for the chart's tendency to *increase* belief in efficacy (blue) and to *reduce* comprehension error (black). Error bars are 95% CIs.</figcaption></figure>

### 3.3.3 Summary of All Experiments

We conducted four experiments in total, which were all similar except for a few details. For example, starting from our third experiment, we improved the design of our comprehension test to minimize inversion errors (i.e., participants erroneously reporting the number of people who are healthy rather than the number of people who are sick). In addition, our last experiment was conducted on Amazon MTurk with a US population (as in BwS), while our other experiments were conducted on CrowdFlower.

Figure 3.15 shows the standardized effect sizes we measured for each of our four experiments, and for all our experiments considered together (at the bottom). For blue estimates, the higher the value the larger the chart's contribution to *persuasion*. For black estimates, the larger the value the larger the chart's contribution to *understanding*. The results of the meta-analysis suggest that overall, the chart's contribution to understanding was most likely positive and its contribution to persuasion was very likely negative.

### 3.3.4 Conclusion

Considered together, our experiments suggest that as we expected, the chart used in the BwS study may have helped people to get a better sense of the data, but the effect is likely very small. Meanwhile, we were unable to replicate the original study's findings, as text with chart happened to be as persuasive or even less persuasive than text alone. This suggests that the effect may not be as robust as claimed and may need specific conditions to be reproduced. Regardless, within our experimental settings and considering our study
as a whole (N = 623), the chart's contribution to understanding was clearly larger than
its contribution to persuasion. Therefore, the BwS study does not provide evidence that
charts can persuade by their mere presence, just because they look scientific.

In order to establish that a chart biases judgment, it is necessary to also rigorously
establish that it does not aid comprehension. Our study illustrates one way this can be
done: if a chart's contribution to understanding is substantially lower than its contribution
to persuasion in terms of standardized effect sizes (so the opposite result of Figure 3.15), it
seems reasonable to assume that some persuasion has taken place through the peripheral
route – i.e., based on extraneous cues rather than facts and logic [Pan+14]. One difficulty
is this method's reliance on a particular comprehension test, which could be addressed by
using a battery of comprehension tests.

Our focus was on closely replicating the BwS study, but there are many ways the study
design can be improved if the goal is to investigate how charts in general can affect
data comprehension or persuasion. In addition, the task may be too abstract and too
artificial to capture persuasion in the real world. There seems to be more promise in
testing more realistic tasks, e.g., asking people to assess real facts on meaningful social
issues [Pan+14], exposing them to ads in their personal environment [Meh00], and
measuring intentions and behavior instead of beliefs [NR15].

## 3.4 Communicating Humanitarian Data

The news are full of reports of human tragedies. While some people deplore a negativity bias, communicating about humanitarian issues is important if the audience can help alleviate them, e.g., by donating or by voting. In order to better ground stories in facts and better convey the magnitude of tragedies, journalists and public communicators have started to move beyond mere stories and visualize data. Academics have started studying this practice, but work so far has remained scattered, there is a lack of clear language for thinking about the space of problems and solutions, and most studies have yielded disappointing results. In this section, I summarize work that attempts to fill these gaps.

This section is based on two papers [Mor+20; Mor+21] authored by Luiz Morais, Yvonne Jansen, Nazareno Andrade, and myself, with elements from a pre-print by myself [Dra22b]. Supplementary material is at https://luizaugustomm.github.io/pages/anthropographics-ds.html and https://osf.io/wfgmp/ (for the design space) and https://anthropographics.github.io/ and https://osf.io/epzub (for the study).
<table><tr><td>Any Age Group</td><td>Any Gender</td><td>Any Location</td><td>Any Date</td><td>No Keywords</td></tr></table>

Matched Deaths: **12,042** or more between Newtown and Dec. 31, 2013

<figure><img src="image_38.png" /><figcaption>**Fig. 3.16:** Newspaper visualization showing people who died by a gun shot between the Newtown attack in 2012, and December 2013. Each icon is a person. Sex and age are shown through the icon's shape. Additional information about each person can be obtained by clicking on their icon. Source: Slate [KK13].</figcaption></figure>

### 3.4.1 Introduction

When communicating data about people, information designers and data journalists regularly create data visualizations meant to create an emotional connection with the persons whose data are represented. Figure 3.16 shows one example where the reader can see the story and personal information of each person who died in a public mass shooting in the USA. Meanwhile, the visualization in Figure 3.17 conveys the hardship of the life of refugees by narrating the story of S.W.G., a 26 years old refugee who left Pakistan and spent 651 days before arriving at his final destination. Both visualizations were crafted to bring readers closer to the persons whose data is visualized.

The practice of visualizing data about people in a way that helps the audience relate has been studied in academia and called *anthropographics* [Boy+17; Ber17]. We subsequently defined anthropographics as "*visualizations that represent data about people in a way that is intended to promote prosocial feelings or prosocial behavior*" [Mor+20]. More recently, I introduced the term *humanitarian visualization* to refer to "*data visualizations or infographics designed to promote human welfare*" [Dra22a]. Although the two concepts are not identical, I noticed that the term anthropographics tends to confuse people, so
<figure><img src="image_39.png" /><figcaption>**Fig. 3.17:** Excerpt from an interactive web visualization showing stories of seven refugees who fled from their home country (here, a migrant who left Pakistan towards Italy). The stroke patterns in the path correspond to different means of transportation (e.g., by foot or by car); the dark blue lines represent the days of stay in a city while the light blue ones represent days of travel. Source: storiesbehindaline.com [FP19]</figcaption></figure>

in this HDR thesis I will use the term humanitarian visualization instead. Although humanitarian visualization is very broad by definition, in this section I will be mostly focusing on conveying data about human tragedies.

Visualization designers have explored many strategies for creating humanitarian visualiza-
tions. A popular strategy is to use human-shaped symbols to represent individuals (e.g.,
in Figure 3.16), while other strategies include the use of text annotations to make each
person appear unique (e.g., in Figure 3.17), or the use of visual metaphors (e.g., a red bar
chart to symbolize blood or death). Although humanitarian visualizations have started
to be studied in visualization research, the community still lacks a precise language for
discussing them, as well as an overview of design strategies and opportunities for design.
To address this, we propose a conceptual framework that is meant to help researchers
and practitioners reason and communicate about humanitarian visualizations.

3.4.2 Quick Overview of the Design Space

Based on a previously-published design space [Boy+17] and the examination of a
collection of 105 communicative data visualizations that convey data about people, we
<figure><img src="image_40.png" /><figcaption>**Fig. 3.18:** The seven dimensions of our design space of humanitarian visualizations.</figcaption></figure>

identified seven design dimensions that can be reasonably believed to have some effect on prosocial feelings or behavior, namely: granularity, specificity, coverage, authenticity, realism, physicality, and situatedness. Those seven design dimensions are grouped into two categories: *what is shown*, which are dimensions related to what information and how much information is represented in a visualization, and *how it is shown*, which is about how the information is represented.

We summarize these dimensions in Figure 3.18 using the minimalistic dataset of Table 3.1 as an example. The dimensions are as follows:

* What is shown:

- **Granularity** refers to the extent to which the people in the dataset map to separate visual symbols. In a visualization with low granularity, each symbol stands for a large population, as is commonly the case in statistical charts. In a visualization with maximum granularity, each data item maps to a single person (e.g., Figure 3.16).
<table><caption>Tab. 3.1: Fictional dataset used to illustrate the dimensions in Figure 3.18.</caption><thead><tr><td>Name</td><td>Gender</td><td>Likes</td><td>Photograph</td></tr></thead><tbody><tr><td>Yousef</td><td>Male</td><td>Dogs</td><td>(image)</td></tr><tr><td>Amir</td><td>Male</td><td>Dogs</td><td>(image)</td></tr><tr><td>Alana</td><td>Female</td><td>Dogs</td><td>(image)</td></tr><tr><td>Olaf</td><td>Male</td><td>Dogs</td><td>(image)</td></tr><tr><td>Jeremy</td><td>Male</td><td>Dogs</td><td>(image)</td></tr><tr><td>João</td><td>Male</td><td>Dogs</td><td>(image)</td></tr><tr><td>Fatima</td><td>Female</td><td>Cats</td><td>(image)</td></tr><tr><td>Nadia</td><td>Female</td><td>Cats</td><td>(image)</td></tr><tr><td>Asha</td><td>Female</td><td>Cats</td><td>(image)</td></tr><tr><td>Michal</td><td>Male</td><td>Cats</td><td>(image)</td></tr></tbody></table>

- **Specificity** refers to how much information is shown about each person or group of people. In a visualization with low specificity, symbols convey little or no information and tend to be visually very similar. In a visualization with high specificity, symbols convey rich information and tend to be visually unique.

- **Coverage** refers to the extent to which the people visualized capture the population the visualization is about. In a minimum-coverage visualization, the visualization only features a single person from that population. A visualization like in Figure 3.17 provides partial coverage (six people) of the entire population of refugees the story is about. Full coverage means the full population is represented, aggregated or not.

- **Authenticity** refers to the proportion of genuine information in the visualization. The illustration of partial authenticity in Figure 3.18 shows a design where symbols are given different silhouettes to make them look unique: we can see, for example, a child and a person in a wheelchair, and personal information about people. However, none of this information is in the dataset (Table 3.1). Meanwhile, in a visualization with full authenticity, all visualized information is genuine.

• How it is shown:

- **Realism** refers to how much the visualization's symbols look like actual persons. A visualization with low realism represents people or groups of people using abstract shapes or symbols, e.g., dots, bars, or glyphs. In contrast, a visualization with high realism is made of shapes or images that closely resemble actual people.

- **Physicality** refers to the degree to which a visualization's marks are embodied in physical objects [Jan+15]. In a visualization with **low physicality**, the marks are shown on a flat medium, such as a computer screen or a sheet of paper. All visualizations designed for the web or for magazines fall in this category. In visualizations
<figure><img src="image_41.png" /><figcaption>**Fig. 3.19:** Art installation using grains of rice to show various statistics about people. Here, the piles of rice show victims of the 1906 earthquake, where each grain of rice stands for one victim. Image from https://www.youtube.com/watch?v=zvv110w5lww.</figcaption></figure>

with **maximum physicality**, the visualization is made of physical objects or actual persons. For example, in Figure 3.19, each grain of rice represents a person.<fnref n="1" />

- **Situatedness** refers to how physically close a visualization is (or was) to the people it represents [WJD17]. In visualizations with low situatedness, which is the most common case, the visualization is either presented far from the people, or its physical location is not under the control of the designer. On the opposite end of the spectrum, a visualization has maximum situatedness if it is made of the people it represents. Examples include so-called human histograms, where people sort themselves by height or other attributes to form life-sized histograms (e.g., http://dataphys.org/list/davenports-physical-distributions/).

### 3.4.3 Overview of Previous Studies

In a subsequent paper [Mor+21], we used our design space to characterize 11 experiments (5 studies) that have been conducted between 2017 and 2020 in information visualization to test humanitarian visualization design strategies. Figure 3.20 gives a visual summary of those studies. It would take too long to describe this diagram in detail

<footnote n="1">¹Using collections of physical objects to convey death tolls is a common theme in art installations and memorials. Other examples include nails (dataphys.org), artificial flowers (dataphys.org), toe tags (data-phys.org), shoes (cnn.com), tattoo dots (dataphys.org), and actual people (dataphys.org, classicfm.com).</footnote>
<figure><img src="image_42.png" /></figure>

<figure><img src="image_43.png" /></figure>

<figure><img src="image_44.png" /><figcaption>**Fig. 3.20:** Visual summary of five prior studies on humanitarian visualization (11 experiments in total), plus the new study reported later in this section (2 experiments).</figcaption></figure>
<figure><img src="image_45.png" /><figcaption>**Fig. 3.21:** Excerpt from an interactive visualization of UK migration data in the study by Liem et al. [LPW20]. Participants can go through the story of any of six fictional characters, represented as animated icons. The stories included the reasons why the person moved to the UK, what they gained from it, and what they did afterward.</figcaption></figure>

(see [Mor+21] for more), so we will only skim over it. Each blue rectangle (and the stuff underneath it) is one study, and the dark blue rectangle on the bottom right is our own study. Each study reports one or more experiments (labeled EXP1, EXP2,...), and each experiment generally compares a control condition with an experimental condition. Conditions are visualization designs, which can be partly described using our design space: for example, a visualization design can have low or high granularity, low or high specificity, etc. Figure 3.20 summarizes those design dimensions for each experiment, for the control condition (to the left of the vertical black line) and for the experimental condition (to the right of the line). The findings are summarized at the very bottom.

Two key takeaways from Figure 3.20 are: 1) many experiments are inconclusive; and 2) many experiments compare statistical charts with designs that emphasize individuals, but where the information provided about individuals is not authentic (see Figure 3.21 as an example). More specifically, several past experiments use a control condition with low granularity, low specificity and low realism (i.e., the visualizations are made of symbols that stand for entire populations, do not convey detailed information, and do not look like people), and compare it with a design with high granularity, medium specificity and medium realism (i.e., visual symbols stand for individuals or small groups of individuals, they convey more detailed information, and they evoke people). These three design
strategies are expected to increase participants' concern about the population represented – which is highlighted in Figure 3.20 with green rectangles. However, we think that visibly unauthentic information could simultaneously make participants distrust the visualization, and thus distance themselves from the people represented (which is highlighted with pink rectangles in the same figure). This may explain why those designs were less effective than the authors expected.

Based on this observation, we decided to run a new study that has analogous control and experimental conditions, but where the information provided about individuals is genuine. We summarize this study next.

### 3.4.4 Our Own Study

We chose to visualize data from the Missing Migrants Project (missingmigrants.iom.int), which "tracks deaths of migrants, including refugees and asylum-seekers, who have died or gone missing in the process of migration towards an international destination". The data is collected from official records and media reports. Each entry in the dataset describes an incident, such as a drowning, a vehicle accident, or a shooting. Every incident contains the number of victims and survivors (split into men, women, children, and unknown), as well as where and when the event occurred.

We visualized subsets of this dataset in two ways: using an information-poor visualization design (*infoPoor*) and using an information-rich visualization design (*infoRich*). The *infoPoor* design (Figure 3.22-top) is a simple bar chart showing the number of migrants who died or survived an incident during their migration attempt in 2018. The *infoRich* design (Figure 3.22-bottom) shows one row for each incident that took place in 2018. A text label indicates the date, nature, and location of the incident. Underneath, each individual migrant involved in the incident is shown as a human silhouette conveying their gender according to the dataset (male, female, unknown), age group (child or adult), and whether they died or survived (red vs. light pink). Participants had to scroll to see the entire visualization. The inset on the right of Figure 3.22 gives an indication of the full height of the visualization.

The two designs differ in three major respects: granularity, specificity and realism. Since the *infoRich* design shows real data, it has the same level of authenticity as the *infoPoor* design. The two visualization designs differ in other respects – importantly, *infoRich* requires a significant amount of scrolling, while *infoPoor* fits in the browser window.
Information-poor visualization (bar chart)

Migrant crisis in the Middle East

In the Middle East, irregular migrants are forced to leave their countries due to civil conflicts that have been occurring in the region for decades. Many migrants die in the process.

Here is a chart showing for 2018 how many migrants died due to accidents or attacks and how many survived (source: https://missingmigrants.iom.int/):

<figure><img src="image_46.png" /></figure>

Information-rich visualization (humanitarian design)

Migrant crisis in the Middle East

In the Middle East, irregular migrants are forced to leave their countries due to civil conflicts that have been occurring in the region for decades. Many migrants die in the process.

Here is a chart showing for 2018 how many migrants died due to accidents or attacks and how many survived (source: https://missingmigrants.iom.int/):

<figure><img src="image_47.png" /></figure>

<figure><img src="image_48.png" /><figcaption>Fig. 3.22: Two stimuli used in our study, showing data from the Missing Migrants Project (missingmigrants.iom.int). On the top is the information-poor control condition, and on the bottom is the information-rich experimental condition (participants had to scroll to see the entire visualization – see overview on the right).</figcaption></figure>
<figure><img src="image_49.png" /><figcaption>Fig. 3.23: Page from experiment 1 asking participants to allocate hypothetical funds.</figcaption></figure>

Similarly, *infoRich* requires considerably more time to be fully examined than *infoPoor*. All these potential confounds will make it hard to identify the cause of an experimental effect should we find one. However, the simultaneous manipulation of multiple design dimensions was a deliberate feature of this initial experiment, whose goal was to confirm that we are indeed able to detect an effect if we try to amplify it as much as possible.

We showed those two visualizations designs to 128 participants on a crowdsourcing platform. Half of the participants saw a subset of the missing migrants dataset (migrants in Southeast Asia, year 2018) shown with the *infoRich* design, and then saw another subset (migrants in the Middle East, year 2018) shown with the *infoPoor* design. The other half of the participants saw the Southeast Asia dataset with the *infoPoor* design, and then the Middle East dataset with the *infoRich* design. After each visualization, we measured participants' concern for the migrant population by asking them how much hypothetical funds they think should be allocated to help the affected population vs. helping people elsewhere in the world (see Figure 3.23). At the end of the experiment, we also asked them to split hypothetical funds between the two populations.

Note that our questions were not designed to measure affect. While many essays and studies on anthropographics are concerned with promoting prosocial emotions such as empathy (e.g., [Boy+17; IDW18; Con+20]), empathy does not necessarily lead to helping
<figure><img src="image_50.png" /><figcaption>**Fig. 3.24:** Raw responses, mean responses, and difference in means for the funds allocation in experiment 2. Error bars are 95% CIs.</figcaption></figure>

behavior [Blo17; WB10b; LD16]. Therefore, we chose to measure people's judgment about the extent to which a certain population needs help, regardless of whether this judgment is the result of deliberate reasoning or of intuition and emotions.

The experiment yielded inconclusive results: we did not find clear evidence that the *infoRich* design increased people's concern for the population visualized. Therefore, we ran a second experiment which was an improved and simplified version of the first one, with a higher statistical power. The new experiment only used one migrant dataset (and was thus fully between-subjects), adjusted the framing of the scenario and of the funds allocation question, and used a much larger participant sample (N = 786). The main result is reported in Figure 3.24. The left chart shows the distribution of responses to the funds allocation question for each of the two visualization conditions (e.g., a value of 25 indicates that the participant decided to allocate 25% of the available funds to the migrants and 75% to an alternative cause). The middle chart shows the mean responses, and the rightmost chart shows the difference of means. There is some evidence of an effect in favor of the information-rich design, which seems to prompt larger funds allocations on average, though the difference is small (*diff* = 3.4%, 95% CI [0.02, 6.8]).

### 3.4.5 Conclusion

There is increasing interest in humanitarian visualization, and especially in ways to convey data about human tragedies in order to make readers compassionate with the victims. Humanitarian visualization is a rich and growing area, but the work so far has remained scattered. Despite preliminary empirical work and a few web essays written by practitioners, there is a lack of clear language for thinking about and communicating about such visualizations. We addressed this gap by introducing a conceptual framework and a design space for humanitarian visualizations that other researchers can use.
Empirical studies have started to test designers' recommendations and intuitions by examining whether various design strategies for humanitarian visualizations indeed promote prosocial feelings or behavior. Results have been mostly disappointing so far, in contradiction with the intuitions of many designers and researchers.

We applied our conceptual framework to provide a detailed overview of past experiments. We then conducted two new experiments that use large samples and a combination of design strategies to maximize the possibility of finding an effect. But our information-rich design had a small effect at best on funds allocation. Such a small effect may yield non-negligible outcomes in a large-scale donation campaigns, but it seems very difficult to study experimentally. Indeed, our second experiment was barely able to provide sufficient evidence, despite having involved about 800 participants for a cost of about 500 GBP (550€ or $650). As this effect arises from three design strategies combined and other potentially influential factors (e.g., asking participants to scroll, different exposure times), disentangling these effects may require sample sizes that are impractical to achieve.<fnref n="2" />

It remains possible that there exist alternative design strategies that have a large effect on measures of judgment, decision making, or behavior. However, the overall inconclusive results from the range of experiments conducted by researchers so far call for some skepticism. There is presently no clear evidence that if designers employ current design strategies, this will have a clear and observable impact on people's decisions and behavior. Nevertheless, many areas of the vast design space of humanitarian visualization have not been tested, some of which appear promising (see, e.g., discussions in [Mor+20] and in the next chapter), and it is crucial to test them to advance knowledge.

It is also possible that different ways of measuring prosociality (e.g., observing donations in actual charity campaigns) would lead to different results, perhaps revealing a clear effect where crowdsourced studies could not. It would be surprising if an ecologically more valid measure of prosociality yielded a larger effect (since it is generally harder to measure an effect in the wild than in the lab), but such studies are nevertheless needed. Finally, other ways of using visualization to support humanitarian causes deserve to be considered; For example, it can be useful to study how visualizations can help existing donors better allocate their money to charities, which carries the potential of doing far more good than trying to maximize donations for any single charity [Ord19]. This avenue of research will be discussed in the next chapter.

<footnote n="2">²The relationship between sample size and effect size is not linear. For example, being able to detect an effect four times smaller than experiment 2 (Cohen's *d*=0.05) while maintaining a power of 0.8 would require *N*=12,562 participants.</footnote>
<figure><img src="image_51.png" /></figure>
# Perspectives — Supporting Rational Decisions and Effective Communication on Humanitarian Issues

4

In this chapter, I explore possible future directions of research that combine the goals of the two main chapters of this thesis (supporting rational judgment and decisions, and supporting effective communication), and applies them to the area of humanitarian visualization (discussed in section 3.4), which is an area that I am particularly interested in developing further. Much of the discussions here are informed by the effective altruism movement. I have been aware of this movement for nearly a decade, but only after working on humanitarian visualization with Luiz Morais have I realized that it could inform my research. This chapter is based on an unpublished pre-print by myself [Dra22a]. In another pre-print, I elaborate on the opportunities offered by emerging user interface technologies [Dra22b].

**Dec 2023 Update.** This chapter is largely based on a pre-print I wrote back in June 2022. In the light of recent revelations about serious problems in certain branches of the effective altruism (EA) movement, I no longer support EA as a whole and will update this chapter in the final version of the dissertation. I do still support EA's general philosophy and its branch focusing on alleviating global poverty, which are the focus of this chapter.

## 4.1 Introduction

A lot has been written on how data visualizations can be useful for a variety of tasks [Fek+08], but only recently have researchers started to explicitly consider how they can be used to promote human welfare. In particular, as we have seen in the previous
<figure><img src="image_52.png" /><figcaption>**Fig. 4.1:** Interactive web visualization showing deaths at the borders of Southern Europe between 1990 and 2013. Each small dot represents one dead migrant documented by local authorities. Source http://www.borderdeaths.org/.</figcaption></figure>

chapter, visualization researchers have started to look at how data journalists use data visualizations to raise the public awareness about humanitarian issues (see Figure 4.1).

Despite the great potential of humanitarian visualization as a practice and a research area, its ability to make progress and have a positive impact is limited by the way problems are generally framed, and by the types of solutions and metrics of success typically considered. Suppose a newspaper publishes a striking infographic about the number of blind persons in the U.S. who need a guide dog, resulting in lots of people donating to a charity that provides guide dogs. Most practitioners and researchers would certainly consider this infographic a success, and a benefit to society. This is clearly the case if the counterfactual is a world where the same donors keep the money for themselves. However, the situation is less clear if the counterfactual is a world where the donors choose to give their money
<figure><img src="image_53.png" /><figcaption>**Fig. 4.2:** Distribution of the cost-effectiveness of 100+ health interventions, expressed in disability-adjusted life years (DALYs) per $1,000. Source [Ord19].</figcaption></figure>

to a charity that provides even more benefits. In fact, training a guide dog in the U.S costs $40,000, and the same amount of money can be used to cure more than 2,000 people in Africa of blindness by paying for surgeries to reverse the effects of trachoma [Ord19]. How money should be allocated in this case is a matter of personal judgment, but a lot can be gained by helping people – both the general public and decision makers – allocate their limited resources in an informed manner, with sufficient knowledge about the range of options they have at their disposal, and their expected impact.

Impact would not be a central concern if charity programs and interventions were roughly comparable in terms of their human impact, but this is very far from being the case. For example, a global health study has estimated the effectiveness of more than 100 health interventions in terms of disability-adjusted life years (DALYs) saved per $1,000, finding an remarkably wide and skewed distribution (see Figure 4.2). Effectiveness ranges from 0.02 to 300 depending on the intervention (a factor of 15,000), with a median of 5, suggesting that the vast majority of health interventions are far less effective than the most effective ones. Thus, moving money from the many ineffective interventions to the most effective ones is likely to be helping people considerably more than donating even a lot of money to a random intervention.

Yet donating money to a random intervention is precisely the kind of task currently supported by most humanitarian visualizations, due to their focus on *case-by-case persuasion*: a cause is pre-identified, and the goal is to raise people's concerns about that cause as
much as possible. Similarly, in research, the focus is on finding the design strategies that are the most effective independently from the cause.<fnref n="1" /> However, strategies that help make a message persuasive are not necessarily the ones that promote the best decisions. As an example, a lot has been written on the power of storytelling in visualization [KM13]; But as Neil Halloran, the designer of the celebrated data-driven documentary *The Fallen of WWII* has pointed out, “you can tell a story about a crisis of any size, and tell a compelling story” [Hal17], implying that stories do not necessarily help people to think rationally about the extent of human suffering. Visualization researchers could greatly benefit from new thinking frameworks to help them move beyond case-by-case persuasion and reason about how to best use visualization to alleviate human suffering on a global scale. Effective altruism provides such a framework.

## 4.2 What is Effective Altruism?

The term *effective altruism* (often abbreviated as EA) was coined in 2011 at Oxford University, by a small group of academic philosophers and individuals involved in charity and philanthropy organizations [Mac19]. In 2016, the head of this group, William MacAskill, worked with many leaders involved in the EA community to write a definition that has been widely endorsed by the community:

> Effective altruism is about using evidence and reason to figure out how to benefit others as much as possible, and taking action on that basis. [Mac19]

In 2018, using again input from many EA leaders, MacAskill proposed a more precise definition to be used in academic discussions:

> Effective altruism is: (i) the use of evidence and careful reasoning to work out how to maximize the good with a given unit of resources, tentatively understanding 'the good' in impartial welfarist terms, and (ii) the use of the findings from (i) to try to improve the world. [Mac19]

This definition highlights the double aspect of EA as (i) an intellectual project (a research field) and (ii) a practical project (a social movement). The definition is non-normative: it

<footnote n="1">¹To be clear, these criticisms apply to my own research on humanitarian visualization which I have summarized in section 3.4.</footnote>
does not say how people should behave (e.g., that we should make personal sacrifices to help others). *Welfarist* means that views that assign intrinsic value to other things than well-being (e.g., biodiversity, art, or knowledge) are excluded, while *impartial* means that views that do not weigh people equally (e.g., prioritizing nationals over foreigners) are also excluded. *Tentative* means that the impartial welfarist view is a working assumption that can be debated and refined within EA. For example, while animal welfare is a central concern for many EA proponents, how much moral weight should they be given compared to humans remains unclear.

These broadly-accepted definitions are very helpful when discussing the merits and weaknesses of EA, because many criticisms of EA arise from people using their own interpretation of what it is. This leads to common misconceptions, such as: EA is just applied utilitarianism, it is only about fighting poverty, it is only about donations or earning to give, and it ignores systemic change (for discussions see [Mac19]).

## 4.3 How Can EA Inform Visualization

Because people have different moral intuitions, not all researchers working on – or considering working on – humanitarian visualization will find the EA philosophy compelling enough to embrace it. But for those who do, EA can provide a clear thinking framework in an area that has been lacking one. Indeed, many discussions so far have focused on how to design visualizations that elicit empathy, often ignoring that empathy does not necessarily promote helping behavior [Mor+20]. Even when a visualization does cause people to act, their actions can have a negligible, null, or possibly even negative impact on global human welfare. EA provides clear grounds to think about research goals and metrics of success.

The EA lens can also help researchers think out of the box and broaden the scope of humanitarian visualization research by identifying new types of solutions and approaches. In particular, some visualizations may not promote prosocial feelings or behavior – and thus might not be considered conventional humanitarian visualizations – but may still promote welfare. For example, a visualization that helps a charity director effectively allocate money across different health programs does not promote prosocial feelings or behavior (since all the money will be used to help people no matter what), but it can tremendously increase human welfare.
<table><caption>**Fig. 4.3:** Table showing impact metrics for six charities identified as among the most effective by GiveWell in 2021. Source https://www.givewell.org/cost-to-save-a-life.</caption><thead><tr><th>Intervention</th><th>Top charities implementing this intervention</th><th>Outcome</th><th>Cost of outcome (2021 average, rounded up to nearest $500)</th><th>Output</th><th>Cost of output (2021 average, rounded to nearest dollar)</th></tr></thead><tbody><tr><td>Malaria nets</td><td>Against Malaria Foundation</td><td>Lives saved</td><td>$5,500</td><td>Number of nets delivered to households</td><td>$5</td></tr><tr><td>Malaria medication</td><td>Malaria Consortium</td><td>Lives saved</td><td>$5,000</td><td>Number of children treated with a full course of medicine</td><td>$7</td></tr><tr><td>Incentives for immunizations</td><td>New Incentives</td><td>Lives saved</td><td>$5,000</td><td>Number of infants vaccinated who would not have been vaccinated without the program</td><td>$160</td></tr><tr><td>Vitamin A supplementation</td><td>Helen Keller International</td><td>Lives saved</td><td>$3,500</td><td>Number of children receiving supplements</td><td>$1</td></tr><tr><td>Deworming</td><td>SCI Foundation, Sightsavers, Evidence Action, The END Fund</td><td>N/A</td><td>N/A</td><td>Number of children dewormed</td><td>$1</td></tr><tr><td>Cash transfers</td><td>Give Directly</td><td>Cash transferred</td><td>83% of dollars donated</td><td>N/A</td><td>N/A</td></tr></tbody></table>

EA is a thinking framework but it is also a community. This community is full of people who are deeply knowledgeable about humanitarian issues or have been extensively involved in humanitarian actions, and thus visualization researchers could learn a lot by connecting with them. In addition, the EA community has unique needs that visualization could help address. For example, several EA organizations do research on the effectiveness of different charities and charity programs, in order to guide potential donors. GiveWell is a known example: it maintains a list of top effective charities, primarily based on the cost of life saved (see Figure 4.3). GiveWell shares a range of spreadsheets with data and calculations to explain how it arrived at its estimates. All such initiatives generate lots of useful data, but the amount of information can rapidly become overwhelming for potential donors. And yet, data is currently largely communicated through numbers and text, and very rarely through visualizations. Perhaps visualization could also be used by the EA communicators to better explain its general principles to naive audiences, and by EA researchers to help them analyze the effectiveness of different charity programs.