<figure><img src="image_1.png" /></figure>

HAL
open science

# Memory Recall for Data Visualizations in Mixed Reality, Virtual Reality, 3D and 2D

Christophe Hurter, Bernice Rogowitz, Guillaume Truong, Tiffany Andry, Hugo Romat, Ludovic Gardy, Fereshteh Amini, Nathalie Henry Riche

► To cite this version:

Christophe Hurter, Bernice Rogowitz, Guillaume Truong, Tiffany Andry, Hugo Romat, et al. Memory Recall for Data Visualizations in Mixed Reality, Virtual Reality, 3D and 2D. IEEE Transactions on Visualization and Computer Graphics, 2024, pp.1-13. 10.1109/TVCG.2023.3336588. hal-04595774

HAL Id: hal-04595774

https://enac.hal.science/hal-04595774v1

Submitted on 23 Sep 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
# Memory Recall for Data Visualizations in Mixed Reality, Virtual Reality, 3D and 2D

Christophe Hurter,¹ Bernice Rogowitz,¹ Guillaume Truong,¹ Tiffany Andry,¹ Hugo Romat,¹ Ludovic Gardy,¹ Fereshteh Amini,¹ and Nathalie Henry Riche¹

**Abstract**—This article explores how the ability to recall information in data visualizations depends on the presentation technology. Participants viewed 10 Isotype visualizations on a 2D screen, in 3D, in Virtual Reality (VR) and in Mixed Reality (MR). To provide a fair comparison between the three 3D conditions, we used LIDAR to capture the details of the physical rooms, and used this information to create our textured 3D models. For all environments, we measured the number of visualizations recalled and their order (2D) or spatial location (3D, VR, MR). We also measured the number of syntactic and semantic features recalled. Results of our study show increased recall and greater richness of data understanding in the MR condition. Not only did participants recall more visualizations and ordinal/spatial positions in MR, but they also remembered more details about graph axes and data mappings, and more information about the shape of the data. We discuss how differences in the spatial and kinesthetic cues provided in these different environments could contribute to these results, and reasons why we did not observe comparable performance in the 3D and VR conditions.

**Index Terms**—Data visualisation, recall, virtual reality, mixed reality, locomotion, human memory, perception.

This work was supported in part by European Union's Horizon 2020 within the framework of the SESAR 2020 research and innovation program, under Grant 101114765 for the project in part by 'CODA- Controller adaptive Digital Assistant' under Grant 101114838 for the project in part by 'Trustworthy Intelligent System for Remote Digital Tower (TRUSTY),' under Grant HORIZON-SESAR-2022-DES-ER-01. The work of Christophe Hurter extends his gratitude to the National Research Foundation, Prime Minister's Office, Singapore, for their support under the Campus for Research Excellence and Technological Enterprise (CREATE) programme, specifically for the DESCARTES programme. Recommended for acceptance by A. Jones. (*Corresponding author: Christophe Hurter*.)

This work involved human subjects or animals in its research. Approval of all ethical and experimental procedures and protocols was granted by Université catholique de Louvain (UCLouvain) under Application No. 203486, and performed in line with the accordance with the ethics and Data Management Plan of the Université catholique de Louvain.

Christophe Hurter is with the ENAC, Université de Toulouse, 31000 Toulouse, France, and also with IPAL Singapore International Research Laboratory on Artificial Intelligence, Singapore 138632 (e-mail: christophe.hurter@enac.fr).

Bernice Rogowitz is with the Visual Perspectives Research, Ossining, NY 10562 USA (e-mail: bernice.e.rogowitz@gmail.com).

Guillaume Truong is with the ENAC, Université de Toulouse, 31000 Toulouse, France (e-mail: guillaume.truong@enac.fr).

Tiffany Andry is with the Université Libre de Bruxelles, 1050 Bruxelles, Belgium (e-mail: tiffany.andry@uclouvain.be).

Hugo Romat, Fereshteh Amini, and Nathalie Henry Riche are with the Microsoft Corp, MSR, Redmond, WA 98052 USA (e-mail: romath-ugo@microsoft.com; fereshteh.amini@microsoft.com; nath@microsoft.com).

Ludovic Gardy is with the CNRS, CERCO, 67075 Paris, France (e-mail: ludovic.gardy@cnrs.fr).

## I. INTRODUCTION

THE emergence of the metaverse has moved the discussion of immersive technologies from the laboratory into the real world. Our goal is no longer to simply provide new technologies, but to demonstrate their advantage on tasks that are important and relevant to real-world applications. New Virtual Reality (VR) and Mixed Reality (MR) applications are being developed for gaming, flight control, shopping, and education, where it is hoped that the enhanced cues provided by these technologies will enhance memory [1] and problem solving [2].

Our work explores whether these technologies enhance recall memory for data visualizations. Henceforth, we detail the fundamental aspects that distinguish your contribution from prior research efforts.

1. Previous studies comparing data visualization performance in VR and MR environments, relative to 3D representations on a computer screen, have been limited by the fact that the physical geometries across these modalities have not been explicitly, and precisely, controlled. Does one technology afford better performance because it is intrinsically superior, or could it be because the experiment was instrumented in a way that did not provide comparable spatial and 3-D cues? To address this issue, we present a set of experiments in which the 3D and VR renderings recreate the same physical rooms in which the MR tasks were performed. To do so, we used LiDAR (light detection and ranging) technology to capture the geometry and texture of physical rooms and turn them into virtual rooms. The world explored in our 3D and VR conditions, thus, is a detailed replica of the physical world explored in our MR condition; this method more fully equates the affordances of our experimental conditions, and reduces potentially confounding factors. (2) While research has shown that richer visual environments, such as 3D representations, VR, and MR can augment performance, there have been no studies that explicitly compare the perception of data visualizations in these different environments. (3) One important hallmark of 3D technologies is that the information is located in space. While there have been studies that explore memory for spatially-located photographs or documents in VR and MR, our experiments are the first to measure how well observers remember the spatial arrangements of data visualizations and their order of presentation. This is important, since many data visualization applications require recalling information distributed in spatially-separate locations, such as air traffic control or multi-screen monitoring tasks.
<figure><img src="image_2.png" /><figcaption>Fig. 1. Ability to recall features in data visualizations was measured in four environments: 2D and 3D using an external monitor, Virtual Reality (VR) and Mixed Reality (MR) using head mounted displays. In each environment, observers navigated through sets of ten Isotype visualizations, and were asked to recall their topics, their spatial order/location, data shape and syntactic details.</figcaption></figure>

(4) Previous research on the recall of data visualizations has focused on remembering features of individual visualizations. In our research, we study how participants recall a set of data objects. For each set, we measure the number of visualizations correctly remembered, their spatial location, and memory for features of the visualization that contribute to data understanding. We measure recall for syntactic features, such axis labels and the semantic to each glyph. We also ask participants to report on semantic features of the data, such as whether the effects being represented are ascending or descending, whether there are peaks or outliers, etc.

Our experiments measure recall memory for data visualizations presented in 2D and in three geometrically matched 3D environments: 3D, VR and MR (Fig. 1). For each environment, we measured the number of visualizations recalled, their location in sequence or space, the syntactic and semantic features of the data being represented, and user preference. Our stimuli are Isotype (International System of Typographic Picture Education) visualizations [3], which use pictorial glyphs to represent units of magnitude in a histogram. The example in Fig. 4, shows preference for five pizza toppings, with each icon representing a 5% unit of preference. Isotypes are easy to interpret, are visually compelling, and memorable [4]. In our baseline condition (2D), a scrollable set of Isotype visualizations was presented on a computer display. In our 3D condition, the visualizations were rendered on the walls of a computer model of a 3D room, viewed on a computer screen. In our VR condition, the same 3D model was viewed with a head-mounted display, and in our MR condition, the Isotypes were viewed in a Hololens, which projected the Isotypes onto the walls of the same physical rooms used to create the 3D models. These conditions allow us to examine how technology differences affect memory for data visualizations, the data they represent, and their spatial location. They also allow us to explore the effects of spatial memory cues (placing the visualizations in a 3D room), spatial interaction cues (virtually changing viewpoint within a 3D room), and locomotion cues (enabling users to use kinesthetic cues as they move).

Results of our study show increased recall and greater richness of data understanding in the MR condition. Participants recalled significantly more visualizations by topic (e.g., Preferred Pizza Toppings), and more information about their spatial location in the room (3D, VR, MR) or sequence (2D). They also recalled significantly more information about the syntactic elements (e.g., units and axes) in the visualizations, and more information about the meaning of the data (e.g., trends and features). Moreover, the

MR environment was preferred over the 2D, 3D, and VR environments. We discuss the contribution of spatial and kinesthetic cues in MR to these results, and factors that may have led to lower performance in 3D and VR than expected based on previous research (e.g., Data Mountain [5]) and Memory Palace [6]).

## II. RELATED WORK

Our work draws on the long history of psychological research on how we recall information from long-term memory [7]. In the classic paradigm, the observer views a set of test objects, and is asked, to recall as many objects as possible, and may also be asked to report on particular features of the objects. In our experiments, we present sets of Isotype visualizations to our participants, who, after a time interval, report on the topic of the visualization, features of the data rendering, and on the shape of the data. We also study how spatial context affects retrieval by controlling the order or position of the visualizations, and asking our participants to report on their spatial location.

It has long been understood that spatial cues can enhance recall. The Roman rhetoric textbook, Memoria ad Herennium, from 90 BCE, explained that "natural memory" can be stimulated by memory artifices, like backgrounds and images. For example, remembering the concept of a horse is enhanced by seeing an image of a horse, or imagining the context in which the horse naturally appears. Spatial cues make objects "cling lastingly in our memory" [8]. The Method of Loci [9] or Memory Palace [10] is a mnemonic technique that explicitly trains the user to associate lists of objects to imagined spatial locations. The spatial location creates a more robust encoding of the information, improving its likelihood of being transferred to long-term memory. On recall, the spatial position *primes* the memory by providing an external, associated cue, increasing retrieval [11]. This method has produced remarkable feats of memory. For example, Maguire, et al. [12] studied participants in the World Memory Championships who used the Method of Loci technique. They found that these high-performing memorizers did not have higher general intelligence. However, they did have more activity, measured via fMRI (functional magnetic resonance imaging), in brain regions important for memory, spatial memory and navigation. The Method of Loci traditionally uses imagined spatial environments, but recent research has shown similar advantages provided when the Memory Palace is created virtually [6], [13]. We extend this work by examining recall memory for Isotype data visualizations that are ordered
<figure><img src="image_3.png" /><figcaption>Fig. 2. Graphical renderings of the three physical rooms we scanned (top view). The numbers show the ten predefined places to display the Isotypes (see Fig. 5). The two shortest walls had two Isotype locations, while the two longest walls had three. Each room was a graphical replica of the physical room used in the MR condition, and provided a rich environment with many landmarks, including tables, chairs, trash bins, an extinguisher and office furniture.</figcaption></figure>

sequentially (2D) or arranged spatially in 3D, VR or MR environments. Spatial memory depends on many factors, and we consider specifically those that are expressed differentially in the four technology environments we examine. For example, kinesthetic cues are very strong in Mixed Reality environments where the observer is free to move around the space, are inferred in 3D displays, and missing in 2D presentations (see Fig. 3).

## A. Memory in Data Visualization

Memory for data visualizations is a relatively new research topic. Several researchers have studied recognition memory, which is the ability to correctly identify which visualizations have been viewed previously. These have examined the effects of extraneous visual features, such as embellishments, graphical marks, and *chart junk* [14], [15], [16] on recognition. Recall memory, which requires actively retrieving the memory from longterm store, has also been studied. For example, Saket et al. [17] showed that map-based visualizations produced higher recall than node-link diagrams of the same data, and Haroz et al. [4] found that the pictorial content of Isotype visualizations can help participants recall data values. Borkin, et al. [18] measured recognition memory for labeled graphs, then asked the participants to recall information about each recognized visualization. They found that the labels provided important context information and enhanced recall. Ghani [19] and Basak, et al. [20] have shown that providing spatial cues to node-link diagrams improves memory for features.

Research into the role of spatial cues for memory and navigation has grown in prominence with the growth of immersive environments. Willett et al. [21] introduced the concept of *embedded visualizations* to describe the use of overlay projections and see-through videos to position individual presentations of data

close to their corresponding physical referents in mixed reality environments. Moreover, navigating in a virtual 3D environment provides enhanced spatial cues, such as depth information, and richer information about the relative spatial positions of objects, which Krokos et al. [6] argue, contribute to increasing the recall of portrait images in a Memory Palace design. In addition to these factors, work on embodied cognition [22] has drawn research attention to the importance of kinesthetic cues in forming and recalling memories. In some Virtual-, Augmented-, and Mixed-Reality designs, the user physically moves through the space, which provides cues about body movement (propri-oception) and movement relative to the physical world (kines-thetics) [23]. Legge, et al. [13], have shown that locomotion can enhance memory recall in immersive environments. Drawing on these ideas, we examine recall memory for sets of data visualizations in 2- and 3-D environments that provide different spatial and kinesthetic cues.

## B. Spatial Context in Real and Virtual Environments

In his classic work, Gibson [24] developed the idea that human perception evolved in, and is designed to operate in a world that is actively explored. Objects are situated in spa-tial contexts, and characteristics of these contexts affect their perception. Ideas that are important in user interface design, such as *realism*, *affordance* and *immersiveness* spring from this foundation, which has roots in Gestalt Psychology [25] and the philosophy of phenomenology [26]. The more an environment emulates the real world, the argument goes, the more natural cues are available for cognitive processing [27]. Some important dimensions include increased field of view, high resolution, and depth, which enable perceptual abilities such as visual acuity, depth perception, and motion parallax. The goal of creating
<table><caption>Fig. 3. Comparison of sensory characteristics of the four technologies: 2D, 3D, Virtual Reality and Mixed Reality. All conditions provided high spatial resolution Isotypes. The Mixed Reality environment provided higher spatial resolution and stronger kinesthetic cues through locomotion.</caption><thead><tr><th></th><th>2D</th><th>3D</th><th>Seated VR</th><th>MR</th></tr></thead><tbody><tr><td>Visual Interface</td><td>Direct Viewing</td><td>Direct Viewing</td><td>VR Headset (HP Reverb)</td><td>MR Headset (Hololens 2)</td></tr><tr><td>2D vs 3D</td><td>2D</td><td>2D, 3D</td><td>2D, 3D, Stereo</td><td>2D, 3D, Stereo</td></tr><tr><td>Field of View</td><td>30 x 50cm screen at 60cm= 28° x 45° visual angle<br>58cm diagonal = 51° visual angle</td><td>30 x 50cm screen at 60cm = 28° x 45° visual angle<br>58cm diagonal = 51° visual angle</td><td>Headset: 95° diagonal FOV, sampling the environment<br>360° visibility with head rotation and joystick</td><td>Headset: 54° diagonal FOV Isotypes embedded in human visual field (130° H; 80° V)<br>360° visibility of environment with head and body rotation</td></tr><tr><td>Latency</td><td>60Hz</td><td>60Hz</td><td>90Hz</td><td>60Hz</td></tr><tr><td>Rendering resolution of the environment: Spatial quality</td><td>No environment</td><td>Sampled 3D geometry, with high-resolution texture mapping</td><td>Sampled 3D geometry, with high-resolution texture mapping</td><td>Real environment with highest 3D geometry and spatial resolution; high-resolution digital Isotypes</td></tr><tr><td>Navigation</td><td>Keystrokes to scroll a sequence of Isotypes</td><td>Keystrokes to navigate within a 3D room</td><td>Keystrokes, joystick, and head movements to move within a VR room</td><td>Direct navigation in a real room</td></tr><tr><td>Kinesthetic Cues</td><td>No kinesthetic cues</td><td>Kinesthetic clues from third-person motion</td><td>Kinesthetic cues from head motion</td><td>Kinesthetic cues from head and body motion</td></tr></tbody></table>

ever more realistic environments motivates the development of larger displays and environments supporting active navigation, where objects can be manipulated and explored naturally. 3D and immersive technologies not only make the experience more visceral [28], but can also increase data understanding [29]. Our approach explores how spatial context affects recall memory for ordered sequences of visualizations, and how differences in the affordances of 2D, 3D, VR and MR display modes, such as kinesthetic cues, affect the ability to recall spatial position, and syntactic and semantic features of the data.

### C. Moving in Real-World Environments

An important aspect of real-world environments is that users can move freely wherever they choose, direct attention where they choose, and use affordances of the real world. Moving through the world not only provides better sensory information to our brains, but the act of moving in the environment is itself an important process, called embodied cognition [30], [31]. Since short term memory is a limited store, encoding information onto our sensorimotor cortex, through associated body motions, can free up working memory [32], providing more cycles for encoding memories into the long-term store. The more information reaching long-term memory, the greater the potential for recall. In kinesthetic perception research, a distinction is made between egocentric and allocentric movement. In egocentric movement, the observer actively travels in physical space, taking advantage of kinesthetic memory, optic flow, and the

encoding of information into context-dependent units (episodic memory [33]). In allocentric memory, the observer passively views a representation of a physical space, such as a 2D map. Vielledent, et al. [34] found that the ability to remember a path was improved when observers physically walked the path, or mentally imagined the turns taken. This similarity provides a link between the mental imagination used in the Method of Loci and the physical act of self-driven locomotion [35]. To our knowledge, the literature does not contain a controlled and comprehensive exploration of how recall depends on the realism and kinesthetic affordance of the environment. Our work contributes to this research by investigating the impact of real world affordances and kinesthetic cues on recall memory for Isotype visualizations in environments with varying degrees of realism.

### D. Moving in Virtual Environments

The link between body movement and recall was implied in the early work by Robertson, et al. [5], who created a 3-D graphical representation of a desktop on a computer screen, and positioned document icons on this inclined plane. The Data Mountain interface provided faster and more accurate retrieval than its non-spatial counterpart, because the spatial cues provided a framework for organizing and recalling the graphical entities. By allowing users to reach out, grab and move the virtual documents, it also simulated the kinesthetic experience of manipulating objects in the real world. Tan, et al. [23], for example, found that users were better able to recall the
positions of objects when they could physically touch them on a touchscreen.

Waller et al. [36] provide a valuable summary of the availability of different modes of sensory information in real and virtual environments and Ruddle [37] identified how body motion is involved. The results are mixed on whether movement in a virtual environment enhances memory. Gaunet et al. [38] did not reveal an advantage in scene recall when the observer navigated with a joystick on a 3D screen versus simply viewing images of the scenes without context. Krokos et al. [6] found an 8.8% increase in recognition for portrait pictures in this same set-up. Friedrich et al. [39] investigated the effect of body motion in VR for placement retrieval. Yang et al. [40] measured the amount of knowledge retrieved and retained from scholarly articles, and found that using a memory place technique in VR provided better results than an imagined memory palace. Vindenes, et al. [41] compared word recall in three Memory Palace environments and found that more words were recalled using desktop VR than immersive VR, and that the conventional non-VR setup provided the best recall. McCormick [42] described how immersive analytics systems could benefit from MR, which overcome some of the sensory limitations in VR. In Memory Palace experiments in Augmented reality, Lim and Lim [43] showed that physical movement and creating annotations on a hand-held device improved recall for ordered historical facts and groupings of items. In our experiments, we extend previous work by examining recall memory in four test conditions where the participants can manipulate their viewpoints. In our 2D and 3D conditions, participants can move their heads and bodies, but their movements do not change the visual representation. In our VR condition, participants can move their heads and navigate virtually in the 3D room, with access to depth cues like optical flow. In MR, participants physically move through the space, with access to a full range of visual depth and kinesthetic cues.

### III. METHODS

The goal of our experiment is to explore whether data presentation technologies (2D, 3D, VR and MR) could affect visualization recall and understanding, and to shed light on how different perceptual cues might contribute. To compare between technologies, we selected environments that are commonly used for visualization research, and configurations that are commonly used in gaming and visualization applications. We used a within-subject design, with all subjects serving in all four conditions. The same visualization stimuli and the same memory recall tasks appeared in all four test conditions. Many design decisions were considered for making our virtual environments as similar as possible, balancing the relevance to our research direction with rendering and navigation limitations for the different technologies. We also explicitly considered the safety of our observers, and took user fatigue into consideration when deciding on the number of conditions and the length of the experiment.

#### A. Creating Similar 3D Environments for 3D, VR and MR

The environments used in the 3D and VR conditions were graphical replicas of the physical rooms used in the MR

<table><caption>TABLE I RENDERING DETAILS FOR THE ROOMS USED IN THE 3D AND VR</caption><thead><tr><td>Room</td><td>Dimension</td><td>Nbr Vertexes</td><td>Texture size</td></tr></thead><tbody><tr><td>Rest Area</td><td>6m x 12m</td><td>700k</td><td>12 Mb</td></tr><tr><td>Open-space</td><td>6m x 8m</td><td>45k</td><td>12 Mb</td></tr><tr><td>Office</td><td>5m x 7m</td><td>134k</td><td>10 mb</td></tr></tbody></table>

condition. We digitally captured three physical rooms (Rest Area, Office and Open Space) and rendered them as full 3D models. Fig. 2 shows a top view of the three digital rooms. The numbers show the ten locations where the visualizations were placed. The 3D geometry of the rooms, and the locations of the visualizations, thus, were the same, in 3D, VR, and MR. Observers viewed sets of visualizations embedded in 3D models of their home environments, two in France and one in Belgium. We chose rectangular rooms, ranging in area from 35 m² to 60 m² (see Table I). Every room displayed ten Isotypes: 2 on its two short walls and three its two long walls (see Fig. 2). Every room contained many functional and decorative objects like furniture and also paintings or posters on the wall. This created complexity in the scene which can provide visual landmark cues [44]. All these items were displayed in the graphical environments (3D and VR) and in the actual physical rooms (MR). In order to scan the physical rooms and create their digital representation, we applied the following method. We first removed from the physical room all items whose location may be displaced by chance over time. This helped us maintain the similarity between the virtual and physical rooms over the course of the experiment, which took many days. We performed the scan at night under controlled lighting conditions.

We scanned the rooms with an iPad pro 12 with Lidar to capture depth maps of the three physical rooms and store them digitally. We captured the data with a 3D scanner App (https://www.3dscannerapp.com/). The system captured the physical shape of the room and created a 3D model, a connected triangle mesh in 3D space, using MeshLab (https://www.meshlab.net/). Finally, we exported the data in OBJ file format with vertex descriptions (location and texture mapping) and corresponding 8 K texture files (JPEG file format). Thus, we mapped onto these 3D shapes a texture map consisting of images from the real room, which were captured during the digitization process. The digital rooms had the general shape and the visual appearance of their physical counterparts (see Fig. 2). Our 3D and VR models were not perfect replicas of the MR rooms, owing to the limits of sampling. Nevertheless, all the main landmarks were present, and the texture-mapped image captured the same visual information as in the real room, making the digital rooms easy to recognize. In order to limit potential platform-specific experimental bias, we developed a common 3D visualization engine able to display a room and floating images (i.e., data visualization billboards) and let the users navigate in this environment. To address portability issues, we used a web-based development with threejs (https://threejs.org/) release 132. Each modality had its own navigation paradigm which was device-specific (see Fig. 3 for details). To run our software, we used a 15-inch laptop under windows 10 with 32 Gb memory and a RTX 3080 graphic card.
## B. Perceptual Affordances in 2D, 3D, VR and MR

The conditions we examined differed in many ways, and provided different possible cues to support recall and understanding. We did not control all these cues experimentally, and it is therefore beyond our scope to provide a formal test of how these different factors participate in recall. Nevertheless we have identified several perceptual dimensions along which our conditions varied. Fig. 3 summarizes the physical characteristics of the four conditions in our experiment. For 2D and 3D, the stimuli were presented on a 30 x 50 cm screen, whose diagonal subtended 51 degrees of visual angle, when viewed at 60 cm. The observer scrolled the images (2D) or navigated around the room (3D) via keystrokes. In VR and MR, the participants wore headsets with limited field of view. In VR, the participants were able to move their heads to view the whole 360-degree environment in stereo, and used a joystick or key strokes to navigate. With MR, the experiment was conducted in a real room that provided full 360-degree visibility, in stereo, and unrestricted locomotion. The Hololens 2 headset provided a reduced visual angle for the holographic visualization of the Isotypes (52 degrees). The spatial resolution of the Isotypes was identical in all four conditions. In 3D, VR, and MR, they could be zoomed into by changing the user's point of view. In 3D, the participant could move through a representation of the room visually, which provided some sense of kinesthetic movement. In VR, the participants could also move their heads, which provided some direct kinesthetic cues. In MR, the navigation physically involved the whole body, affording strong kinesthetic cues.

## C. User Interface Design

In the 2D control condition, which provided no spatial cues, the participant sat at the screen and used the keyboard to scroll back and forth through the sequence of ten randomized Isotypes. For 3D, VR and MR, the Isotype stimuli were rendered onto ten locations along the walls of the 3D environment. In the 3D condition, the observer was seated and viewed the rendered room models with the embedded stimuli on a 1920 x 100 screen. Keystrokes were used to navigate the 3D model on a computer screen. VR technologies differ considerably in their methods for providing interaction and navigation [45]. Locomotion Vault [46] provides a database and visualizations of over 100 navigation techniques in VR. For our VR condition, we used the technology most commonly used for video games and other VR applications, a technique di Luca et al. [46] call third person. In third person, the user is seated on a chair, views the 3D room with a head-mounted display, and navigates the virtual environment at normal walking speed. Navigation is done with a joystick, which appears at the location of the users' hand in the virtual environment. This technique provides a good balance between providing natural navigation and good visual cues. We used the HP Reverb B G1 head mounted display (90hz frame rate, 2160 x 2160 pixels per eye or 4320 x 2160 pixels combined, 114 degrees field of view, Fresnel-Aspherical). For our MR condition, the ten Isotypes were superimposed digitally on the ten locations in each room. Participants viewed the Isotypes through a head-mounted display, and moved freely about the physical room. We used a

Hololens V2 (60hz frame rate, 1440 x 936 pixels per eye, 52 degrees field of view).

## D. Isotype Visualization Stimuli

Isotype visualizations(International System of Typographic Picture Education) [3], pioneered by Otto Neurath [47] are particularly memorable [4] and thus compelling for our memorization experiments. In an isotype visualization, the magnitude of a variable is represented by a stacked set of pictographs, where each pictograph icon represents a given quantity. Unitizing the data and mapping it onto a concrete scale [48] provides a compact visual representation of magnitude for different categories (e.g., different countries), or for different levels of a continuous scale (e.g., different years). In our experiments, participants viewed ten Isotype visualizations in each of the four conditions (2D, 3D, VR, MR), and were then asked to recall the topic, the order or location, and semantic and syntactic features for each Isotype presented. We created four sets of ten Isotypes, which were counterbalanced across subjects, across the four conditions. We selected simple data sets with time-varying or country-varying information. In order to avoid memorability bias effects, we tried to balance the salience of the icons and the topic choices. We also balanced the shape of the data (e.g., whether the data were ascending, descending, peaked (outlier) or flat (independent)), the topics selected, and the units of measure. We first selected data from the *Our World In Data* web site (https://ourworldindata.org/) which contains many data sources classified by topic. We extracted potential icons to create our Isotypes from *The Noun Project* icon source (https://thenounproject.com/). We internally discussed the topics, the collected data, and the selected icons in order to create four comparable sets of ten Isotypes. Table II describes our Isotype stimuli.

## E. Novel Isotype Generation Software

We designed our own system for producing Isotypes, in order to have fine-grained control over the parameters, and ensure homogeneity across the four sets of ten images. We grounded our Isotype generator with previous work. DataInk [49] enables authors to draw a sketched icon and bind it to each row of a data table. DataIllustrator [50] and DataQuilt [51] enables authors to bind an icon to a visual element in a visualization. These tools all used a textual description to automate image generation. Our system used a simple markup syntax for the title, the records, the legend, the icon and the document size. The keyword tick represents the numerical value of one pictograph and the keyword X axis is a textual value which will display the legend. Our user interface is shown in Fig. 4, which depicts the creation of an Isotype representing the popularity of different pizza toppings. Using our software was especially useful to iteratively adjust the number of icons to display.

# IV. PROCEDURES

Sixteen volunteers (ten female, six male) participated in these experiments, ranging in age from 22 to 44 (M = 29.47, SD =
<table><caption>TABLE II
CHARACTERISTICS OF THE FOUR SETS OF ISOTYPES USED IN OUR EXPERIMENTS. THE TOP SECTION SHOWS THE ICONS USED IN EACH SET, THE MIDDLE SECTION SHOWS THE DISTRIBUTION OF ASCENDING, DESCENDING, PEAKED AND FLAT SHAPES IN EACH. THE THIRD SECTION SHOWS RANGE OF VALUES</caption><thead><tr><th>Icons Set 1</th><th>Icons Set 2</th><th>Icons Set 3</th><th>Icons Set 4</th></tr></thead><tbody><tr><td><img src="icon1.png" alt="Icon 1: A figure with a circle in the top-left corner, a line segment connecting the top-right and bottom-left corners, and a circle in the bottom-right corner."></td><td><img src="icon2.png" alt="Icon 2: A figure with a circle in the top-left corner, a line segment connecting the top-right and bottom-left corners, and a star in the bottom-right corner."></td><td><img src="icon3.png" alt="Icon 3: A figure with a circle in the top-left corner, a line segment connecting the top-right and bottom-left corners, and a coffee cup in the bottom-right corner."></td><td><img src="icon4.png" alt="Icon 4: A figure with a circle in the top-left corner, a line segment connecting the top-right and bottom-left corners, and a clock in the bottom-right corner."></td></tr><tr><th>Shapes Set 1</th><th>Shapes Set 2</th><th>Shapes Set 3</th><th>Shapes Set 4</th></tr><tr><td>2 x Ascending</td><td>3 x Asc</td><td>3 x Asc</td><td>3 x Asc</td></tr><tr><td>5 x Descending</td><td>4 x Desc</td><td>3 x Desc</td><td>3 x Desc</td></tr><tr><td>2 x Peak</td><td>2 x Peak</td><td>3 x Peak</td><td>3 x Peak</td></tr><tr><td>1 x Flat</td><td>1 x Flat</td><td>1 x Flat</td><td>1 x Flat</td></tr><tr><th>Units Set 1</th><th>Units Set 2</th><th>Units Set 3</th><th>Units Set 4</th></tr><tr><td>1%</td><td>5%</td><td>2 x 10%</td><td>2%</td></tr><tr><td>10%</td><td>2 x 10%</td><td>2 x 1%</td><td>2 x 10%</td></tr><tr><td>1</td><td>3 x 1%</td><td>10%</td><td>2 x 1%</td></tr><tr><td>2 x 10</td><td>2 x 1000%</td><td>100%</td><td>1000%</td></tr><tr><td>100</td><td>100 000%</td><td>1000%</td><td>100 000%</td></tr><tr><td>1000</td><td>1 million%</td><td>3000%</td><td>2 x 1 million%</td></tr><tr><td>100 000</td><td>-</td><td>10 millions%</td><td>100 million%</td></tr><tr><td>2 x 1 million</td><td>-</td><td>1 billion%</td><td>-</td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>Fig. 4. Our Isotype Generator software uses a simple descriptive language to set the title and the size of the image (markup title, width and height). Records are defined for corresponding icons and measures (markup Record, Icon and Tick).</figcaption></figure>

5.20). They were communication students, computer scientists, researchers, and engineers, with some experience with virtual technologies (they had all tried a Virtual Reality headset at least once, and half had already tried a Mixed Reality headset). Subjects were not paid, but were offered coffee and refreshments, throughout. To comply with local COVID-19 policies, participants wore their own face mask at all times and received disposable gloves to wear during the study. Participants read and signed a consent form and filled out a demographic questionnaire.

## A. Pilot Experiments

To tune the parameters of our experiment, we conducted pilot tests where we asked three participants to view and then recall information about Isotypes in the 2D, 3D, VR and MR conditions. Our goal was to calibrate the number of test Isotypes and the time to allow for memorization. We wanted the task to be sufficiently difficult to avoid a ceiling effect, which would mask performance differences between conditions, but not so long as to produce fatigue. The choice of items to memorize was also informed by Miller's seminal magic number paper [52], which showed the retention of 7-plus-or-minus-2 items in working memory [53]. Based on these pilot experiments, we decided to use 10-item sets of Isotypes for each condition, and to have all participants serve in all four conditions (within-subject design). We set the viewing time to 5 minutes, since most pilot participants were able to comfortably memorize all ten Isotypes within this time.

## B. Experimental Design

The goal of these experiments was to measure recall performance using four technologies that varied in the richness of the exploration experience. Participants viewed 10 Isotype visualizations in each condition. Each subject served in all four conditions, and the order of the technologies, and the order of Isotypes within each technology were counterbalanced using a Latin square design. In our 2D condition, observers were seated and viewed the ten Isotypes on a computer screen. These Isotypes were ordered in a presentation and the observer could scroll through the ordered sequence using keystrokes, and could choose how long, and how frequently, to view each chart. The three other conditions embedded this task within a three-dimensional physical space. In the 3D condition, the ten Isotypes were placed in a graphical model of a physical room, with two or three of them presented on each wall. In 3D, users sat in front of a computer screen and used keystrokes to navigate from Isotype to Isotype. In VR, the same graphical room models were viewed using a head-mounted display. A joystick and standard control buttons were used to navigate from Isotype to Isotype. In MR, the users navigated in the same physical rooms used to create the graphical models used in the 3D and VR conditions. In MR, the ten Isotypes appeared superimposed on the physical space, in the same positions where they appeared in the graphical models of that space. Our companion web page https://christopheHurter.github.io/DataMemoryRecallXR/ provides a video that describes the experiment and shows the participant's visual experience in 2D, 3D, VR and MR. The "study" tab shares all four Isotype stimulus sets and provides the online experiment for 2D, 3D, and VR.

*Counterbalancing of Conditions.* In our experiments, visualization recall was measured in 4 conditions (2D, 3D, VR, MR), with 3 rooms (Rest area, Office, Openspace) and 4 stimulus sets, each containing 10 unique Isotypes. To ensure that the results were not tied to some particular aspects of the room, or its 3D model, the experiment was conducted using three different rooms. By counterbalancing the order of the technologies and the set of Isotypes viewed for each condition across subjects, any effect of fatigue or learning, that could impact recall, would
differentially affect the results. The 64 trials total within our four conditions:

2D ∈ (Set[1, 2, 3, 4] + Arrangement[Linear])
3D ∈ (Set[1, 2, 3, 4] + Rooms<sub>Virtual</sub>[1, 2, 3])
VR ∈ (Set[1, 2, 3, 4] + Rooms<sub>Virtual</sub>[1, 2, 3])
MR ∈ (Set[1, 2, 3, 4] + Rooms<sub>Physical</sub>[1, 2, 3])

Overall, we collected data from 16 participants who served in all four conditions, for a total of 64 trials. Some participants expressed fatigue or dizziness and were not able to complete all conditions. Their data were not included in the analysis.

### C. Experimental Procedures

Each observer participated in a 2 hour session where they viewed a set of 10 Isotype visualizations in four different environments (2D, 3D, VR, MR). For each set, they navigated from image to image, at their own speed and discretion, with an informal prompt from the researcher at the halfway point. After each set, they were asked to recall as many Isotypes as they could, and information about their position/location, shape and details. The session began with coffee and refreshments and an overview of the experimental design. They were told that the purpose of the experiment was to help evaluate emerging technologies. Before beginning, the experimenter explained the structure of Isotype visualizations to the participants, and how to read them. They explained that after viewing each set, the task would be to report on the information they recalled, in particular:

* The Topic: a defining feature of the visualization, e.g., the general topic, the label or the meaning of a symbol

* The Order: its position in the sequence for 2D; its location in the room for 3D, VR and MR

* The Shape: the pattern or trend of the data, e.g., ascending, descending, peaked or flat

* The Details: syntactic details about the Isotype, e.g., the axis labels, units of measure

To ensure that the observers understood the Isotype representation and the recall measures, we provided training on an Isotype visualization not used in the experiment. After viewing the visualization, there was a 2-minute delay, after which they were asked to recall the topic, shape and details. All participants were able to understand the Isotype data representation, the recall process, and to recall detailed features like units or values.

Before each condition began, participants were encouraged to adjust their seats and HMD devices so they could easily interact with them. We also ensured that the observer was comfortable with the technology and understood how to use the controls to move from image to image. We also ensured that they knew they could take a break if they experienced any dizziness or queasiness, especially for the VR and MR conditions. This familiarization process took longest for the 3D and VR conditions, especially for subjects who had limited previous experience with the VR technologies.

After training and familiarization with the technologies, each observer completed four conditions in counterbalanced order, with each observer participating in 2D, 3D, VR, and MR. As defined by our pilot study, we set the viewing time to 5 minutes. However, we decided to allow up to an additional minute if

needed to visit all Isotypes. This extra time was used in just 4 cases, and with different technologies.

After each condition, the experimenter had a casual, 2 minute conversation with the observer, to clear their short-term memory [54], followed by the recall phase, which lasted from 5 to 20 minutes, depending on the participant. In the recall phase, observers were asked to recall the ten Isotypes with their locations, shapes, and details. During the recall phase, we allowed participants to navigate in the environment they had previously experienced. However, no additional information about the isotypes was provided. They viewed empty panels showing A number from 1 to 10, instead of reviewing the Isotypes. After completing all four conditions, participants were asked to rank the 4 modalities by preference. We also interviewed the participants to capture qualitative feedback about the modalities and the overall experience.

### D. Coding of the Responses

Fig. 6 outlines the criteria we used to code our data. Recall is the percentage of the 10 Isotypes correctly recalled. Correctly identifying the topic, the label, or the icon were coded as correct responses. Order/Location is the percentage of Isotypes correctly recalled where the order (2D) or spatial location (3D, VR, MR) was also correctly recalled. For the 2D condition, we measured whether the Isotype was recalled in its correct position in the sequence; for 3D, VR, and MR, we measured whether it was recalled in its correct location in the room. This was an exact measure, and no credit was given, for example, for near misses. Shape is the percentage of the Isotypes where the shape of the data was correctly recalled. All valid descriptions of the shape of the data were coded as correct, such as “ascending, “peaked,” “trending upward,” “uniformly distributed,” etc. Details is a count, for each Isotype, of the number of additional correct details recalled. These included the recall of the country or time represented along the y axis (“axis”), recall of the icon representing the dependent variable (e.g., broccoli, book) and its numerical value (“unit”), or the number of individual data values (“single value”) or relationships (“comparisons”) recalled. For example, a participant would receive 8 detail points for the Isotype of cat population by country (bottom left in Fig. 5) as follows: 1 cat unit [+1]; sorted by countries[+1]; three countries recalled– USA [+1], Japan [+1], and France [+1]; USA shows 2 units [+1]; France 4 [+1]; Japan shows the maximum value [+1].

## V. RESULTS

This section presents our results for the four display modes: 2D, 3D, VR, and MR. For each, we counted the number of visualizations recalled (recall), the number of visualizations recalled with correct order (2D) or location (3D, VR, MR), and the number of semantic (shape), or syntactic (details) features recalled.

### A. Statistical Model

Since the count and frequency measurements for recall, order and shape are not continuous, we analyzed our data using a generalized linear model with mixed effects. We used a binomial family of models to perform the order, recall and shape analyses.
<figure><img src="image_5.png" /><figcaption>Fig. 5. One of our four sets of ten Isotype visualizations (Set 1). Each Isotype is composed of a title, a legend and quantity for each of five independent variables (time or country). Each Isotype was randomly ordered (2D) or randomly assigned to one of ten available locations in each room (3D, VR, MR), as shown in Fig. 2.</figcaption></figure>

<table><caption>Fig. 6. Coding Criteria for the four perceptual measures: Recall, order/location, shape and details.</caption><thead><tr><th>Measure</th><th>Description</th><th>Coding Criteria</th></tr></thead><tbody><tr><td>Recall</td><td>Percentage of Isotypes correctly recalled</td><td>Recall is correct if the observer correctly identifies at least, one defining feature (the general topic, the main icon, or title).</td></tr><tr><td>Order/ Location</td><td>Percentage of Isotypes correctly recalled with correct order (2D) or location (3D, VR, MR)</td><td>For 2D, order recall is correct if the observer identifies the precise position in the sequence where a correctly-recalled Isotype occurred.<br>For 3D, VR, and MR, isotype location is correct if the observer identifies the precise spatial location it occupied in the room.</td></tr><tr><td>Shape</td><td>Percentage of Isotypes whose shape was correctly recalled</td><td>Shape recall is correct if the observer correctly identifies a feature of the data's shape.<br>Examples: Ascending, Descending, Flat, Outlier, etc.</td></tr><tr><td>Details</td><td>Average number of details recalled per Isotype</td><td>Details are additional correctly-identified features of an Isotype.<br>Examples: the numerical value of an icon, the axis label, the entities compared, etc.</td></tr></tbody></table>

This was a within-subjects design, with each participant serving as his/her own control. Participants were considered as a random effect and the display mode (2D, 3D, VR, MR) as a fixed effect. While order, shape, and recall are binomial values (1 for good answer, 0 for bad), details contains the number of details the user recalled (min:0, max:13, mean 3.11). For the details analyses, we then applied a non-parametric Wilcoxon pairwise comparison between the four conditions (2D, 3D, VR, MR). We applied the standard alpha threshold of 0.05 to evaluate the effects adjusted by the Bofferoni correction (n = 8). We performed our analysis with SPSS Statistics V27.0.1. Statistical details are provided in

Fig. 8 and annotations in our graphical summary of our results in Fig. 7.

## B. Recall and Order

Fig. 7 compares our four measures of recall performance in each of the four visual environments. The left-most panel shows that the mean number of Isotype visualizations recalled (Recall) was significantly higher in MR than with 2D, 3D or VR. MR provides greatest advantage over 2D (p = 0.004), followed by 3D (p = 0.014) and then VR (p = 0.037). VR did not afford a significant advantage over 2D or 3D. The second panel (Order), shows that MR also afforded significantly better recall for the position/location of the visualizations than the other display modes. That is, when the participants were able to walk through the environment, viewing the visualizations co-located with the physical room, not only did they remember significantly more Isotypes, but they recalled significantly more information about their spatial location. Moreover, the order effects are stronger than the simple recall results, significant at the p < 0.001 level for MR versus 2D and 3D, and at the p = 0.004 level relative to performance in VR. Although VR did not provide significantly better recall for location than 3D, participant's performance in VR exceeded that of 2D, when the same visualizations were viewed simply as an ordered sequence (p = 0.045).

**Drill Down.** Fig. 9 provides histograms of observer performance at different criterion levels (x axis) for each of the four technologies tested (rows). The left-hand column provides Recall data; the right-hand column shows Recall with Order. Polynomial curves fit to the data highlight the structure. Overall, participants performed quite well in the recall condition (left-hand column). This is not surprising since the task involved recalling simple features of a relatively small set of visualizations. The MR condition afforded the highest performance level, with all participant recalling at least six of the visualizations, and six recalling all ten, suggesting a ceiling effect. We saw in Fig. 7 that
<figure><img src="image_6.png" /><figcaption>Fig. 7. Summary of the four types of recall measured for images in the four display conditions. The first panel shows the mean number of Isotype visualizations recalled (Recall), whether or not the order or location was correct. The second panel shows the mean number of visualizations recalled at the correct location (Order). The next two panels provide a deeper look into what was remembered. The third panel shows the mean of semantic features the participants remembered about the visualization (Shape), and the fourth panel captures the mean number of syntactic details participants recalled (Details). Compared with the other display modes, Mixed Reality provides significantly better Recall of Visualizations (Recall), Spatial Location (Order), Semantic Information about the meaning in the data (Shape), and Syntactic Details (Details).</figcaption></figure>

<table><caption>Fig. 8. Summary and details of our statistical results. We applied the standard alpha threshold of 0.05 to evaluate the effects adjusted by the Bonferroni correction (n = 8). The visual representation of these results is provided in Fig. 7.</caption><thead><tr><th>Condition</th><th>Results</th><th>Sig</th><th>Statistical Measures</th></tr></thead><tbody><tr><td><strong>Recall</strong></td><td></td><td></td><td>MR [Mean = 0.87, Chi-Square=65.16]</td></tr><tr><td rowspan="3">MR provided significantly better recall than 2D, 3D and VR</td><td rowspan="3">MR> [2D, 3D, VR]</td><td>*</td><td>2D [Mean = 0.73, Chi-Square=31.5, p=0.011]</td></tr><tr><td>***</td><td>3D [Mean = 0.74, Chi-Square=33.05, p=0.014]</td></tr><tr><td>*</td><td>VR [Mean = 0.76, Chi-Square=37.81, p=0.037]</td></tr><tr><td><strong>Order</strong></td><td></td><td></td><td>MR [Mean = 0.84, Chi-Square=58.55]</td></tr><tr><td rowspan="3">MR provided significantly better recall with order than 2D, 3D and VR</td><td rowspan="3">MR> [2D, 3D, VR]</td><td>***</td><td>2D [Mean = 0.55, Chi-Square=1.59, p&lt;0.001]</td></tr><tr><td>***</td><td>3D [Mean = 0.63, Chi-Square=9.78, p&lt;0.001]</td></tr><tr><td>**</td><td>VR [Mean = 0.68, Chi-Square=20.04, p=0.004]</td></tr><tr><td><strong>Order</strong></td><td></td><td></td><td>VR [Mean = 0.68, Chi-Square=20.04]</td></tr><tr><td>VR provided significantly better recall with order than 2D</td><td>VR> [2D]</td><td>*</td><td>2D Mean = 0.55, Chi-Square=1.59, p=0.045</td></tr><tr><td><strong>Shape</strong></td><td></td><td></td><td>MR [Mean = 0.71, Chi-Square=25.54]</td></tr><tr><td rowspan="3">MR provided significantly better shape recall than 3D and VR</td><td rowspan="3">MR> [2D, 3D, VR]</td><td>*</td><td>2D [Mean = 0.57, Chi-Square=3.00, p=0.039]</td></tr><tr><td>***</td><td>3D [Mean = 0.46, Chi-Square=0.9, p&lt;0.001]</td></tr><tr><td>***</td><td>VR [Mean = 0.49, Chi-Square=0.1, p&lt;0.001]</td></tr><tr><td><strong>Details</strong></td><td></td><td></td><td>MR [Mean = 3.11]</td></tr><tr><td rowspan="2">MR provided significantly better detail recall than 2D, 3D and VR</td><td rowspan="2">MR> [2D, 3D]</td><td>**</td><td>2D [Mean = 2.18, p=0.007]</td></tr><tr><td>***</td><td>3D [Mean = 1.88, p&lt;0.001]</td></tr></tbody></table>

mean performance was significantly lower for 2D than for MR. However, the histogram's fine structure reveals that, like MR, all observers recalled at least six of the ten stimuli correctly. That is, although mean performance peaked around 7, none of the observers in 2D did poorly in the recall task. That is, more observers had lower recall scores in 3D and VR, where the spatial cues were stronger than in the 2D environment. The right-hand column reveals that, with the exception of MR, recalling order was more difficult for the participants. The 2D order data are very different from the 2D recall data. The distribution of responses for order was quite flat, with many participant remembering the order of just a few visualizations. That is, in a context without spatial or kinesthetic cues, recall is high, but recall of order is degraded. The distributions for 3D and VR are also broader for the order task, with many low scores. The curve for the VR condition is particularly flat. Three participants correctly identified the order of all ten visualizations, but an equal number could locate three or fewer. Memory for order in MR, on the other hand, was similar to simple recall. All participants not only correctly remembered over 60% of the visualizations, but also remembered their spatial location.

<figure><img src="image_7.png" /><figcaption>Fig. 9. Histograms showing the number of responses at each level of performance for Recall and Order. For MR, 6 observers correctly recalled all 10 Isotypes. In MR and in 2D, all observers correctly recalled at least 6 visualizations. For Order, 5 observers remembered the order for all 10 Isotypes, and all observers correctly ordered at least 6. This level of performance was not observed for the other conditions.</figcaption></figure>

### C. Shape and Details

The right two panels of Fig. 7 are especially important for data visualization, since they ask not only whether an Isotype visualization, and its position, were recalled, but what the participant remembered about the meaning of the data and relationships
<figure><img src="image_8.png" /><figcaption>Fig. 10. Technology preference. Users ranked the technologies in order of preference. 65 percent rated MR first, and all subjects rated MR either first or second. VR was rated first by 35% of the participants, but many others rated it second or third. Almost 70 percent rated 2D in last place, and 3D was consistently rated third.</figcaption></figure>

represented in the visualization. The Shape panel shows that MR provided the best environment for recalling the semantic meaning of the data. These data plot the percentage of Isotypes where the participant recalled not only the topic of the visualization, but how the variable represented was distributed or trending. For example, did they recall that smoking per capita peaked in the 60' s and 80' s, or that Germany is the European country represented with the highest production of potatoes? Or that adoptions have been decreasing steadily in Switzerland? MR, again, provided the best environment for recall. The advantage of MR versus VR or 3D was significant at the p < 0.001 level; and showed significant advantage over 2D (p = 0.039). That is, information about the semantics of these visualizations was encoded in long term memory, and retrieved more successfully with the MR condition.

We also examined the degree to which participants recalled syntactic features of the Isotypes they had memorized. Did they recall the variable encoded along the y axis? The numerical value of the icon represented? The country or date encoded along the y axis? For each Isotype, we counted the sum of details recalled for each display mode. Mean values are shown in the Details panel (far right) in Fig. 7. The ability to recall details was greatest in the MR environment. We did not observe a significant difference between MR and VR, However, detail recall in MR outperformed 3D (p < 0.001) and 2D (p = 0.007).

In summary, recall in MR exceeded performance in the other display environments, in all other three measures. The cues provided in MR enabled better recall of the topic of the visualization, its spatial order, semantic shape, and details. The strongest advantages were observed in recalling the spatial location where the charts were viewed. Importantly, MR augmented the ability to recall the semantic meaning being represented. VR, which shares many of the same affordances of MR, provided consistently lower performance than MR. The ability to recall spatial order was greater in VR than in 2D, but no other advantages were observed.

D. User Modality Preference

After running the four conditions, we asked users to rank
the 4 modalities (2D, 3D, VR, MR) in order of preference (see
Fig. 10). Fig. 10 shows each preference level (1 is highest), and
the proportion of ratings at that level for each technology. There
was a clear preference for MR. 100% of the participants rated

MR either first (65%) or second (40%). Some users mentioned that the hololens provided an easy way to navigate in the environment and was “pleasing and fun to use”. On the opposite side of the scale, almost 70% of the participants rated 2D last. They commented that the 2D task as “boring and tedious”. VR was rate first 45% of the time, with a second peak in responses in the third position. Some complained about the navigation and the potential discomfort in VR. They also complained about the image quality in the VR environment and pointed out the lack of details in the room. Most participants rated 3D in the third position.

E. User Feedback

*Mixed Reality Hololens (MR).* In the training phase, we let users spend the time they needed to feel confident and natural with the navigation, using the floating panels (displaying only the number and not the actual Isotype visualization). The size of each floating panel was 1m50 x 1m50, so that the users facing it at a distance of 1 m would be able to see and read the whole area without being cropped by the reduced field of view. We asked users if they felt any discomfort with the reduced device field of view, but none reported an issue. No navigation issues were reported.

*Head Mounted Display (VR)*. Users reported some discomfort with the Virtual Reality device. A few experienced headaches, which is a known issue with such device. In the training session, we trained the participants in using the joystick to change point of view, since this is a known difficulty. Still, five users mentioned they found it difficult to navigate within the virtual environment in VR. For this reason, every participant went through a training phase before the experiment in which we asked them to face every floating panel (displaying only a number and not the actual Isotype visualization). While this takes time, we let every user spend enough time practicing until they felt comfortable with the environment.

Desktop 3D Experience (3D). Users all reported prior experience with 3D video games on computers or consoles. Prior to the experiment, we had participants familiarize themselves with our navigation keyboard setup: using arrow keys for forward/backward movement and turning, with an optional mouse mode for those who preferred it. This mouse mode enables users to adjust their point of view by clicking and dragging the mouse. While this mouse-based approach is common in many games, only three users opted for it. Some users encountered challenges in navigating and orienting themselves toward Isotype visualizations. To address this, we provided these users with extra time to adapt to the 3D environment navigation. Ultimately, all participants were successful in facing the floating panels.

*Desktop 2D Experience (2D).* This task, which we considered as our baseline, showed an ordered list of ten Isotypes visualizations users had to remember. No users expressed any difficulty with the navigation nor with the task to performed. Some users reported that this task was easy since this is something they are used to doing when learning charts or maps at school. Some users also reported that they applied specific memorization techniques
to improve their performance. Some memorised the first five images and then the last five, thus creating two different sets of images. One user also reported that he mentally ordered the images with icon associations and tried to build a story to better remember the image order. This memorization strategy only occurred with this 2D condition.

## VI. DISCUSSION

The goal of this research is to explore how different spatial environments affect the ability to recall information about data visualizations. We used a classic recall memory protocol, where participants memorize a set of objects, and at a later time, recall the objects and their features. We measured not only the number of Isotype visualizations recalled (Recall), but also asked them to pay attention to, and recall their spatial arrangement (Order), syntactic features (Details), e.g., units of measure, axes, etc., and most importantly, their semantic (Shape) features, which contribute to interpreting the meaning of the data. To do so, we tested recall memory for Isotype visualizations in carefully controlled 2D, 3D, VR and MR environments. We found (Fig. 7) that participants were able to recall significantly more Isotype visualizations, significantly more locations, and significantly more semantic information about the visualizations in MR than in 2D, 3D, and VR. Recall of details was also significantly higher in MR than in 2D or 3D (but not VR). 3D was significantly better than 2D for recalling Isotype position. In this section, we discuss how these four technologies could facilitate recall memory, highlighting perceptual processes that might be involved. We explore the advantages of MR technology, since it provided such strong benefit in recalling the topic, location, shape and details of the Isotypes. We also discuss why results in our VR condition did not show the same advantage. In particular, we examine the role of physical navigation and kinesthetic cues in enhancing spatial memory.

**Presence and Immersiveness.** Presence deals with the subjective feeling the user has of being transported into another environment, and can include feelings of engagement, attention and enjoyment. Presence in a VR environment can be described as the sense of being in a virtually mediated location instead of being in the real location [55] (the place the person is actually located in). Smith [1] points out that Immersiveness and Presence are often used interchangeably, but commonly, immersiveness refers to the objective aspects of the environment. Environments with greater presence subtend larger fields of view, offer stereo depth cues, and immerse the user in the visual experience. In our experiments, VR and MR provided greater presence. In both, large visual fields with high spatial resolution provided greater spatial context, and both provided stereo depth cues and egocentric navigation. However, while performance in MR exceeded recall in 2D and 3D in all measures, performance in the VR condition was not uniformly superior, despite increased presence. While VR can improve user experiences, some research reports have provided mitigating results. For instance, Roettl et al. [56] showed in the context of video games, that recall for embedded brand logos was better in the 2D/3D than in VR conditions.

**Situated Analytics and Spatial Embodiment.** The concept of Situated Analytics refers to the usage of data representations in relation to objects, places, or persons in the physical world for the purpose of improved reasoning [57]. A tight relationship between data and its environment, i.e., Spatial Embodiment, can help in supporting spatial reasoning [21]. Tasks requiring not just object recall, but recall of location, would be expected to benefit from this spatial embodiment. Superior performance in our MR condition, especially for recalling order, is consistent with this explanation. Krokos et al. [6] demonstrated that people immersed in VR better recalled images of people compare to 3D or 2D. We expected these empirical results to transfer from photographs to data visualization, as both rely on memorizing pictorial information. Like faces, Isotype visualizations, with their pictorial icons, provide strong recognition memory [15], which was a factor in our selecting these as our stimuli. However, in the simple recall condition, which is most comparable to the face recall task in Krokos et al. performance in VR was not better than in 2D or 3D. This may be because in that experiment, participants were only provided a HMD to control changes in viewpoint. In our VR condition, we provided participants with controls to change their position and viewpoint in the virtual world, providing a richer 3D spatial experience. We did find that recalling the order of the visualizations was significantly better in VR than in 2D, which suggests that participants in both experiments were making use of spatial cues in VR, which were not present in the 2D.

**Physical Navigation and Kinesthetic Memory.** Experiencing one's body and its movement may improve recall by helping the observer encode information through physical movement and its associated kinesthetic cues [34]. In our experiments, the degree of kinesthetic contribution varied from condition to condition. In the 2D case, the visualizations were simple screen images and there were no 3D or kinesthetic cues. In the 3D condition, the observer could change their point-of-view, and perhaps imagine moving, but there were no explicit movement or kinesthetic cues. In the VR condition, the user could navigate virtually to each isotype location and move their heads to navigate. The projection of the 3D room changed as if the user were walking, providing implicit kinesthetic cues, since the optical flow matched the movement path. The MR condition provided explicit kinesthetic cues, since the observer was free to locomote freely through the physical space. For the three 3D conditions, performance tended to improve monotonically as the level of kinesthetic experience increased. These differences, however, were not all significant. Performance for recall, order, and shape was significantly better for MR than for the other technologies, and significantly better than 3D for details. One possible explanation of these results is that the act of physically moving through space enabled the observers to take advantage of their kinesthetic memory while examining the objects, and that these kinesthetic cues might also have been reinforced as they moved through space, again, to recall information at specific locations. This hypothesis is consistent with our results, but is not conclusive. To test the kinesthetic hypothesis, it would be valuable, in future research, to measure recall for visualizations in a Virtual Reality environment where the observer could locomote freely, adding an
explicit kinesthetic component to the exploration. It would also be interesting to provide controlled levels of locomotion in VR and MR. Experiments like Vielledent et al. [34], who explored the parameter space for visual imagery in memory, provides a useful model for studying the effects of kinesthetic memory in visualization.

*The Role of Location in Memory Recall.* The method of loci (memory palace) phenomenon demonstrates remarkable increases in recall if objects are linked to specific spatial locations [12]. In real-world situations, spatial location plays a large role in directing behavior. This effect is so strong that grocery stores routinely rearrange product positions to disrupt the mapping between an item and its remembered spatial location, forcing shoppers to visit more aisles and hopefully add additional items to their carts [58]. Virtual shopping applications typically provide hierarchical product search, but some are starting to provide spatial metaphors and virtual shopping experiences [59]. With visualization systems, information can be placed in arbitrary spatial locations on a screen or a wall. Recent guidance for creating data dashboards recommends spatially-consistent configurations to aid users with navigation and data reasoning [60]. The role of spatial linking has been demonstrated for remembering faces [6] and our experiments extend the exploration of memory and spatial location for data visualization in 3D, VR and MR.

## VII. FUTURE RESEARCH DIRECTIONS

*Beyond Isotypes.* In this study, we limited our material to Isotype visualizations. We selected this data representation because they are composed of graphical entities (icons) that directly represent physical objects. Future work could explore other types of visualization, composed of more abstract shapes, such as scatterplots, networks, 3-D geographic models, etc, to investigate whether our results with Isotypes generalizes to the memorization and recall of other visualization type.

*Extending VR.* In our experiments, the 3D and VR environments were created to match the room in MR, making them as close to reality as possible. In this configuration, MR fostered better recall than 3D and VR. Although these digital models were similar to the real room, both in shape and in texture, there were unavoidable sampling errors. Future work could certainly investigate creating even more accurate renderings, which might provide more engaging and potentially memorable experiences. In our experiments, participants in the VR condition were seated with a head-mounted display, and navigated through the room via a joystick, keystrokes and head motion. For future work, we could repeat these experiments using a VR paradigm that afforded full movement, allowing the participant to physically move through the space using full kinesthetic motion cues. In addition, future experiments could tap more fully into the potential of VR. For example, in our experiments, the Isotypes in 3D, VR and MR were akin to billboards. In VR, they could be transformed into fields of virtual objects that people could walk into [61]. Also, VR can dramatically change what the reality is, potentially providing more magical and perhaps more memorable environments. VR can also incorporate dramatically

different virtual spaces, such as transporting users to different locations [62], providing more effective memory palaces. VR might also be capable of enhancing memory by changing the viewers' perception of their body [63], or by providing different navigation experiences such as jumping or teleportation [64].

*Extending MR.* Our experiments controlled the visual aspects of the environment. Future research could explore how other sensory information might contribute. This could be accomplished by running a Physical Reality condition, where physical billboards of Isotypes would be placed in a physical room. We can also envision experiments in which participants could hear or smell or touch an aspect of the environment, which could help them anchor a particular memory to a particular location; novel sonification [65], olfactory [66] or even taste [67] cues could potentially produce even stronger recall.

*Extended Experimental Conditions and Controls.* When designing our experiments, we considered using supplementary cognitive tests. For example, participants could be tested for spatial reasoning. Cognitive load could be tested using NASA-TLX. Since the experiment was already long and challenging, we did not include these measures. Recording physiological measurements (i.e., electroencephalogram (EEG), Functional near-infrared spectroscopy (fNIRS), and eye tracking [68]) could also be an interesting direction for future work.

*Future Directions for Spatial Reasoning.* Our work provides valuable insights into the factors influencing the impact of locomotion, guiding future research in this area. Locomotion Vault [46] shows many opportunities for testing various locomotion paradigms in VR, which would be interesting for future research. Another interesting direction for future research would be to examine individual differences in spatial reasoning. Mayer and Sims [69] argued that high spatial ability learners should benefit more from 3D environments since they would require less cognitive effort to perform the tasks. In our data, all participants recalled the order of at least 6 of the 10 Isotypes in MR, and 5 recalled the order of all 10. Future research could test whether these subjects who seemed to benefit so strongly from the MR environment, indeed, had higher levels of spatial reasoning.

## VIII. CONCLUSION

This research investigates how different display modes affect how we encode and recall the information in data visualizations. We constructed four viewing modalities for displaying Isotype visualizations. In 2D, the visualizations were presented in an ordered sequence on a 2D display. In 3D, VR and MR, visualizations were anchored in the same 3D environment, which was either virtual or physical. In MR, participants outperformed the other techniques in simple recall (the number of visualization remembered), in recall of the spatial location where the Isotype appeared, and in recall of the semantic shape of the data. There was no significant difference in recall of syntactic details between VR and MR, but we observed significantly higher detail recall in MR versus 2D or 3D. Recall of order was significantly better in VR than in 2D, but no other advantages of VR were uncovered. MR was also the most highly preferred environment.
This research highlights the importance of advanced immer-
sive technologies for visual data analysis. We showed that the
MR environment not only increased simple recall of Isotypes,
but provided superior performance in remembering the spatial
locations of the data, and the analytical meaning portrayed.
There is a clear link between remembering location and remem-
bering key information about the data, such as whether there is
a descending or descending trend, a peak or an outlier. We think
this result is promising, especially for situations where advanced
technologies might be used to support analytical thinking [42].
For example, many spatially-oriented applications currently rely
on VR such as pilot training or managing a complex control
system. Allowing the analyst to move freely through an MR
environment, providing key information superimposed visually
in key locations, might enhance recall of critical data and rela-
tionships. Another promising avenue would be to enhance data-
driven storytelling [70], where the goal is to engage the audience
and communicate memorable sequences of information.

Finally, this work highlights the role of moving through space, kinesthetic motion, for improving recall. Teaching activities in front of a computer can miss utilizing important human skills associated with kinesthetic motion. In this sense, this work extends the usages of MR technologies for educational purposes, using user motion to improve data exploration, understanding and recall. In our experiments, Mixed Reality provided superior recall of visualization objects, their locations, structure and meaning. These findings pave the way for future research into the use of MR in visualization applications where data are spatially arrayed, or where remembering order is important.

ACKNOWLEDGMENT

Additionally, Christophe Hurter personally acknowledges
Microsoft Research, Redmond, Washington, for insightful dis-
cussions and its two fruitful visiting periods.

REFERENCES

[1] S. A. Smith, "Virtual reality in episodic memory research: A review," *Psychon. Bull. Rev.*, vol. 26, pp. 1213-1237, 2019.

[2] P. Araiza-Alba, T. Keane, W. S. Chen, and J. Kaufman, "Immersive virtual reality as a tool to learn problem-solving skills," *Comput. Educ.*, vol. 164, 2021, Art. no. 104121.

[3] O. Neurath, "International picture language," in *Department of Typography and Graphic Communication*. Hassell Street Press, 1936.

[4] S. Haroz, R. Kosara, and S. L. Franconeri, "Isotype visualization: Working memory, performance, and engagement with pictographs," in *Proc. ACM SIGCHI Conf. Hum. Factors Comput. Syst.*, 2015, pp. 1191-1200.
[Online]. Available: https://doi.org/10.1145/2702123.2702275

[5] G. Robertson, M. Czerwinski, K. Larson, D. C. Robbins, D. Thiel, and M. Van Dantzich, "Data mountain: Using spatial memory for document management," in *Proc. 11th Annu. ACM Symp. User Interface Softw. Technol.*, 1998, pp. 153-162.

[6] E. Krokos, C. Plaisant, and A. Varshney, "Virtual memory palaces: Im-
mersion aids recall," *Virtual Reality*, vol. 23, no. 1, pp. 1-15, 2019.

[7] H. Ebbinghaus, "Memory: A contribution to experimental psychology," *Ann. Neurosciences*, vol. 20, no. 4, pp. 155-156, 2013.

[8] Rhetorica ad herennium, english translation by harry caplan (loeb), 1954.
Accessed: Mar. 28, 2022. [Online]. Available: https://www.laits.utexas.edu/memoria/Ad_Herennium_Passages.html

[9] A. R. Luria, *The Mind of a Mnemonist: A Little Book about a Vast Memory, With a New Foreword by Jerome S. Bruner*. Cambridge, MA, USA: Harvard Univ. Press, 1987.

[10] F. A. Yates, *The Art of Memory*. Chicago, IL, USA: Univ. Chicago Press, 1966.

[11] E. Tulving and D. L. Schacter, "Priming and human memory systems," *Science*, vol. 247, no. 4940, pp. 301-306, 1990.

[12] E. A. Maguire, E. R. Valentine, J. M. Wilding, and N. Kapur, "Routes to remembering: The brains behind superior memory," *Nature Neurosci.*, vol. 6, no. 1, pp. 90-95, 2003.

[13] E. L. Legge, C. R. Madan, E. T. Ng, and J. B. Caplan, "Building a memory palace in minutes: Equivalent memory performance using virtual versus conventional environments with the method of loci," *Acta Psychologica*, vol. 141, no. 3, pp. 380-390, 2012. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S000169181200145X

[14] S. Bateman, R. L. Mandryk, C. Gutwin, A. Genest, D. McDine, and C. Brooks, "Useful junk? The effects of visual embellishment on comprehension and memorability of charts," in *Proc. SIGCHI Conf. Hum. Factors Comput. Syst.*, 2010, pp. 2573-2582.

[15] M. A. Borkin et al., "What makes a visualization memorable?," *IEEE Trans. Vis. Comput. Graph.*, vol. 19, no. 12, pp. 2306-2315, Dec. 2013.

[16] H. Li and N. Moacdieh, "Is "chart junk" useful? An extended examination of visual embellishment," in *Proc. Hum. Factors Ergonom. Soc. Annu. Meeting*, SAGE Publications Sage CA, 2014, pp. 1516-1520.

[17] B. Saket, C. Scheidegger, S. G. Kobourov, and K. Börner, "Map-based visualizations increase recall accuracy of data," *Comput. Graph. Forum*, vol. 34, no. 3, pp. 441-450, 2015.

[18] M. A. Borkin et al., "Beyond memorability: Visualization recognition and recall," *IEEE Trans. Visual. Comput. Graph.*, vol. 22, no. 1, pp. 519-528, Jan. 2016.

[19] S. Ghani and N. Elmqvist, "Improving revisitation in graphs through static spatial features," in *Proc. Graph. Interface*, Waterloo, Canada:Canadian Human-Computer Communications Society, 2011, pp. 175-182.

[20] B. E. Alper, N. Henry Riche, and T. Hollerer, "Structuring the space: A study on enriching node-link diagrams with visual references," in *Proc. SIGCHI Conf. Hum. Factors Comput. Syst.*, New York, NY, USA, 2014, pp. 1825-1834. [Online]. Available: https://doi.org/10.1145/2556288.2557112

[21] W. Willett, Y. Jansen, and P. Dragicevic, "Embedded data representa-
tions," *IEEE Trans. Vis. Comput. Graph.*, vol. 23, no. 1, pp. 461-470,
Jan. 2017.

[22] L. Shapiro, *Embodied Cognition*. Evanston, IL, USA: Routledge, 2019.

[23] D. S. Tan, R. Pausch, J. K. Stefanucci, and D. R. Proffitt, "Kinesthetic cues aid spatial memory," in *Proc. Extended Abstr. Hum. Factors Comput. Syst.*, 2002, pp. 806-807.

[24] J. J. Gibson, *The Perception of the Visual World*. Boston, MA, USA: Houghton Mifflin, 1950.

[25] J. J. Gibson, *The Ecological Approach to Visual Perception: Classic Edition*. London, U.K.: Psychology Press, 2014.

[26] J. J. Gibson, "The myth of passive perception: A reply to richards, "Philosophy Phenomenological Res., vol. 37, no. 2, pp. 234-238, 1976.

[27] D. A. Norman, "Affordance, conventions, and design," *Interactions*, vol. 6, no. 3, pp. 38-43, 1999.

[28] B. Lee, D. Brown, B. Lee, C. Hurter, S. Drucker, and T. Dwyer, "Data visceralization: Enabling deeper understanding of data using virtual reality," *IEEE Trans. Vis. Comput. Graph.*, vol. 27, no. 2, pp. 1095-1105, Feb. 2021.

[29] B. M. Brooks, "The specificity of memory enhancement during interaction with a virtual environment," *Memory*, vol. 7, no. 1, pp. 65-78, 1999. [Online]. Available: https://doi.org/10.1080/741943713

[30] M. Wilson, "Six views of embodied cognition," *Psychon. Bull. Rev.*, vol. 9, pp. 625-636, 2002.

[31] P. Robbins and M. A. Editors, *The Cambridge Handbook of Situated Cognition*. Cambridge, U.K.: Cambridge Univ. Press, 2008.

[32] M. Wilson, "The case for sensorimotor coding in working memory," *Psychon. Bull. Rev.*, vol. 8, pp. 44-57, 2001. [Online]. Available: http://doi.org/10.3758/BF03196138

[33] E. Tulving, "What is episodic memory?," *Curr. Directions Psychol. Sci.*, vol. 2, no. 3, pp. 67-70, 1993.

[34] S. Vieilledent, S. M. Kosslyn, A. Berthoz, and M. D. Giraudo, "Does mental simulation of following a path improve navigation performance without vision?," *Cogn. Brain Res.*, vol. 16, no. 2, pp. 238-249, 2003.

[35] P. H. Vicki Bruce, *Mike Burton, Remembering Faces*. Evanston, IL, USA: Routledge, 2006.

[36] D. Waller and E. Hodgson, "Sensory contributions to spatial knowledge of real and virtual environments," in *Human Walking in Virtual Environments*, Berlin, Germany: Springer, 2013, pp. 3-26.
[37] R. A. Ruddle, *The Effect of Translational and Rotational Body-Based Information on Navigation*, New York, NY, USA: Springer, 2013, pp. 99–112. [Online]. Available: https://doi.org/10.1007/978-1-4419-8432-6_5

[38] F. Gaunet, M. Vidal, A. Kemeny, and A. Berthoz, "Active, passive and snapshot exploration in a virtual environment: Influence on scene memory, reorientation and path memory," *Cogn. Brain Res.*, vol. 11, no. 3, pp. 409–420, 2001.

[39] T. Friedrich, A. Prouzeau, and M. McGuffin, "The effect of increased body motion in virtual reality on a placement-retrieval task," in *Proc. 27th ACM Symp. Virtual Reality Softw. Technol.*, New York, NY, USA, 2021, Art. no. 14. [Online]. Available: https://doi.org/10.1145/3489849.3489888

[40] F. Yang, J. Qian, J. Novotny, D. Badre, C. D. Jackson, and D. H. Laidlaw, "A virtual reality memory palace variant aids knowledge retrieval from scholarly articles," *IEEE Trans. Vis. Comput. Graph.*, vol. 27, no. 12, pp. 4359-4373, Dec. 2021.

[41] J. Vindenes, A. O. d. Gortari, and B. Wasson, "Mnemosyne: Adapting the method of Loci to immersive virtual reality," in *Proc. Int. Conf. Augmented Reality Virtual Reality Comput. Graph.*, Springer, 2018, pp. 205–213.

[42] J. McCormack et al., *Multisensory immersive analytics*. Berlin, Germany: Springer, 2018, pp. 57–94. [Online]. Available: https://doi.org/10.1007/978-3-030-01388-2_3

[43] K. Y. Lim and R. Lim, "Semiotics, memory and augmented reality: History education with learner-generated augmentation," *Brit. J. Educ. Technol.*, vol. 51, no. 3, pp. 673–691, 2020.

[44] J. J. Gibson, *The Ecological Approach to Visual Perception*, Psychology Press, 1986.

[45] M. Al Zayer, P. MacNeilage, and E. Folmer, "Virtual locomotion: A survey," *IEEE Trans. Vis. Comput. Graph.*, vol. 26, no. 6, pp. 2315–2334, Jun. 2020.

[46] M. Di Luca, H. Seifi, S. Egan, and M. Gonzalez-Franco, "Locomotion vault: The extra mile in analyzing VR locomotion techniques," in *Proc. CHI Conf. Hum. Factors Comput. Syst.*, New York, NY, USA, 2021, Art. no. 128. [Online]. Available: https://doi.org/10.1145/3411764.3445319

[47] V. Nader and D. Camp, Eds., *Otto Neurath: The Language of the Global Polis*. Rotterdam, The Netherlands: NAi Publishers, 2008.

[48] F. Chevalier, R. Vuillemot, and G. Gali, "Using concrete scales: A practical framework for effective visual depiction of complex measures," *IEEE Trans. Vis. Comput. Graph.*, vol. 19, no. 12, pp. 2426-2435, Dec. 2013.

[49] H. Xia, N. Henry Riche, F. Chevalier, B. De Araujo, and D. Wigdor, "DataInk: Direct and creative data-oriented drawing," in *Proc. CHI Conf. Hum. Factors Comput. Syst.*, New York, NY, USA, 2018, pp. 1–13. [Online]. Available: https://doi.org/10.1145/3173574.3173797

[50] Z. Liu et al., "Data illustrator: Augmenting vector design tools with lazy data binding for expressive visualization authoring," in *Proc. CHI Conf. Hum. Factors Comput. Syst.*, New York, NY, USA, 2018, pp. 1–13. [Online]. Available: https://doi.org/10.1145/3173574.3173697

[51] J. E. Zhang, N. Sultanum, A. Bezerianos, and F. Chevalier, "DataQuilt: Extracting visual elements from images to craft pictorial visualizations," in *Proc. CHI Conf. Hum. Factors Comput. Syst.*, New York, NY, USA, 2020, pp. 1–13. [Online]. Available: https://doi.org/10.1145/3313831.3376172

[52] G. A. Miller, "The magical number seven, plus or minus two: Some limits on our capacity for processing information," *Psychol. Rev.*, vol. 63, no. 2, pp. 81–97, Mar. 1956. [Online]. Available: http://www.musanim.com/miller1956/

[53] A. D. Baddeley and G. Hitch, "Working memory," *Psychol. Learn. Motivation*, vol. 8, pp. 47–89, 1974.

[54] R. C. Atkinson and R. M. Shiffrin, "Human memory: A proposed system and its control processes," *Psychol. Learn. Motivation*, vol. 2, pp. 89–195, 1968.

[55] M. V. Sanchez-Vives and M. Slater, "From presence to consciousness through virtual reality," *Nature Rev. Neurosci.*, vol. 6, no. 4, pp. 332–339, 2005.

[56] J. Roettl and R. Terlutter, "The same video game in 2D, 3D or virtual reality–how does technology impact game evaluation and brand placements?," *PLoS One*, vol. 13, no. 7, 2018, Art. no. e0200724.

[57] N. ElSayed, B. Thomas, K. Marriott, J. Piantadosi, and R. Smith, "Situated analytics," in *Proc. Big Data Vis. Analytics*, 2015, pp. 1–8.

[58] J. S. Larson, E. T. Bradlow, and P. S. Fader, "An exploratory look at supermarket shopping paths," *Int. J. Res. Marketing*, vol. 22, no. 4, pp. 395–414, 2005.

[59] A.-C. Jeandrain, "Consumer reactions in a realistic virtual shop: Influence on buying style," *J. Interactive Advertising*, vol. 2, no. 1, pp. 2–9, 2001.

[60] Z. Qu and J. Hullman, "Keeping multiple views consistent: Constraints, validations, and exceptions in visualization authoring," *IEEE Trans. Vis. Comput. Graph.*, vol. 24, no. 1, pp. 468–477, Jan. 2018.

[61] H. Romat, N. H. Riche, C. Hurter, S. Drucker, F. Amini, and K. Hinckley, "Dear pictograph: Investigating the role of personalization and immersion for consuming and enjoying visualizations," in *Proc. CHI Conf. Hum. Factors Comput. Syst.*, New York, NY, USA, 2020, pp. 1–13. [Online]. Available: https://doi.org/10.1145/3313831.3376348

[62] N. de la Peña et al., "Immersive journalism: Immersive virtual reality for the first-person experience of news," *Presence: Teleoperators Virtual Environ.*, vol. 19, no. 4, pp. 291–301, Aug. 2010. [Online]. Available: https://doi.org/10.1162/PRES_a_00005

[63] M. Slater, B. Spanlang, M. V. Sanchez-Vives, and O. Blanke, "First person experience of body transfer in virtual reality," *PLoS One*, vol. 5, no. 5, 2010, Art. no. e10564.

[64] H. Xia, S. Herscher, K. Perlin, and D. Wigdor, "SpaceTime: Enabling fluid individual and collaborative editing in virtual reality," in *Proc. 31st Annu. ACM Symp. User Interface Softw. Technol.*, 2018, pp. 853–866.

[65] H. Zhao, C. Plaisant, and B. Shneiderman, ""I hear the pattern" interactive sonification of geographical data patterns," in *Proc. Extended Abstr. Hum. Factors Comput. Syst.*, 2005, pp. 1905–1908.

[66] B. Patnaik, A. Batch, and N. Elmqvist, "Information olfactionation: Harnessing scent to convey data," *IEEE Trans. Visual. Comput. Graph.*, vol. 25, no. 1, pp. 726–736, Jan. 2019.

[67] Y. Wang, X. Ma, Q. Luo, and H. Qu, "Data edibilization: Representing data with food," in *Proc. CHI Conf. Extended Abstr. Hum. Factors Comput. Syst.*, New York, NY, USA, 2016, pp. 409–422. [Online]. Available: https://doi.org/10.1145/2851581.2892570

[68] T. Andry, C. Hurter, F. Lambotte, P. Fastrez, and A. Telea, "Interpreting the effect of embellishment on chart visualizations," in *Proc. CHI Conf. Hum. Factors Comput. Syst.*, New York, NY, USA, 2021, Art. no. 613. [Online]. Available: https://doi.org/10.1145/3411764.3445739

[69] R. E. Mayer and V. K. Sims, "For whom is a picture worth a thousand words? Extensions of a dual-coding theory of multimedia learning," *J. Educ. Psychol.*, vol. 86, pp. 389–401, 1994.

[70] N. H. Riche, C. Hurter, N. Diakopoulos, and S. Carpendale, *Data-Driven Storytelling*, 1st ed. Natick, MA, USA: A. K. Peters, 2018.

<figure><img src="image_9.png" /></figure>

**Christophe Hurter** is a professor with the University of Toulouse, France, leading the Interactive Data Visualization Group (DataVis), French Civil Aviation University (ENAC). His research covers Big Data manipulation and visualization, immersive analytics, explainable AI, and human-computer interaction.

<figure><img src="image_10.png" /></figure>

**Bernice Rogowitz** received the PhD degree in human vision from Columbia University. She is the founding scientist with Visual Perspectives and editor-in-chief of the *Journal of Perceptual Imaging*. Her research focuses on how we process and synthesize information. Her work in color vision, spatial organization, spatial-temporal interactions and memory has been inspired by real-world applications of visualization and imaging technologies.

<figure><img src="image_11.png" /></figure>

**Guillaume Truong** received the master's degree of human computer interaction, in 2017. He is a research engineer working since 2018 with the French Civil Aviation University. He works as a data scientist and expert web designer.
<figure><img src="image_12.png" /></figure>

**Tiffany Andry** is a professor of Communication and Digital Societies, ULB (Université libre de Bruxelles). She conducted research on the appropriation of data visualizations in information and communication sciences with the Social Media Lab (UCLouvain) and Elico (Université Lumière Lyon 2). At the crossroads of disciplines (communication sciences, infovis, HCI), she brings together qualitative and quantitative methods.

<figure><img src="image_13.png" /></figure>

**Hugo Romat** received the PhD degree from the Université Paris-Saclay, in 2019. He then worked as a postdoctoral researcher with ETH Zurich, and is now a research scientist with Microsoft. His research interests include information visualization and novel input devices with new sensing capabilities to provide new interaction techniques.

<figure><img src="image_14.png" /></figure>

**Ludovic Gardy** received the PhD degree in the field of neuroscience, in 2020. He is an independent data scientist. He has developed novel techniques to analyze brain activity, particularly focused on detecting the brain regions associated with seizures in epileptic patients.

<figure><img src="image_15.png" /></figure>

**Fereshteh Amini** received the MSc and PhD degree from the University of Manitoba in computer science studying with the Human Computer Interaction Research Group. She is a senior applied scientist with the M365 Data Science Team, Microsoft and an instructor with UC Berkeley's data science program teaching a course on data visualization and communication.

<figure><img src="image_16.png" /></figure>

**Nathalie Henry Riche** is a principal researcher with the EPIC Group (Extended Perception, Interaction and Cognition) led by Ken Hinckley. She envisions a future in which all of our devices will work together to help us create, think and communicate. Her work spans technical and fundamental contributions to cross-device interaction, digital inking, information visualization, and storytelling.