# The State of the Art of Spatial Interfaces for 3D Visualization

Lonni Besançon, Anders Ynnerman, Daniel F Keefe, Lingyun Yu, Tobias Isenberg

<figure><img src="image_1.png" /></figure>

## To cite this version:

Lonni Besançon, Anders Ynnerman, Daniel F Keefe, Lingyun Yu, Tobias Isenberg. The State of the Art of Spatial Interfaces for 3D Visualization. Computer Graphics Forum, 2021, 40 (1), pp.293-326. 10.1111/cgf.14189. hal-03012861

HAL Id: hal-03012861

https://inria.hal.science/hal-03012861

Submitted on 18 Nov 2020

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# The State of the Art of Spatial Interfaces for 3D Visualization

Lonni Besançon,<fnref n="1" />,<fnref n="2" /> Anders Ynnerman,<fnref n="1" /> Daniel F. Keefe,<fnref n="3" /> Lingyun Yu,<fnref n="4" /> and Tobias Isenberg<fnref n="5" />

## Abstract

We survey the state of the art of spatial interfaces for 3D visualization. Interaction techniques are crucial to data visualization processes and the visualization research community has been calling for more research on interaction for years. Yet, research papers focusing on interaction techniques, in particular for 3D visualization purposes, are not always published in visualization venues, sometimes making it challenging to synthesize the latest interaction and visualization results. We therefore introduce a taxonomy of interaction technique for 3D visualization. The taxonomy is organized along two axes: the primary source of input on the one hand and the visualization task they support on the other hand. Surveying the state of the art allows us to highlight specific challenges and missed opportunities for research in 3D visualization. In particular, we call for additional research in: (1) controlling 3D visualization widgets to help scientists better understand their data, (2) 3D interaction techniques for dissemination, which are under-explored yet show great promise for helping museum and science centers in their mission to share recent knowledge, and (3) developing new measures that move beyond traditional time and errors metrics for evaluating visualizations that include spatial interaction.

## CCS Concepts

*Human-centered computing → Human computer interaction (HCI); Visualization*

## 1. Introduction

The visualization research community has long recognized the importance of user interface research and the special role that interactive techniques can play in data visualization processes. Over the years, calls for additional research on interactive techniques have been raised repeatedly, highlighting the critical and foundational role of interaction within the visualization communities that focus on both non-spatial data (e.g., [Rhe02, TM04, CT05, YKSJ07]) and spatial (often 3D) data (e.g., [Sut66, Hib99, Rhe02, Joh04, TM04, Kee10, BDP11, LK11, KI13, Mun14, CSVBS15, FCC*15]). However, more study of vis-centric interaction is needed. Our specific interest in this survey is spatial 3D data. While interactive systems and techniques are certainly published at visualization venues, we have noticed that research papers that introduce new interaction techniques for exploring, filtering, selecting, or otherwise manipulating 3D data are frequently published at non-visualization venues, so that visualization researchers may not always learn about them. We hope to bridge this gap, paying special attention to spatial user interfaces. We believe there is significant potential to make 3D interactive visualization systems more effective by leveraging new readily available sensing technologies [Bes17, LKM*17] and adapting 3D interaction techniques developed in other contexts [JH13] to work for the spe-

cial needs of interactive data visualization tasks. Such an approach would make use of the skills to interact with the physical 3D world that people naturally possess, and, thereby, have potential for great positive impact since so many important datasets have an inherent 3D structure: data acquired from simulations as well as spatial data, medical data, or biological data. To contribute to this future, this state of the art report surveys the spatial 3D interaction techniques that have been presented in the literature, presents a task-based framework for guiding new research on vis-specific spatial 3D interaction techniques, and repeats the call for additional research on spatial 3D interfaces specifically to support 3D visualization tasks.

Spatial 3D datasets are particularly challenging to visualize. Unlike general 3D interaction, visualization of 3D datasets is less focused on creation than it is on sense-making. Making sense of 3D datasets requires an ability to manipulate the data or the view, to select in 3D specific regions of interest, and to place and manipulate visualization widgets to better understand the inherent structure of the dataset or some of its internal properties. While 3D interaction techniques address some of these challenges on pre-defined objects, 3D visualization techniques should enable users to achieve all operations on non-predefined structures. This additional requirement is not satisfied by most of the classical 3D interaction techniques when

<footnote n="1">¹ Linköping Universitet, Norrköping, Sweden. *email:* lonni.besancon@gmail.com & anders.ynnerman@liu.se</footnote>

<footnote n="2">² Faculty of Information Technology, Monash University, Australia</footnote>

<footnote n="3">³ University of Minnesota, USA. *email:* dfk@umn.edu</footnote>

<footnote n="4">⁴ Xi’an Jiaotong-Liverpool University, China. *email:* mail@yulingyun.com</footnote>

<footnote n="5">⁵ Université Paris-Saclay, CNRS, Inria, LRI, France. *email:* tobias.isenberg@inria.fr</footnote>
used in a spatial visualization context. Moreover, 3D interaction, such as manipulation, selection and annotation, becomes more challenging when applied to complex features or structures of 3D VIS datasets, especially when more precise interaction is needed. For instance, selection of neural fibers becomes more difficult since they are a lot thinner and denser than the objects that are used to develop more generic 3D selection techniques. Similarly, annotation is more challenging when those annotations need to be linked precisely to a 3D volumetric context rather than just recorded as a Voiceover.

For the purpose of this report, we characterize *spatial 3D interaction techniques for data visualization* as post-WIMP user interaction techniques that employ tangible interaction proxies, tracked gestures, and/or 3D input devices to enable users to better leverage natural, human skills for working with data visualized in 3D spaces. This notion closely relates to the term *3D interaction*, which is included as a keyword in several past surveys. Our report is unique in its combination of (1) focusing on interaction techniques to support data exploration tasks and (2) surveying multiple classes of spatial interaction techniques. We discuss prior work on both visualization-specific interaction techniques and more generic 3D interaction techniques. The latter have traditionally appeared at venues such as IEEE VR (which merged with IEEE 3DUI), ACM CHI, ACM I3D (especially, in the early years of the conference), ACM SUI, ACM UIST, IEEE ISMAR, and sometimes also ACM ISS, which only a small portion of the visualization research community regularly attends. Thus, an important contribution of our report is to bring the results from these communities together within a single document.

Past surveys on 3D interaction techniques [Han97, LKM*17, JH13, JH14, LKM*17, MCG*19] have focused on the generic tasks for 3D interactions—namely selection, manipulation, navigation, and system control—but not on specific tasks that are of paramount importance for visualization applications such as 3D picking/seLECTIONs [Wil96b], concurrent manipulation of data and exploration objects, specification of 3D primitives for seeding or path planning, temporal navigation etc. Hand's survey [Han97], written in 1997, covers just the important early work in this area, while Christie et al. [CON08] focused exclusively on camera control. Other reviews focus on specific interaction paradigms. For example, Paneels and Roberts' review of haptic data visualization [PR10] discussed solely how data can be visualized or perceived through haptic interaction. The survey by Groenewald et al. [GAI*16] only covered 3D control with mid-air gestures. Another relatively recent survey of 3D inter-action techniques by Jankowski and Hachet [JH13, JH14] placed the focus on generic 3D manipulation with mouse-based and touch-based systems. This is also the focus of the more recent work from Mendes et al. [MCG*19]. Finally, some authors focused on 3D data visualization but did not address 3D interaction. For instance, the survey from Oeltze-Jafra et al. [OJMN*19] focuses on medical data generation and its analysis without highlighting the large body of work done on interactive visualization tasks for 3D datasets.

Our paper is focused on spatial interfaces, which include inter-action paradigms such as: tangible interaction (e. g., [HPGK94, DGHCM*03, SGF*11, JLS*13, IGA14b, HRD*19]), mid-air ges-tures (e. g., [GPC11, TTSI*12, LGK*13, NHH*16]), haptic inter-faces (e. g., [LCP*07, PCT*07]), or hybrid interaction paradigms (i. e., combining several input technologies; e. g., [PTW98, OBF03,

SSSS11, BIAI17a]). These spatial interaction paradigms provide several theoretical advantages: tangible interaction mimics everyday interaction with the real-world [Fit96, BIAI17b], mid-air gestures enable hands-free interaction with medical data during surgeries [NHH*16], and hybrid interaction leverages the benefits of multiple interaction paradigms [Bes17].

We organized our state of the art report as follows. In Sect. 2 we define the interactive tasks users must perform with visualizations. In Sect. 3 we present the actual survey of the literature, using the tasks defined in Sect. 2 as an organizing principle. Finally, in Sect. 4, we discuss opportunities for future work that result from our review.

## 2. Defining a classification system

Before surveying the spatial interaction literature, it is important to have a common understanding of both the interactive tasks users need to perform with visualizations (e.g., view manipulation, working with widgets, data selection) and the major interaction paradigms (e.g., tactile, tangible, and mid-air) that are possible with spatial interaction techniques. These two topics form the two axes of the classification system used for the survey presented in Section 3.

### 2.1. Axis 1: Spatial interaction paradigms

The first axis is the spatial interaction paradigm. A variety of spatial interaction paradigms have been investigated for both 3D manipulations and visualization-specific interaction techniques; we focus in particular on *tactile/touch interaction*, *tangible interaction*, *mid-air gestural interaction*, and *hybrid interaction*, i. e., interaction techniques combining several interaction paradigms, since these paradigms are most readily supported by current spatial interface hardware. While *voice input* could also be considered, using voice for direct manipulation is generally discouraged [KI13] and it is seldom used alone. Consequently, voice input falls under our category of hybrid interaction paradigms.

#### 2.1.1. Paradigm 1: Tactile and pen-based interaction

Sutherland's Sketchpad [Sut64], created in the 1960s, used a light-pen to interact on a screen, demonstrating an early form of the direct-manipulation interactions that are now common in pen and touch-based interfaces. Research on interacting with touch screens followed with different sensing strategies: capacitive sensing [Joh65, Joh67], optical tracking [EJG73], or resistive sensing [CJWC75]. The first multi-touch screen followed in 1976: the keyboard with variable graphics [KM76]. Since then, multiple sensing systems and configuration have been explored. With the widespread adoption of mobile touch-enabled smartphones, horizontal projection surfaces integrated into a tabletop soon also became touch-enabled. Shortly thereafter, tabletops became possible desktop surrogates.

The benefits of tactile interaction over other forms of inter-action have been deeply studied for a variety of tasks and pa-rameters. Studies have compared mouse and tactile interaction for speed [SS91, FWSB07, GBC13], error rate [SS91, FWSB07], minimum target size [AZ03], etc. Similarly, studies have com-pared tactile with tangible interaction for tasks as various as puz-zle solving [TKSI07, Wan10], layout-creation [LJZD10], photo-sorting [TKSI07], selecting/pointing [RGB*10], 3D manipulations
[BIAI17b], and tracking [JDF12]. To summarize, tactile interaction appears to be a good compromise between fast and precise input. Tactile interaction also lends itself to a direct style of interaction [BIRW19] where users' place their fingers right on top of the 2D or 3D representations of the data they wish to manipulate. The directness of tactile interaction has been studied in previous work [SS91, MCN94, PM03, SBG09, KH11, LOM*11, SG15, BIRW19]. Studies confirm that it increases the user's impression they are making direct manipulations [Shn83] of the data they are visualizing, which can make the interaction more engaging and can encourage further manipulations. Despite these interesting advantages, tactile interaction is often limited and limiting. It is limited because it is often used as a discrete interaction mechanism, while our human interaction mechanisms are continuous [FTW12]. It is also limiting because many complex tasks (in particular for 3D manipulations) require input/control with more than three degrees of freedom. Providing them using tactile input usually requires multiple fingers, thus leading to occlusion issues.

It is possible to distinguish two main types of devices offering tactile interaction. First, there are touch-enabled tabletops or wall displays, which are fixed and usually facilitate the viewing of large data with a possibility to carry out co-located cooperative work. Second, virtually all mobile devices today offer a multi-touch interface; they are easy to transport and affordable. These two types of devices, because of their inherent size, lead to different interaction designs. Indeed, while a large display can easily support more than 3 fingers without much occlusion, mobile devices are not that permissive. Similarly, large screens allow designers to add widgets on the screen, but this is not possible on mobile devices due to their much smaller screens where widgets could waste some precious visualization space. Other forms of touch interaction can also be found in the literature (e. g., skininput [HTM10]), but to the best of our knowledge, are not used for 3D spatial visualization applications.

### 2.1.2. Paradigm 2: Tangible and haptic interaction

The first prototypes and platforms for tangible interaction were developed and studied as early as 1976 with [PMIoT76]'s Slot Machine to help children discover programming languages. Other prototypes followed [Ais79, FFF80]. In 1996, Fitzmaurice introduced the concept of graspable user interfaces [Fit96]: an interaction paradigm that used physical objects to synchronously manipulate digital counterparts. In his work, the graspable props were associated with specific functions and allowed users to interact with both hands simultaneously. This concept evolved and expanded into Tangible User Interfaces (TUIs) [IU97]. Tangible User Interfaces aim to leverage peoples' natural skills for manipulating the surrounding physical environment [Fit96, IU97, Ish08]. Tangible input inherently offers 6 integrated DOF per prop. Several studies have investigated the benefits of TUIs when compared to other interaction paradigms for different tasks (e. g., [CMS88, HTP*97, TKSI07, Wan10, RGB*10, TKI10, BIAI17b]). Overall, tangible interactions have been proven to be useful for 3D rotations [CMS88, HTP*97] and, more generally, for fast and precise 3D manipulations [BIAI17b], collaboration [MFH*09, OAWH11], and entertaining [XAM08, BIAI17b].

Tangible interaction is promising for visualization tasks and purposes: it allows users to achieve complex 3D manipulations with

simple real-world style gestures [Fit96, IU97]. Consequently, tangible interaction is perceived as more flexible than other interaction paradigms usually are (e. g., [HPGK94, BIAI17a]).

Tangible props may take the form of the data, serving as both a physical representation of data and a means of interacting with the data, or their physical form may be more abstract, providing passive haptic context or support for the interface, but without much visual feedback on the prop itself e. g., [HPGK94, GHP*95, FBZ*99, IGA14a, JLS*13, IGA14b]. Extending beyond passive haptic aids, we also include active haptics in this paradigm. Haptic devices enable 3D manipulation and tactile feedback within a restricted interaction space (due to the limited range of robotic arms, cables, etc.). Manipulations with these devices can be programmed to feel realistic, as they would in the real world, or “extra” effects can be added using programmatically controlled vibrations and forces. This property has been used for visualizing 3D datasets. One possible advantage of feeling the data through these output forces is the ability to explore and understand dense 3D datasets where occlusion or cluttering prevent clear visual-only displays (e. g., [TRC*93, AS96, LPYG02, LPGY05, LCP*07, COPG15, YJC15]).

### 2.1.3. Paradigm 3: Mid-air gestural interaction

Mid-air gestural interaction is often traced back to *The ultimate display* concept introduced Ivan Sutherland [Sut65], although concrete implementations are only recent, with a first step taken by the commercialization of the Wii controller [KV19]. Like tangible interaction, mid-air gestural interaction mimics the physical actions we make in the real world [FKK07] and, thus, has been studied as a promising approach to 3D manipulation [KTY97, HIW*09, WPP11, SGH*12], including for the purpose of increasing accuracy [FKK07, Osa08]. While it is possible to manipulate and track tangible objects in the air, the research notes significant differences in mid-air gestures made using only the hands; thus, this paradigm focuses exclusively on inputs made in mid-air without the need to hold an object. Such gestures can be tracked via wearable technologies, such as a glove [FMHR87], or optically. Optical tracking sometimes requires placing markers on the body (e. g., [FH00]). Solutions for precise tracking of the fingers have traditionally been elusive or expensive, but recent devices, such as the Microsoft Kinect and Leap Motion, now support precise hand and finger tracking (e. g., see [CPLCPFR14, SKR*15]), enabling a richer set of hand or body gestures. While this passive optical tracking helps freeing users from the need to wear any markers or devices [MS16, EAG17, Iss17], the accuracy is quite to the same level [EAG17].

In the medical field, the need to maintain a sterile environment naturally leads to an interest in touchless interaction [OGS*14, EAB*15, MHWH17]. Research has thus explored facial expressions [NHK*03], gaze [YH00, NHK*03, HLN*17], or hand and body gestures [JW14]. The first two categories are out of the scope of this paper as they tend to be used for browsing through 2D images [HPMD13, YBM13, NHH*16] or controlling medical instruments [NHK*03] rather than manipulating 3D medical datasets. However, hand and body gestures are quite relevant. They have been used for visualization tasks (e. g., [KRW*11, RRA*12, Gal13, YBM13]) but also for a variety of other tasks in the operating room, as recently surveyed by Mewes et al. [MHWH17].
### 2.1.4. Paradigm 4: Hybrid interaction

Recognizing the distinct advantages of different paradigms and devices, researchers have also sought to combine multiple interaction paradigms together. Interaction techniques that combine tactile, tangible, and/or mid-air aspects appear in the research literature in the 1990s and 2000s [Bux95, PTW98, SS99, OBF03, Yee03, DD05, JGAK07, Wil07] and seem to increase in frequency recently [SSD09, CSH*14, MBH14, LODI16, SvZP*16, BIAI17a, BAI17, BSY*19, Bes17]. Early work by Hinckley et al. [HPSH00] included adding low-cost components to mobile devices; the authors concluded that the resulting hybrids “may prove to be the most practical approach.” Others also argued for the benefits of hybrid approaches to augment a limited interaction space [KR09], overcome the inherent limitations of a device (e. g., augmenting the number of DOF that can be manipulated [KRG*12], reduce the occlusion limitation with tactile interaction [BIH08]), combine the benefits of two interaction paradigms [BIAI17b], or simply tackle complicated tasks (e. g., seeding point placement in 3D [BIAI17a]). The resulting hybrid interaction paradigms can be used to support tasks ranging from abstract visualization tasks [AMR16, CVLB18] to 3D manipulations [LODI16, BIAI17a, BAI17, Bes17], and the combinations of paradigms are varied: pressure and tactile interaction (e. g., [CVLB18]), tactile and tangible interaction (e. g., [JGAK07, BIAI17a, BSY*19]), pressure and tangible interaction [BAI17], mid-air gestural interaction and tactile interaction (e. g., [WB03, HIW*09]), mid-air gestural interaction with tangible interaction (e. g., [SLM*03]) or vocal interaction with others [TFK*02]. We, however, limit our review to hybrid paradigms that specifically address 3D visualization problems and tasks.

## 2.2. Axis 2: Interaction tasks for 3D visualization

The second axis categorizes the 3D visualization tasks users must accomplish using the various interaction paradigms identified in Axis 1. Formal task taxonomies have been developed previously in both the visualization and 3D user interface research communities. Accordingly, our classification combines aspects from both areas of related work. The tasks involved in data visualization have been studied extensively, and task classifications have been proposed, both in early work [WL90, CR96, Shn96] and more recently [YKSJ07, BM13, RCDD13, SNHS13, Mun14, RAW*15, KK17, LTM18]. Most of these visualization task classifications are generic in the sense that they can apply to any type of visualization, including our focus on 3D visualization. Likewise, classifications also exist for understanding 3D interaction. LaViola et al. [LKM*17] identify the major 3D user interface task categories as selection, manipulation, navigation, and system control. These categories are similarly generic—they can apply to any 3D user interface, therefore also including the focus of this survey on interactive visualizations.

Although task taxonomies from both areas clearly apply, our work also builds upon the arguments laid out by Keefe and Isenberg [KI13] who suggest that 3D visualization does introduce special requirements for interaction tasks. One example is exploring dense data within 3D neural pathway visualizations; the precision required for making 3D selections in this visualization context is far greater than in the scenarios typically studied within more generic 3D user interface research (e.g., quick 3D modeling, selecting items on a

shelf during a virtual shopping experience). In addition to selection, other generic 3D interaction tasks such as, manipulation, and navigation, also have special requirements in the context of 3D visualization. To emphasize these and connect as closely as possible to earlier classification systems (sometimes a direct 1-to-1 mapping is impossible), we organize this Axis of the taxonomy around three high-level task groups: (1) Volumetric view and object manipulation, (2) defining, placing, & manipulating visualization widgets, and (3) 3D data selection & annotation. In the following discussion we place these task groups as closely as possible within the context of earlier classifications and describe the special 3D visualization challenges these tasks present and how they can be addressed by spatial interaction techniques.

### 2.2.1. Task Group 1: Volumetric view and object manipulation including clipping

Volumetric view and clipping manipulation tasks are fundamental to visualize spatial 3D data effectively because it is rare that a single viewpoint can be found where all of the important aspects of the data may be analyzed. This issue is most often addressed via interaction to adjust the viewpoint of the rendering(s) or to manipulate clipping planes within the data. As a category, *Volumetric View and Object Manipulation* corresponds to *3D data space/view navigation* and *temporal navigation* in Keefe and Isenberg’s taxonomy [KI13] and relates closely to more general VIS tasks of *explore* and *reconfigure*; the closest link in the 3D tasks extracted from LaViola et al. [LKM*17] is *Manipulation and Navigation*.

3D manipulations are often studied in Human Computer Interaction to allow users to translate objects, rotate around the three axes, and perform uniform (or non-uniform) scaling. Considering that any manipulation of an axis requires 1 Degree of Freedom, this translates to providing at least 7 Degrees of Freedom (DOF), and possibly up to 9. A wide variety of techniques have been proposed (e. g., [Han97, HCC07, LGK*13, IBIA16, LKM*17]) and most have also already been surveyed [JH13].

However, Volumetric view and object manipulation goes beyond “simple” 3D manipulations and raises specific challenges that are not typically present in general spatial interfaces for navigation (e.g., redirected walking techniques, WIMs, wand-based flying). While simple rotations and/or translations make it possible to view 3D data externally, many 3D spatial datasets are dense, and relevant internal aspects of the data are, therefore, naturally occluded. Interactions for Volumetric view and object manipulation should directly address this need. Cutting planes or transfer function editors are often used for this purpose, and since these are widget-based, one might consider these as falling under a visualization widget manipulation task. However, from the standpoint of the user’s cognitive approach they are tied so tightly to view manipulations (e.g., moving the camera inside volumetric data necessarily involves clipping) that it can be useful to think of these as integral Volumetric view and object manipulation tasks. In fact, we argue that this is the type of insight that is useful when determining the best ways to translate 3D user interfaces created for more generic 3D environments (e.g., architectural walkthroughs, simulations) to 3D visualization applications.

Many 3D visualization view manipulations consider only cutting planes to slice through the data, but it is interesting to notice
that some experts might need non-planar or free-form surface slicing of their data to provide an easier and more natural analysis of some datasets [PTH98, GPB99, PSOP01, MFOF02, MFF03, SGH03, KVLP04, RRRP08, REM11, KGP*12, LSG*16, PCE*17]. These approaches can be linked to techniques such as peeling, which can be useful for surgical planning [SGH03, KVLP04, BHWB07, MRH08, REM11, HMP*12, PCE*17], but can also be used in other domains, such as reservoir visualizations [SSSS11]. Non-planar slices are often defined relative to the data but can also be specified with significant user input (e. g., [BHWB07] Beyer combines 2D mouse input with medical scan data). Specifying, modifying, and positioning non-planar slicing objects or without relying on underlying data poses an interesting challenge for spatial interfaces, and this is a topic that we return to in later sections (see Sect. 3.2 and Sect. 3.1).

Another challenge is the manipulation of data and cutting planes with axis-based constraints [BKBN12]. 3D visualization users must often manipulate/zoom heterogeneous datasets, including many manipulations along a single axis [FGN10], and more generic 3D manipulation techniques, as typically studied in the user interface literature, do not often address this latter point.

### 2.2.2. Task Group 2: Defining, placing, & manipulating visualization widgets

Spatial 3D data may be analyzed simply by looking, but interacting with filters, probes, and other visualization widgets is required to more deeply explore and interrogate the data. Visualization widgets are virtual tools that are manipulable by users in much the same way as any traditional 2D or 3D user interface widget but that have a primary purpose of displaying data. A cutting plane that users can grab and manipulate relative to volume data is one example that fits well within this category. As a category, Defining, Placing, & Manipulating VisualizationWidgets corresponds to positioning/manipulating data exploration objects or probes such as drilling cores (2 DOF) and specifying/manipulating 3D points and other primitives for particle seeding, picking, or path planning in Keefe and Isenberg's taxonomy [KI13]. Like task group 1, this task group relates most closely to the more general VIS tasks of explore and reconfigure and LaViola et al.'s [LKM*17] Manipulation and Navigation task. Like Keefe and Isenberg, we believe it is important to highlight this as a separate task category because of its longstanding importance in exploratory 3D visualization systems.

Visualization widgets are extensively used in 3D flow visualization. For instance, aerodynamicists studying fluid flows might begin a visualization session by manipulating cutting planes to understand the internal structure of the visualized data. Then, they often need to rely on placing and manipulating widgets (e.g., particle emitters, streamline rakes) to further explore and understand the data or create useful pictures for communicating their findings. Many flow visualization widgets rely upon particle tracing and appropriate particle seeding. Weightless particles are placed within a vector field and then advected with the flow. It is then possible to integrate the path of particles along the flow as a function of time [Man01] and to visualize the resulting path with lines, ribbons or stream surfaces [PvW94, SFL*04]. The quality of the resulting visualization, often relates to the quality of the original particle seeds. Thus, controlling this seeding interactively using a widget is often a major benefit.

Semi-automated techniques are also available, for example, specifying a single 3D origin from which several particles are generated with randomly jittered 3D offsets. This technique has proven useful for analyzing reservoir data [Wil96a] or other forms of flow visualization [SBPM98, Man01, Sch07, KI13]. Aerodynamicists also make use of streakline or filament line visualizations [Fre93, BJS*98], which can be implemented as virtual smoke emitter widgets. The results help to visualize vortices more directly [Fre93], and, again, 3D placement of the emitter benefits from interactive control. Particle seeding is also used in medical visualization to depict pulsatile blood flow [Ste00]. Similarly, traces can help meteorological visualization of typhoons [LGY15]. While it is possible to display all streamlines simultaneously for each field in the data, this can lead to occlusion. Automated algorithms have been developed to minimize occlusion (see e. g., [TEC*16]), but the issue can be avoided altogether with the help of interactive placement.

Interactive visualization widgets have also been used in other contexts. The Glyph Lens technique uses a magic lens effect to overcome issues of occlusion for viewing volumetric tensor fields [TLS17]. A full overview of lenses and their use in visualization is available in the survey from Tominski et al. [TGK*17]. In addition to these primitives, domain expert sometimes need to assess the values of specific points in their datasets, a feature that is often implemented with a probe widget. Interactively positioned 3D probe widgets have been used to facilitate the computationally-heavy inspection of 4D MRI Blood-Flow [vPOBB*11] and other complex data [MEV*06, KGP*12]. Filter widgets have been explored [GNBP11], as have measurement widgets for assessing spatial relations to help, for example, for surgical planning [PTSP02, RSBB06].

### 2.2.3. Task Group 3: 3D data selection & annotation

Selection is the first step in accessing deeper information about some subset or feature of the 3D spatial data, annotating these data to include insights or questions, and many other operations that are critical to interactive data analysis. Selection can take many forms depending upon the data involved. Dense data with small features of interest and/or features that are not well defined, often make this task a significant challenge. As a category, 3D Data Selection and Annotation corresponds to 3D picking or selection of data subsets for further analysis in Keefe and Isenberg's taxonomy [KI13]. 3D Selection maps to the more general VIS tasks of abstract/elaborate and filter. The equivalent 3D task from LaViola et al. [LKM*17] depends upon the implementation but can fall under Selection, Manipulation and Navigation, System Control, or even Symbolic Input.

Selecting specific regions of interest is essential for revealing interesting patterns, properties, or internal structures in 3D data [Wil96b]; thus, selection is a critical task to support for data visualization [Ban14]. 2D regions are usually defined using picking, brushing, or lassos—often achieved with a mouse/pen or on a tactile screen (both modalities provide the needed 2 DOFs). Many generic 3D object selection techniques in virtual environments rely on 3D ray-casting [AA13]: a ray, cast from the user's hand, selects the first object it hits. A number of variations on ray-casting are possible, and it is probably the most widely used 3D selection technique [TJ00, CSD03, dHKP05a, OF03, DHKP05b, GB06, VGC07, AAT08, KGDR09, BPC19, BS19, RBP*19, LYS20]. A major limitation of ray-based selections is, of course, the difficulty
<figure><img src="image_3.png" /><figcaption>Figure 1: Visualization (adapted from [PDF14]) of our classification system and the number of papers found for each category of our two classification axes (image: ©① CC-BY 4.0 L. Besançon, A. Ynnerman, D. F. Keefe, L. Yu, and T. Isenberg).</figcaption></figure>

of selecting small and/or far-away objects, which is often complicated by hand jitter. Expanding the ray to a cone helps with this [LG94, FHZ96, OBF03, SBB*06, SP04, Ste06], and other primitives may also be used [ZBM94, WHB06, VGC07].

Unfortunately, many of these classic 3D selection techniques do not translate directly to 3D spatial visualization. The level of precision needed to make useful 3D selections for scientific or medical analysis tasks is one factor. Another factor is that spatial data are often volumetric, without clearly defined or discrete objects or structure; this makes it difficult to apply 3D object selection techniques that commonly rely on 3D intersection tests.

Annotation does not appear by name in Keefe and Isenberg's 3D visualization taxonomy [KI13] but is mentioned in general VIS tasks [BM13]. Depending on the implementation, it may include or require 3D picking or selection of data subsets for further analysis. For that reason, we grouped it here with 3D selection, even though it requires an additional input (which is often categorized as System Control or Symbolic Input). The need to integrate annotation into visualization systems has been highlighted by many different researchers in the literature [SBM92, HPRB96]. Springmeyer states, "while images may be the goal of visualization, insight is the goal of the analysis" [SBM92]. Annotation is essential to sharing these insights. Scientists use annotation to keep track of their own findings and points of interest or easily share findings with collaborators or lay people. Providing a good contextual-aware annotation system fosters knowledge-sharing, teaching, and remote collaboration. Annotation can take the form of textual notes, drawings, voice recordings, and other records input by users. In the research context, the contextual information needed to place annotations within the context of the data is typically also included [HPRB96]. Thus, supporting interactive 3D annotation for visualization means that users must be able to record insights and other information within the spatial context provided by the 3D data. Automated positioning algorithms can assist with this challenge (e. g., [PHTP*10]), but defining the proper interface for annotating 3D visualizations remains a major challenge. Indeed, annotation within

virtual environments, even outside of the visualization context is a longstanding topic of research that continues to be actively studied today [AS95, BHMN95, MBJS97, CL17, CG17, PMMG17].

## 3. Survey of the state of the art

Now that the major spatial interaction paradigms (Axis 1) and visualization tasks (Axis 2) are defined, this section presents a survey of the state of the art of spatial interaction for visualization organized according to these two axes. To find relevant papers, we followed a semi-systematic approach. We used Google Scholar to find papers with specific keywords (e. g., "3D visualization", "spatial interaction", "3D interaction"). Once we found a relevant paper, we followed the trail of citations: we looked at the references in that specific paper and the papers citing that specific paper. We also included papers suggested by reviewers of our manuscript. Finally, we classified all of the papers using the two axes. Fig. 1 provides an overview of the entire collection of papers. The four major sections below correspond to the four interaction paradigms of Axis 1 and, within each section, we further divide the discussion into three subsections to correspond to the three task groups of Axis 2.

### 3.1. Visualization with tactile & pen-based interaction paradigms

This first group of techniques covers approaches that provide spatial input directly on a screen surface, via touch or pen input.

#### 3.1.1. Volumetric view and object manipulation tasks with tactile input

3D object manipulation on tactile screens has been widely researched in general (e. g., [Han97, LKM*17, HCC07, RDH09, LAFT12, JH13, LWS*13, PBD*16, KKKF18]). Researchers have also explored 3D user interfaces for touch-based control using spherical or cubic screens (e. g., [GWB04, dIRKOD08]). However, none of these approaches address tasks that are specific to 3D spatial data visualization, such as ways to see through the data with cutting
<figure><img src="image_4.png" /><figcaption>Figure 2: Examples of multi-touch interaction for 3D spatial visualization. From left to right: (a) the tBox widget for 3D manipulations (image courtesy of and ⊕① A. Cohé and M. Hachet) [CDH11], (b) the Power-of-10-Ladder technique (image courtesy of and ⊕① Chi-Wing Fu) [FGN10], and (c) a user twisting a 3D mesh by placing one hand on top of the other and rotating them [JSK12] (image © the Canadian Human-Computer Communications Society, used with permission).</figcaption></figure>

planes, or axis-aligned manipulations. One of the key design decisions in implementing tactile manipulations of 3D content is whether to control all DOFs simultaneously (e. g., [RDH09, LAFT12]) or to separate them using constraints or some other menthod. The trade-offs have been discussed in the non-visualization-specific literature (e. g., [ZS97a, ZS97b, VCB09, MCG10]), but some researchers note a special benefit to separating DOFs in visualization-specific cases [Ise16, CML*12]. In the remainder of this section, we limit the discussion to tactile interactions that have been designed explicitly with visualization purposes in mind.

The most common tactile 3D manipulation techniques from non-visualization applications have also been used for data visualization. For instance, Lundström et al. [LRF*11] implemented a 3D RST (1 finger for x-/y-rotations, 2 fingers for z-rotations, panning, and zooming) technique for medical data visualization. To provide axis-constrained cutting plane manipulations, they added GUI-based pucks. The 3D RST technique was estimated to be the most widely implemented for manipulation and visualization of 3D data in software for mobile devices in 2016 [BIAI17b].

Tactile 3D manipulation techniques have also been designed, from the start, specifically to address the needs of visualizations [Ise16]. Au et al. [ATF12], for example, proposed to use multitouch gestures on a large display for camera control, object selection, uniform scaling, axis-constrained rotation and translation (two-finger gestures on a specific axis), and object duplication (three-finger gestures). They compared their approach to a traditional widget-based interface and concluded that a gesture-based approach can be just as efficient. One limitation of this approach is that users must discover and learn the set of tactile gestures before they can be used.

To overcome the discoverability issue, Yu et al. [YSI*10] developed FI3D. The FI3D widget surrounds the data visualization like a rectangular frame, and each edge of the frame is used to activate a different 3D manipulation. Translations around the x-/y-axes are initiated with a single finger interaction in the central space. Arcball (x-/y) rotations are initiated with a single finger touch on the frame and a drag into the center visualization region. Touching the frame with a second finger during this interaction, constrains the rotation to a single axis (depending on the frame). Rotations about the z-axis are controlled by dragging a single finger along a frame

(as opposed to perpendicular to it).Widgets in the corners of the frame activate zooming operations, and two additional horizontal bars along the top and bottom of the frame provide z-translations. Yu et al. also mention that the mapping could be changed to adapt to other datasets which might require different manipulations based on their inherent properties, as exemplified in the implementation of FI3D for the exploration of fluid flow data [KGP*12].

The principle of widget-controlled interaction was also used by Cohé et al. [CDH11], who developed tBox (see Fig. 2a) to provide users with easy control over 9DOFs based on the context set by the location of their touches and the number of fingers used. The technique can easily be applied to 3D data views and relies on a cube-shaped widget overlaid on the scene. The widget contains multiple interaction zones and is oriented to match the orientation of the scene being viewed. One-finger manipulations along the edge of the cube translate along the parallel axis. One-finger manipulations on the sides of the cube control single-axis rotations. Scaling is controlled by pinching the cube, on the cube sides for uniform scaling while a pinch gesture and on opposite edges will initiate a non-uniform scaling. In an other study [LODI16], the tBox technique was found to increase the feeling of precision for 3D interaction.

As noted by Yu et al. [YSI*10], dataset-specific interaction techniques are sometimes needed. Fu et al. [FGN10] present an example, combining trackball rotations with a custom “powers-of-ten-ladder” (see Fig. 2b). The technique facilitates exploration of astronomical datasets, which require rotations and scaling operations that span large magnitudes (translations are less useful in this scenario). Arcball rotations are controlled using a single finger, panning operations are controlled with 5-finger gestures, and zooming operations are controlled using a bimanual two-finger pinch. Two-finger inputs activate the ladder widget, where each region corresponds to a power of ten zoom level. Fu et al. created this technique to alleviate the strain on users’ hands when performing large zooming operations in astronomical datasets. While Fu et al. extend more traditional tactile interaction to support the special needs of astronomical spatial data, Kim et al. designed a tactile interface to support the special needs of navigating through and comparing spatial datasets that change over time [KJK*15]. The approach, applied to historical architectural reconstructions across different time periods, combines a timeline widget, multi-layer map-based navigation, and immersive visual-
ization with staged, animated transitions between datasets. Other dataset-specific, or dataset-inspired, interfaces include the work by Sultanum et al. [SVBCS13], which addressed the challenge of navigating within geological outcrops via a two-step technique; users first indicate a navigation surface onto which the camera will be constrained, and touch gestures are then used to tilt, zoom, or pan the camera with respect to the x-/y axes.

Finally, some tactile interactions for visualization take the approach of augmenting tactile 2D input with additional inputs. This has most frequently been done on tabletops and in hybrid virtual environments (e. g., [BI05, SAL06, HIW*09, MJGJ11]) with hand tracking to augment touch input. Jackson et al. [JSK12] applied this concept to 3D data visualization, using the posture of the hand above a 3D stereoscopic table to allow users to tilt, bend or twist datasets within the 3D space (see Fig. 2c). Song et al. [SYG*16] also augmented touch input with hand-posture sensing to help manipulation and exploration of 3D visualizations. They distinguish between the left/right hands, thumb and other figures, and hand tilting vs. finger movement to provide methods for manipulating 3D data and cutting planes. Several of these techniques rely upon more-than-2-finger gestures or screen-space widgets that are appropriate for large displays but may not translate well to smaller, mobile displays. For smaller displays, pressure has been used to augment tactile interaction, in particular to separate DOFs when manipulating 3D objects [WBAI17, WBAI19]. In this work, a combination of light and hard touches with one or two fingers were used to independently manipulate translation and rotation along the x- and y-axes or the z-axis. Panchaphongsaphak et al. [PBR07] also use pressure-augmented touch but for the purpose of orienting and translating a cutting plane within medical data. Pressure beyond a given threshold was used to translate the slicing plane in the direction of its normal.

Several other tactile input techniques have been developed for manipulating cutting planes. For example, Song et al. [SGF*11] enabled users to move cutting planes with one- and two-finger motions on a mobile phone. Klein et al. [KGP*12] used a three-finger technique to control a cutting plane within a FI3D widget: two fingers on the cutting plane specified a rotation axis, and a third finger somewhere else in the data view specified the amount of rotation. Or, by moving the third finger along one of the FI3D frames, the cutting plane was translated in the direction of its normal. Sultanum et al.'s [SSSS11] splitting and peeling techniques also relate to the use of cutting planes when the cutting operations that are constrained to the data's axes. The tactile input is used to either separate the data into two sub-parts or perform a local distortion that helps geologists explore the data's spatial structure. Recently, Sousa et al. [SMP*17] used a VR setup and touch sensing on a table with gesture based control of cutting planes to enable radiologists to explore 3D data. By placing the touch surface on the desk before the users and, thus, explicitly separating the 2D display from the stereoscopic 3D data display, Sousa et al. avoid the disconnect between 2D surface input and 3D graphical displays cited as a concern by other researchers in previous work [VSB*10, SHSK08, VSBH11].

To summarize, a myriad of touch techniques and platforms have been explored to support volumetric view and object manipulations. Overall, tactile input has been shown to be useful for 3D visualization, especially when combined with axis-constrained interaction

[BKBN12, Ise16]. Researchers have adapted tactile interactions for visualization to different computing platforms (e. g., small displays), in part, by augmenting touch with additional inputs, such as pressure [WBAI19]. Researchers have also shown the utility of dataset-specific tactile interfaces (e. g., [FGN10, SVBCS13]). While the work in this area covers a broad range of topics, the community has yet to establish platform- or dataset-specific interface guidelines or standards that might help developers to follow best practices for 3D visualization with touch input [BIAI17b].

### 3.1.2. Visualization widget tasks with tactile input

Interactive seed point selection and manipulation is an important task for 3D visualization, especially for fluid flow data. Particle tracing based on these seed points helps researchers understand the motion of the fluid, and is one of the most common 3D flow visualization strategies (as explained previously in Sect. 2.2.2).

Using touch input and a dedicated widget, Butkiewicz and Ware [BW11], for example, facilitate the seeding of particles at various depths to explore ocean currents (see Fig. 3a). Their setup is quite unique: they combine a stereoscopic screen that displays the 3D data with touch input, a setup which usually creates problems [SHSK08, VSBH11]. In their specific case, however, they place the physical touch surface (stereoscopic display) at an angle and render the data such that it is displayed at a similar angle, with the ocean surface coinciding with the physical touch surface. Butkiewicz and Ware then use data exploration widgets called “dye poles” placed at the surface, with controls to create and manipulate seed point placement at varying ocean depths. Other widgets can be used to specify points or paths in 3D space, e. g., using Butkiewicz et al.’s [BSW19] recent Pantograph technique. A different approach to particle seeding was taken by Klein et al. [KGP*12]. They use a monoscopic display, but treat the cutting plane that they place in the projected view of a generic flow dataset as a proxy to specify 3D locations for seed point placement. This is combined with an unprojected view of the same cutting plane that acts as a widget. Using this widget, users can place particles around a small region (single-finger input), along a line embedded into the cutting plane (two-finger input), or around a larger circular volume (input from three or more fingers). In addition to particle seeding, they also make it possible for users to place drilling cores as columns oriented perpendicular to the cutting plane for data read-out. Coffey et al. [CML*12] also use a stereoscopic data projection, but in contrast to the two previously described techniques, they separate the stereoscopic (and vertical) data display from a monoscopic horizontal touch-sensitive surface, which is used for input. Their SliceWIM technique reinterprets the classic VR World-in-Miniature (WIM) interface technique to apply to volumetric data. Touch input is used to manipulate the WIM widget, which includes features for controlling slicing planes and selecting flow lines that pass through these planes as well as defining 3D points and curves relative to the volume data. The ability to touch with many fingers simultaneously enables users to specify and rapidly adjust complex selection shapes on the slicing planes and the linked 3D visualization displays the results in real time.

To summarize, researchers have used touch input to control visualization widgets in a variety of ways, introducing creative solutions to manipulate 3D contexts through this type of 2D input, and provide features impossible to implement using single-cursor techniques.
<figure><img src="image_5.png" /><figcaption>Figure 3: Examples of multi-touch interaction for 3D spatial visualization. From left to right: (a) a seeding point placement technique with a stereoscopic screen [BW11] (image courtesy of and © Thomas Butkiewicz, used with permission), (b) a tactile structure-aware selection technique [YEII12] (image © IEEE, used with permission), and (c) a mobile device's screen used for selection and annotation [SGF*11] (images courtesy of and © Peng Song, Wooi Boon Goh, and Chi-Wing Fu, used with permission).</figcaption></figure>

### 3.1.3. 3D data selection & annotation tasks with tactile input

In addition to view changes and data object manipulation, one of the most essential tasks in visualization is data selection and picking. While they can be achieved with established techniques such as ray casting (we review some of the main selection metaphors in Sect. 2.2.3) for datasets that consist of explicit objects, additional techniques are needed for continuous data, such as volumetric scalar fields, particle clouds, or flow fields. Tactile input selection techniques often mirror selection techniques developed for more traditional input modalities [Wil96b, AA09, AA13, Ban14], but researchers showed that the more direct style of control often possible with tactile input as compared to mouse input leads to benefit for visualization [BIRW19]. To provide direct manipulation with 3D content, such interactions are often designed to be *view-dependent* and possibly *structure-aware*, to help users specify depth.

For picking in volumetric data, e. g., Wiebel et al. [WVFH12] introduced WYSIWYP—a technique that can easily be applied in a tactile input context. Given a selected 2D point on the filmplane, they analyze the corresponding view ray passing through the volume data, take the current transfer function into account, and select the largest jump in accumulated opacity. This typically denotes a feature that is locally visually dominant. The picking technique is thus *view-dependent* and *structure-aware*. Shen et al. [SLC*15] later described a variation of WYSIWYP which computes a saliency measure and picks the 3D point accordingly. Yet, picking single 3D points is often insufficient for preparing for further data analysis—in such cases, users have to be able to specify spatial subsets of the 3D data.

Structure-aware selection techniques that support selecting sub-volumes of interest were pioneered by Owada et al. [ONI05]. Their Volume Catcher relies on a user-drawn stroke on the visible contour of a subset of the volumetric data, which Owada et al. then use to segment the underlying data to return the intended volume of interest. Inspired by this technique, Yu et al. [YEII12] presented CloudLasso, which used a user-drawn 2D lasso shape, extended it as a generalized cylinder into 3D space, and then used kernel density estimation to select the subset within the cylinder whose scalar property surpassed a given threshold (Fig. 3b). This approach had the added benefit that the threshold could be adjusted after the lasso had been drawn, which enables users to adjust their selection. Shan et al. [SXL*14] presented a further extension, which makes it possible to select only the largest connected component of the data

rather than all components within the generalized cylinder, arguing that this is likely to better match the user's intent. Finally, to make it possible for users to better control which connected component is finally selected, Yu et al. [YEII16] later extended their work and introduced three CAST techniques, two of which used the shape of the drawn lasso to control the single component to select, while the third technique, named PointCAST, only relied on a single 2D input point to specify a 3D region of interest.

Selection techniques for other 3D spatial data have been explored. For line data, Akers [Ake06], e. g., described the CINCH fibertract pen-based selection technique, which uses sketched 2D paths to guide the selection of fiber tracks that represent neurological pathways. Coffey et al.'s [CML*12] Slice WIM technique, selects flow lines that pass through a lasso shape, but instead of sketching the lasso, the lasso is defined as the convex hull that surrounds the fingers touching the visualization, making it possible to rapidly change the shape of the selection in real-time.

Although not as precise as pen input, touch input is also well suited for annotating data visualizations through writing and sketching, in particular for supporting collaborative data exploration. For example, Song et al. [SGF*11] make it possible for users to annotate 3D medical data on a mobile device (see Fig. 3c). The annotation was created by drawing on the cutting plane shown on the mobile device, which then updates a larger, linked medical data visualization. This combination of a small mobile display with a static larger display also facilitates several hybrid techniques which we describe later in Sect. 3.4. Ohnishi et al. [OKKT12], in contrast, facilitate the annotation of 3D objects using a tablet placed statically on a table, but again visualize the main data on an additional large vertical display. Users annotate the data by drawing on flattened 3D surfaces displayed on the tablet. Sultanum et al. [SVBCS13], in contrast, use a single, combined display and tactile input device. With their system, users can annotate 3D surfaces of geological outcrops by projecting touches onto the displayed surface.

Using 2D tactile input for selection in 3D visualization seems challenging given the loss of one DoF, but research has shown that this challenge can be overcome. Solutions often involve interpreting input relative to data values or features or combining selection with other tasks and widgets, for example, specifying a 3D selection via interaction on a 2D cutting plane. combinations of data-specific computations, multiple selection steps or tools (e. g., combining
<figure><img src="image_6.png" /><figcaption>Figure 4: Examples of tangible interaction for 3D spatial visualization. From left to right: (a) a stylus to indicate locations on the surface of a 3D-printed tracked coral [KL09] (reprinted from the publication with permission by Springer), (b) seeding point placement and data annotation with a Wiimote [TO10] (image courtesy of and © VCAD Riken 2011), and (c) a stylus allowing cutting-plane manipulations and seeding point placement in a handheld augmented reality setup [IGA14a] (image courtesy of and Ⓔ① Issartel et al.).</figcaption></figure>

selection with cutting planes). 3D data selection and annotation is clearly feasible with tactile input, and could have advantages over alternatives when considering the ease of sketching and writing and the importance of these traditional styles of input for annotation.

## 3.2. Visualization with tangible and haptic paradigms

This second group of techniques works with input that relies on additional sensing and/or feedback that relates to our haptic sense.

### 3.2.1. Volumetric view and object manipulation tasks with tangible input

Interactions via tangible props, proxies, and devices are appealing because they tend to mimics the way we have learned to work in the real, physical world [Fit96, IU97]. Consequently, many tangible visualization interfaces provide full 6-DOF tracking and input. One of the first systems was from Hinckley et al. [HPGK94] who designed passive props for neurosurgeons to manipulate and inspect their data using cutting planes. In addition to laying out the requirement and use of tangible props for scientific visualization, Kruszynski and van Liere [KL09] proposed to use a printed tangible prop that physically visualizes the data (see Fig. 4a). In this way, the props can act as a physical world-in-miniature with any manipulations of the props in the 3D physical space being reproduced in the virtual world visualized on a large stereoscopic display. Couture et al. [CRR08]'s *GeoTUI* makes use of tangible props within a tabletop visualization of geo-data and compared their tangible interface to a more traditional mouse-based alternative. The props were used to indicate slicing planes, and three alternative props were compared (a 1-pluck prop, a 2-pluck prop, and a ruler). They found that the ruler was the most appropriate input device for the geophysicists. Rick et al. [RvKC*11] used a spatially-tracked prop in a CAVE to facilitate visualization of probabilistic fiber tracts. The prop supported 3D data manipulation and a virtual-slicing-cone interaction with a flashlight metaphor. They also provided ways for the users to constrain the slicing plane to specific axis.

Picking up on the importance of constrained manipulation for data visualization that we mentioned in previous sections, other researchers have also combined tangible interaction with constraints. Bonanni et al.'s *Handsaw* [BAC*08] prototype made it possible

to obtain slices of the data by interacting with hand-held objects (such as a laser). Despite the physical ability to move the hand-held object in any direction, the virtual slices were restricted to move only along a normal direction. Spindler and Dachselt [SSD09] make the slicing plane itself physical by supporting interaction with a tracked, physical, paper-like prop (called PaperLens). Their hardware includes a 2D tabletop augmented with a projector and sensors. Multiple interactions are possible and are visualized by projecting imagery directly onto the paper. For example, users can select which layer of multi-layered data to view simply by changing the height of the paper with respect to the table. This constitutes another interesting example of constraining tangible interaction (or at least the interpretation of the users' interaction) rather than treating the 6-DOF manipulation of tangible objects quite so literally.

Another interesting use of tangible interaction for visualization is to use multiple tangible objects to represent different portions of the data. For example, the tangible system developed by Reuter et al. [RCR08] used props to help archaeologists virtually reassemble fractured artifacts, like a 3D puzzle. Following a similar motivation, Khadka et al. [KMB18] use hollow tangible props worn around the wrist to represent individual slices or fields of data. Users can add or remove these from the visualization by manipulating the props.

Interaction using generic tracked VR controllers, AR markers, and the like can also be viewed as a form of tangible interaction as the shape of the controllers or surface the markers are printed on convey some tangible information, even if not dataset or task-specific. For interaction in AR, Tawara and Ono [TO10] relied on a simple visual marker to enable users to manipulate medical data with 6 DOF (see Fig. 4b). In a Desktop AR context, markers have also been used as metaphors for cutting planes to provide arbitrary slicing position and orientation of volumetric datasets, such as to-mographies. Moving beyond a flat marker, while still acting as a generic prop, Chakraborty et al. [CGM*14] used a physical wire-frame cube prop in AR for 3D manipulation of chemistry data. The cube is used as a container for the visualized dataset. Issartel et al. [IGA14b] used a cuboctahedron to manipulate fluid dynamic data with 6DOF in AR, also proposing different slicing techniques for us with handheld AR visualization. The manipulated cuboctahedron is covered with markers and tracked with a tablet's camera. Their approach enables slicing through the data by treating the tablet as
a cutting plane or by using an optically-tracked stylus. Interaction with generic VR controllers is also common, and the research includes techniques for simultaneously manipulating views of multiple volumetric datasets or 3D scenes in order to support comparative visualization. Bento Box [JOR*19] accomplishes this via a bimanual interface for quickly selecting and arranging sub-volumes of interest in a grid. Another approach, Worlds-in-Wedges [NMT*19], accomplishes a similar task by combining a custom world-in-miniature interface with a pie-slice view of several worlds at once. In both cases, generic VR controllers provide 6 DOF pointing and grabbing inputs that are interpreted relative to the data.

As mentioned in Sect. 2.2.1, non-planar slicing of volumetric data is often useful, and one of the interesting ways to achieve this using tangible interaction is with a pile of modular blocks, sand, or clay [PRI02, RWP*04, Lue13, LFOI15]. The data slice can be projected directly onto the material, and, optionally, an extra monitor can be used to provide a contextual visualization. This concept has been applied to landscape models [PRI02] and biological, seismic, and air temperature simulations [RWP*04]. It is also possible to implement a similar approach using optical see-through displays [LFO*13].

In summary, the research on tangible interaction for visualization demonstrates how physical props may be used as intuitive proxies for manipulating data and slicing planes and how constraining the interaction (not utilizing all 6-DOF simultaneously) can often be useful. Additionally, some of the most creative work in this area involves concurrent manipulation of multiple tangible objects or even piles of sand; these provide a decidedly different and potentially useful means of interacting with spatial 3D data.

### 3.2.2. Visualization widget tasks with tangible input

Tangible interaction can also be very helpful to specify and manipulate visualization widgets, for example, virtual probes, which are often controlled with a handheld stylus or controller. De Haan et al. [dHKP02] use a tracked stylus in head-worn VR to read specific data point values. Kruszynski et al. [KL09] use a stylus together with a 3D printed physical visualization to interactively select and measure data properties (data read-outs) of marine coral (see Fig. 4a). The data and results are visualized on a large stereoscopic screen. Following a similar strategy of using one tangible prop for the data and one tangible prop to specify a 3D point, Issartel et al. [IGA14a] employ a stylus to generate particle seeds within volumetric data (see Fig. 4c). The stylus and the dataset prop are visually tracked, thanks to visual markers, and a see-through tablet is used to provide Augmented Reality. Because the data are represented by a physical volume, the seed point origin must be offset from the stylus tip so that it can points can be placed inside the volume, but users still benefit from the tangible aspects and can push a button on the stylus to start emitting particles from the point of origin. A similar approach is used by Tawara and Ono [TO10], who make use of a wiimote augmented with visual markers to provide a seeding point origin visualized in AR with a head-mounted display. Similar to Issartel's approach, the location of the seeding point is not directly located on the wiimote, though this is not because of physical limitations in this case as the data is simply manipulated through a flat 2D marker. Finally, in the context of augmented reality visualization for structural design, Prioeto et al. [PSZ*12] use a specially designed

tool to input the 3D locations where pressure will be applied to a structure in order to visualize its deformation.

Virtual probes are most tangible when implemented using active haptic devices. For example, direct haptic interaction with volumetric data was demonstrated by Lundin et al. [LPYG02]. They avoid using explicit geometry, while maintaining stable haptic feedback, by using proxies. This makes it possible to represent various data attributes and manipulate the orientation of visualized data based on additional attributes and channels. Their work was later extended [LPGY05, LPCP*07] to define haptic primitives for volume exploration, such as lines, planes, attractive forces based on data attributes. Similarly, Van Reimersdahl et al. [vRBKB03] present haptic rendering techniques for interactive exploration of computational fluid dynamics data, such as scalar and vector fields, that promote an intuitive understanding of the data. Direct haptic interaction has also been used to simulate palpations in medical simulator to assist in medical training procedures [UK12]. Finally, Prouzeau et al. [PCR*19] used haptic-augmented VR controllers to explore the density of 3D scatterplots and manipulate cutting planes.

Haptic feedback can also help to guide the placement of probes. Lundin et al. [LPSCY05] used haptic guidance to place streamlines in CFD data from airfoil simulations. Olofsson et al. [OLC*04] used proxy-based volumetric data exploration to plan stereo tactic gamma knife brain surgery by using haptic feedback to convey dose distribution in a brain tumor and guide placement of gamma ray "shots." Related to surgery planning, Reitinger et al. [RSBB06] used jug and ruler widgets to provide volumetric and distance calculations and assists medical staff in their diagnosis and treatment planning.

Beyond virtual probes, 3D magic lenses (also called magic boxes) are another form of visualization widget that can be controlled with tangible input. For instance, Fuhrmann and Gröller [FG98] used a tracked pen to place a 3D magic lens that provides a more focused view of the data or constrains streamlines. The cutting planes discussed in previous sections are also interactive visualization widgets—we chose to group them with volumetric view manipulation tasks but they can be thought of as fitting here as well.

Tangible interaction is used routinely to place and manipulate 3D visualization widgets like virtual probes and magic lenses. These interactions can be successfully guided and/or convey additional data back to the user when they are coupled with active haptics.

### 3.2.3. 3D data selection & annotation tasks with tangible input

Tangible interaction can be particularly useful for the problem of 3D data selection within volumetric data. Indeed, specific devices can be used and tracked in order to allow users to specify the 3D bounds of a subset of the data. Taking advantage of this, research projects have focused on designing and testing specific hardware for this task. For instance, Harders et al. [HWS02] use 3D haptic force feedback to facilitate the segmentation of linear structures. Similarly, Malmberg et al. [MVN06] use a haptic device and stereoscopic rendering to allow users to draw 3D curves based on the 2D live-wire method (see Fig. 5a). This idea was improved with Spotlight [THA10] which adds visual guidance to improve the quality of the segmentation. A similar setup is used by Nyström et al. [NMVB09]. Gomez et al. [GJL10] propose to facilitate selections with two tracked props, a
<figure><img src="image_7.png" /><figcaption>Figure 5: Examples of tangible interaction for 3D spatial visualization. From left to right: (a) use of a PHANTOM device in a stereo rendering to assist the segmentations of volumes [MVN06] (reprinted from the publication with permission by Springer), (b) two tangible props used to identify tracts of interest in brain data [GJL10] (reprinted from the publication with permission by Springer), and (c) selection in dense 3D-line datasets with a haptic-augmented tool [JCK12a] (image © The Eurographics Association, used with permission).</figcaption></figure>

pen-like probe to brush in a 3D volume and a cube to manipulate the data (see Fig. 5b. Their technique allows users to select tracts in a DTI fibertract dataset. De Haan et al. [dHKP02] proposed to combine a tracked stylus and a tracked transparent acrylic plane to facilitate 3D selections of regions of interest in head-worn VR. The position of the plane is used to specify the extents of a selection box while the stylus is used to specify a point of origin. Jackson et al. [JLS*13] use a rolled piece of paper as a tangible prop to facilitate selection of thin fiber structures and manipulate views of the data. Schkolne et al. [SIS04] use custom tangible devices to interact with and select DNA parts in an immersive VR environments with a headset. In particular they use the metaphor of a raygun to select distant parts without having to physically move to these parts. Pahud et al. [POR*18] imagined that a spatially-aware mobile device could be used as the origin of a projection of different selection shapes onto a 3D volume to provide a volume selection mechanism. Finally, based on the haptic-aided drawing on air technique [KZL07], Keefe developed a free-form 3D lasso selection technique that can be used in fishtank VR environments [KZL08, Kee08].

In addition to guiding 3D drawing, haptic devices can use data-
driven feedback to further assist with making accurate 3D selections,
helping to overcome the challenges of occlusion and cluttering.
Zhou et al. [ZCL08] use a Phantom force feedback device with
stereoscopic glasses to draw 2D lassos that are then connected to
select DTI fiber tracts. Jackson et al. [JCK12b] introduced Force
Brushes, which uses progressive data-driven haptics provided by a
Phantom to select subsets of 3D lines in dense datasets (see Fig. 5c).
Chakraborty et al. [CGM*14] combined a Phantom and a visually-
tracked cube prop in an Augmented Reality environment for 3D ma-
nipulation and selection of chemistry data. Lundin et al. [LPLCY06]
use haptic feedback to improve guided segmentation of MRI data.

To facilitate 3D data annotation, tangible interfaces have also
been used as tracked note-taking devices/screens to specify the 3D
position first and input annotations. One of the first prototypes to
provide annotation through a spatially tracked device is the Virtual
Notepad [PTW98]. Users could navigate in their 3D environment
by walking and annotate specific places within the virtual scene.
Cassinelly and Matasoshi [CI09] also use a tracked screen but with
cutting planes of medical data; once fixed by activating a clutching
mechanism, data are annotated on the screen at the position of the

slice. Song et al. [SGF*11] use a similar approach, combining an
iPod Touch and a large vertical display.

Other approaches enable note taking through the manipulation of the device itself and support additional visual annotations beyond text that can be used to add visual notes, showing connections in graphs, etc. For example, Kukimoto et al. [KNEK05] use a tracked PDA in a collaborative VR environment to capture annotations. A single button press activates the note taking that is done by moving the PDA in the 3D space. Similar to this, Benko and Wilson [BW10] use an infra-red laser-pointer with dome-based visualizations of 3D graphs or astronomical data. In their system, a presenter can use the laser to write annotations directly on the dome surface, which are captured by a camera and integrated into the computer graphics renderings shown during public presentations.

Using tracked 3D paintbrushes or generic VR controllers, in Scientific Sketching [KAM*08] 3D sketches are used not just to annotate 3D datasets but also as a visualization design tool. The system aims at involving artists and other visual experts in the task of designing the most effective uses of color, texture, form, and metaphor for multivariate VR visualizations.

Tangible devices provide some valuable affordances for 3D se-
lection and annotation tasks, including a natural support for writing
and sketching (in both 2D and 3D). The research also points to
strong potential to augment these capabilities with active haptics
that help to guide the interactions, helping to reduce the natural hand
jitter that is sometimes a problem with mid-air interfaces and/or to
increase the precision of 3D inputs relative to the underlying data.

3.3. Visualization with mid-air gestural interaction paradigms

Next we discuss techniques that capture input in mid air.

3.3.1. Volumetric view and object manipulation tasks with mid-air input

Similar to tangible interaction, mid-air gestural interaction is mo-
tivated by the potential to provide natural 3D manipulations to
users, though these are arguably less like real-world manipulations,
since users cannot actually hold the object they are manipulating.
A pioneering work in mid-air gestural interaction is the responsive
<figure><img src="image_8.png" /><figcaption>Figure 6: Examples of mid-air gestural interaction for 3D spatial visualization. From left to right: (a) a surgeon navigating in 2D and 3D images in the Operating Room [RRA*12] (reprinted from the publication with permission by Springer), (b) a system to manipulate data and obtain measurement between two 3D points [GPC11] to select and create bounds between molecules (image courtesy of and ⊗① Gallo et al.), and (c) a 3D point cloud selection technique based on Oculus Rift [LBLS14] (image courtesy of and ⊗① P. Lubos).</figcaption></figure>

workbench [CFH97] which provided, in a tabletop VR environment, mid-air gestural interaction to rotate and translate data and cutting planes, specify axis- or plane-constrained manipulations, and pick specific artifacts. The hand gestures were recognized with the help of worn gloves. The potential of mid-air gestural interaction has been demonstrated by Kirmizibayrak et al. [KRW*11] who compared bimanual mid-air gestural interaction tracked with a Kinect 2 and mouse-control of 3D medical data with two different experiments. The first study consisted in an orientation-matching task for which gestural interaction showed strong evidence for outperforming mouse input. The second study focused on slicing techniques for which their data support that mouse control was more accurate but slower than mid-air gestural interaction. Similarly, Theart et al. [TLN17] compared several interaction modalities in VR and suggest that a Leap-motion-based hand tracking system for microscopic data analysis is a good tool for data scaling and rotating.

Ruppert et al. [RRA*12] proposed two different prototypes for 3D rotation, scaling, slicing, and contrast adjustment (which is not directly linked to 3D data manipulation but rather to system control). They successfully implemented their system, which relies on single-hand manipulation in the Operating Room, and tested it during real procedures (see Fig. 6a). Also in the Operating Room, Mewes et al. [MSR*16] proposed a set of LMC-tracked single-hand gestures to manipulate 3D planning models as well as select 2D buttons or navigate trough 2D slices. Hettig et al. [HMR*15] proposed a set of single-hand gestures tracked with a Myo Gesture Control Armband. Gallo et al. [GPC11, Gal13] focused on the possibility to provide many different 3D visualization tasks through bimanual gestures tracked by a Kinect (see Fig. 6b). The visualized data is visible on a traditional 2D display. Since they implemented many different interaction techniques, they defined translations as two-handed (palm facing forward) concurrent manipulations. Zooming is achieved through the same posture but by moving both hands closer or further away. Rotations are also achieved with both hands, but with clenched fists. Similarly, Lubos et al. [LBLS14] proposed a set of bimanual gestures to support 3D manipulation of point clouds for users wearing HMDs (see Fig. 6c). Laha et al. [LB13] proposed *Volume Cracker*, which consists of bi-manual gestures to crack open a dataset to explore its internal structure. The authors aimed to replace traditional slicing plane manipulations with this approach. Users have to close both hands to crack the visualized

volume in two and then can manipulate each individual cracked part or iteratively crack them into smaller subsets. While domain experts are more familiar with traditional axis-aligned slicing planes techniques, Volume Cracker illustrates the potential mid-air gestural interaction techniques have for helping us to rethink the way we analyze data. This work can help us develop new metaphors that could be more efficient or insightful once mastered.

While mid-air gestural interaction leads naturally to unconstrained, gestural interactions, several techniques have specifically addressed the need to provide axis-constrained interaction for visualization. Malkawi et al. [MS05] proposed a set of gestures, tracked with a glove, to provide constrained zooming operations and constrained translations of isoplanes in 3D data rendered with a HMD (see Fig. 7a). Bonanni et al.'s *Handsaw* [BAC*08] includes tangible slicing of volumetric (medical and urban) data (see Sect. 3.2.1) and also supports single hand gestures to perform the slicing of the data. The slicing-plane is constrained to specific axes, and hand gestures allow users to translate the plane along the axis. The data and slices are visualized on a desktop display. Finally, Botero et al. [BODVOGHV17] proposed a set of gestures to manipulate data obtained from medical imaging. A pointing finger can move the data in 3D while gesturing with the whole hand allow users to translate the three axis-aligned slicing planes to analyse internal structures.

The work by Fleury et al. [FDGS12] (see Fig. 7b) brings mid-air gestural interaction interaction for manipulation into collaborative spaces. In their collaborative studies, two users work together to define a cutting plane. The first uses both hands, and the second uses a single hand to provide the three points that will define the cutting plane. They can interactively manipulate the plane's position and orientation by moving their hands together.

From the surveyed work, we take away that one of the most important design decisions is the choice or uni- or bi-manual interaction. Sometimes the right choice is constrained by the application domain, for example, surgeons in the operating room might need to keep one of their hands on the task they are performing while doctors performing a diagnosis may be able to use both of their hands.

### 3.3.2. Visualization widget tasks with mid-air input

Our survey uncovered far less on mid-air gestural interaction interaction with visualization widgets. Nonetheless, as explained in
<figure><img src="image_9.png" /><figcaption>Figure 7: Examples of mid-air gestural interaction for 3D spatial visualization. From left to right: (a) a user interacting with an isoplane [MS05] (reprinted from the publication with permission by Elsevier), (b) a user manipulating the centre of a cutting plane [FDGS12] (image courtesy of and Ⓔ① Fleury et al.), and (c) mid-air gestural interaction hand-gestures to facilitate 3D selection [TLN17] (① Theart et al.).</figcaption></figure>

Sect. 2.1.3, to date, most of the applications of mid-air gestural interaction involve 2D image browsing. An exception is the work of Gallo et al. [GPC11] who proposed a specific gesture to facilitate measurements between points in 3D (see Fig. 6b). Users have to keep a hand outside of the interaction space and use the other hand to point at specific locations. The locations are then recorded, and the distance is measured and shown to the users. Malkawi et al. [MS05] also proposed to use mid-air gestural interaction to provide data read-outs (see Fig. 7a). Among the many commands they propose to support with their system, they explored a specific gesture to allow users to obtain detailed information of a specific 3D point in space, visualized through an AR headset. The point is indicated by making a pointing gesture at the desired 3D position using a tracked glove.

### 3.3.3. 3D data selection & annotation tasks with mid-air input

3D data selection has also been explored through mid-air gestural interaction. Focusing first on object selection for archaeological purposes, Allen et al. [AFT*04] had users wear a glove to select archaeological objects in a head-worn VR environment. Similarly, focusing on selection of objects or subparts of data that can easily be isolated, Benko and Wilkinson [BW10] proposed, in a dome environment, to use pinch gestures in mid-air to provide a selection mechanism. Direct feedback is provided thanks to shadows that are cast on the dome screen while performing the movement. Pinch gestures with shadows as depth cues have also been explored by Wang et al. [WL14] in the context of Computer Aided Design. However, in their work, the authors concluded that the shadows provided insufficient depth cues for 3D selections.

Using bimanual gestures, Gallo et al. [Gal13] proposed an interesting way to specify ROIs. They focused on the possibility to offer 9 DoF manipulations for clipping manipulation. These clipping operations allow users to select a specific subset of the data by clipping-out the unwanted parts of the data. Users translate and rotate a clipping box but also rotate the data. One version of their prototype allows users to separate the control of the box and the data, while the other integrates all three possible manipulations. They compared their approach with a mouse-based approach and found that more simultaneous degrees of freedom could lead to more precise manipulations. Theart et al. [TLN17] proposed, in a HMD VR environment, to manipulate several selection shapes (box or cylinder, see Fig. 7c) or to let users trace Regions of Interest with

one finger and then let users scale this region into 3D through other gestures. This kind of selection gives more freedom and control to the users when compared to manipulating selection primitives, but other approaches can give even more control to the user. For instance, Lubos et al. [LBLS14], in addition to the gesture set they use to provide 3D manipulations, also proposed to support selection in 3D Point Cloud data with a brushing technique. Users, wearing a HMD, can brush in the 3D space to select points. Schönborn et al. [SHLPF14] used tracking of mid-air gestures to interact with 3D representations of nanotubes in public settings.

Similar to the manipulation of widgets with mid-air gestural interaction surveyed in Sect. 3.3.2, it appears that not a lot of work has focused on annotation with mid-air gestural interaction. Bacim et al. [BNB14] proposed to iteratively remove parts of the data that should not be annotated via gestures in mid-air tracked by a Leap Motion in a desktop environment. Annotations are then added by typing on the keyboard attached to the workstation. Also employing mid-air gestures to specify the area/volume to annotate, Lubos et al. [LBLS14] proposed to annotate the data that have been selected through the use of their mid-air brush. Similar to tangible interaction, one could envision using hand movements to directly annotate and write in the air (e. g., [AGLW16]), but, to the best of our knowledge, such a system has not been used to annotate 3D environments.

Several selection metaphors (e. g., selection shapes or brushing) have been proposed to support 3D selection and annotations with mid-air gestural interaction, but overall this task group has received little attention with this interaction paradigm.

## 3.4. Visualization with hybrid interaction paradigms

Finally, we survey techniques that combine the previously mentioned interaction paradigms.

### 3.4.1. Volumetric view and object manipulation tasks with hybrid input

For 3D data and cutting plane manipulation, several approaches rely on a combination of tactile and tangible interaction. In an attempt to "leverage the benefits of precise 2D manipulations combined with fast 3D manipulations", Bornik et al. [BBK*06] designed a custom device on a tablet PC for use in a VR environment (see Fig. 8a).
<figure><img src="image_10.png" /><figcaption>Figure 8: Examples of hybrid interaction for 3D spatial visualization. From left to right: (a) a scientist holding the tracked *Eye of Ra* and interacting on a touch tablet (image courtesy of and ©① B. Alexander) [BBK*06], (b) a spatially-aware tablet combined with touch input to specify the location of seeding point [BIAI17a] (image © IEEE, used with permission), and (c) a tactile-enabled tracked cube to provide 3D selections and manipulations [CBL*17] (image taken from the Cordeil et al.'s video with their permission).</figcaption></figure>

Their system allowed users to manipulate the data (rotation, translation) and place a cutting plane by using the device on the tablet or moving the device directly in 3D space. Particularly relevant for our review, the authors conclude that combining interaction paradigms did not appear to be difficult for users. Cordeil et al. [CBL*17] developed a touch-sensitive cube, augmented with a gyroscope and accelerometer; movement of the cube itself provides a tangible interface for 3D manipulation of a VR visualization registered to the cube, while the touch input is used for other visualization-oriented tasks. Designing custom devices is less common than utilizing the mobile devices that have, for more than a decade now, been able to provide both tactile and tangible input. The Natural Material Browser [FFH13] combines spatial/tangible manipulation of a tablet and multitouch gestures for volumetric material science datasets. The rotations tracked with the inner sensors of the tablet allow domain scientists to visualize different slices of the data on the tablet, while touch-interaction is simply used to either provide rotation information or change datasets and parameters (a system control task). Pushing this approach further, López et al. [LODI16] investigated touch input for 3D manipulations on a tablet, in the context of an additional vertical stereoscopic screen. They provide 3D manipulations through tactile input using the tBox method [CDH11] on the tablet, but also add the possibility to rotate the data by directly rotating the tablet. They propose a discussion on conflicts in perception between the two scenes and how to keep both displays synchronized.

To go beyond the limitations of internal sensors, other researchers have used visual tracking of multi-touch devices. For instance, Song et al. [SGF*11] proposed to manipulate slicing planes through medical data using a visually-tracked iPod Touch. The data is visualized on large vertical display and slices can be transferred to the iPod for annotations. Katzakis et al. [KTKT15] use a mobile device with touch input and rotation tracking to support multiple interactions depending on the mapping specified; their system includes support for manipulating cutting planes and data. With the availability of spatially-aware tablets such as the *Google Tango*, Besançon et al. [BIAI17a] decided to focus on the specific needs of fluid dynamic researchers and combined tangible manipulations of the tablet with tactile interaction to provide full 3D data and cutting plane manipulation. Their interaction mapping was deemed more flexible than the current state of the art tools by fluid dynamics experts.

Since they are complimentary and widely accessible in today's devices, it is unsurprising that most of the hybrid work surveyed thus far combines tactile and tangible interaction paradigms. However, the literature also includes examples that combine spatial interaction paradigms with traditional mouse and keyboard interaction. An interesting recent approach was presented by Mandalika et al. [MCB*18]. They combined a traditional 2D desktop and its usual interaction mechanism with the zSpace fishtank VR system for radiological analysis purposes. The mouse and keyboard and the zSpace can be used sequentially or simultaneously and the authors report that it integrates very well in radiologists' workflow.

### 3.4.2. Visualization widget tasks with hybrid input

Hybrid interaction paradigms have also been used to help with the 3D positioning of points of interest and visualization widgets. For instance, Sultanum et al. [SSSS11] combined a multi-touch table system with tangibles for exploring geologic reservoir data. They used tangible props to control detailed data read-outs and a focus+context view, while tactile input was used for regular data navigation, including data-specific techniques such as splitting and layer peeling. The spatially-aware approach of Besançon et al. [BIAI17a] allowed fluid dynamic researchers to position seeding points in 3D by, for instance, combining touch and tangible input. They could start by placing a cutting plane with tangible manipulations of the tablet. Then, they could use touch input on the tablet to specify the x- and y-position of the seeding point, and the z-position was derived from the intersection of a ray originating from the finger with the cutting plane (see Fig. 8b). Cordeil et al.'s [CBL*17] touch-sensitive cube, on the other hand, allowed users to manipulate the data with direct spatial manipulations of the cube itself and use ray-casting from finger positions onto the cube's side to, among other things, specify parameters for data readouts.

### 3.4.3. 3D data selection & annotation tasks with hybrid input

Similar to 3D data manipulation, most approaches for 3D data selection rely on a combination of tactile and tangible interaction. For this task, some approaches have created custom interaction devices to provide both interaction mechanisms. The 3D selection mappings proposed by Katzakis et al. [KTKT15] enable manipulating the selection volume with either tactile or tangible input. However, while
<table><caption>Table 1: The state of the art of spatial interfaces for 3D visualization. We ordered the papers by their apparition in the text of Sect. 3 and colored all references to indicate the used output device: 2D projection, mobile device, stereoscopic screen, Virtual Reality (VR), or Augmented Reality (AR). We mark approaches that combine several output devices by stripes, e. g., mobile device + stereoscopic screen. We include a more detailed classification of output devices in supplementary materials, available at https://osf.io/jtx75/</caption><tbody><tr><td rowspan="4">High-Level Task 1:<br/>Volumetric View<br/>and Object<br/>Manipulation</td><td>Tactile interaction (Sect. 3.1.1)</td><td>[Ise16], [LRF*11], [BIAI17b], [ATF12], [YSI*10], [KGP*12], [CDH11], [LODI16], [FGN10], [SVBCS13], [KJK*15], [JSK12], [SYG*16], [WBAI17], [WBAI19], [PBR07], [SGF*11], [SSSS11], [SMP*17]</td></tr><tr><td>Tangible Interaction (Sect. 3.2.1)</td><td>[BAC*08], [SSD09], [Fit96], [IU97], [HPGK94], [KL09], [CRR08], [RvKC*11], [RCR08], [KMB18], [JOR*19], [NMT*19], [TO10], [CGM*14], [IGA14b], [LPYG02], [LPGY05], [LPCP*07], [vRBKB03], [UK12], [PCR*19], [PRI02], [RWP*04], [Lue13], [LFOI15], [LFO*13]</td></tr><tr><td>Mid-air gestures (Sect. 3.3.1)</td><td>[CFH97], [KRW*11], [TLN17], [RRA*12], [MSR*16], [HMR*15], [GPC11], [LBLS14], [LB13], [MS05], [BAC*08], [BODVOGHV17], [FDGS12]</td></tr><tr><td>Hybrid Interaction (Sect. 3.4.1)</td><td>[BBK*06], [CBL*17], [FFH13], [LODI16], [SGF*11], [KTKT15], [BIAI17a], [MCB*18]</td></tr><tr><td rowspan="4">High-Level Task 2:<br/>Defining, Placing, &<br/>Manipulating Visu-<br/>alizationWidgets</td><td>Tactile interaction (Sect. 3.1.2)</td><td>[BW11], [KGP*12], [CML*12]</td></tr><tr><td>Tangible Interaction (Sect. 3.2.2)</td><td>[dHKP02], [KL09], [IGA14a], [TO10], [PSZ*12], [FG98], [LPSCY05], [OLC*04], [RSBB06]</td></tr><tr><td>Mid-air gestures (Sect. 3.3.2)</td><td>[GPC11], [MS05]</td></tr><tr><td>Hybrid Interaction (Sect. 3.4.2)</td><td>[SSSS11], [BIAI17a], [CBL*17]</td></tr><tr><td rowspan="4">High-Level Task 3:<br/>3D Data Selection &<br/>Annotation</td><td>Tactile interaction (Sect. 3.1.3)</td><td>[WVFH12], [SLC*15], [ONI05], [YEII12], [SXL*14], [YEIII6], [Ake06], [CML*12], [SGF*11], [OKKT12], [SVBCS13]</td></tr><tr><td>Tangible Interaction (Sect. 3.2.3)</td><td>[HWS02], [MVN06], [THA10], [NMVB09], [GJL10], [dHKP02], [JLS*13], [SIS04], [POR*18], [KZL07], [Kee08], [ZCL08], [JCK12b], [CGM*14], [LPLCY06], [KZL08], [PTW98], [CI09], [SGF*11], [KNEK05], [BW10], [KAM*08]</td></tr><tr><td>Mid-air gestures (Sect. 3.3.3)</td><td>[AFT*04], [BW10], [WL14], [TLN17], [LBLS14], [BNB14], [KTKT15], [CBL*17], [VC14], [SAIB16], [BSY*19], [GSBI20], [PWHK16], [HPRB96], [PTW98], [TFK*02], [BBK*06], [KNEK05], [LBLS14]</td></tr><tr><td>Hybrid Interaction (Sect. 3.4.3)</td><td></td></tr></tbody></table>

they showcase part of their work with visualization data, their 3D selection is limited to an object selection interaction and is not suited for the needs of volume visualization as we previously argued in Sect. 2.2.3. The touch-sensitive cube developed by Cordeill et al. [CBL*17] goes further and implements a selection mechanism based on touch input of the tangibly-manipulated data (see Fig. 8c). For instance, they use pinch gestures to define a rectangular subspace that passes through the entire data volume. Alternatively, free-form drawing on either side of the cube can be translated through the complete volume and multiple side drawings can together iteratively define a cross selection volume.

Moving away from custom-designed hardware, Veit et al. [VC14] proposed 3D selection and annotation of the selected subsets with a spatially tracked multi-touch device in a 3D stereoscopic setting. Their 3D selection technique focuses on point cloud data and proposes to manipulate a selection sphere that is attached to a ray controlled by the tablet's movement. Tactile manipulation changes the size of the selection sphere on the fly. This approach is versatile and can adapt to different datasets and regions of interest. Pushing the versatility even further, Besançon et al. [SAIB16, BSY*19, GSBI20] proposed to use a spatially tracked tablet to provide free form 3D selection of volumetric data. Their approach is not data-dependent and allows users to first trace a lasso on the tablet to obtain a 2D shape that is then extruded in 3D using 6DOF tablet manipulations. This approach relies on a combined tablet and large screen and enables entirely user-controlled free-form selection. Although their research focuses more on the specific workflow of annotations and their review and modifications, Pick et al. [PWHK16] combine custom CAVE hardware with a smartphone to support annotation tasks.

Many different visualization systems rely on immersive displays, such as VR headsets. Taking notes can be particularly challenging in these environments, and hybrid interaction paradigms often provide a solution. The early work of Harmon et al. [HPRB96] and Poupyrev et al. [PTW98] was pioneering in this regard. They used a spatially tracked device to annotate (medical) data for VR environments (with headsets). The users could navigate the environment by walking inside of it, position themselves and the tracked-tablet at a specific location, and start their note-taking process with a pen on the tablet. Tsang et al. [TFK*02] extend this style of spatial annotation in the Boom Chameleon interface, which combines tactile and tangible interaction. It is a mechanically tracked display, augmented with a tactile overlay to capture touch gestures and speech via a microphone. Manipulations of the display make it possible to capture a specific view that then can be annotated. The device developed by Bornik et al. [BBK*06] also allowed users to take notes by combining tactile and tangible input to select where notes should be placed. Interestingly, Kukimoto et al. [KNEK05] proposed to use a tracked PDA to take notes in a VR environment by using it solely as a tangible note-taking device or combining its location with the pen input. Finally, Lubos et al. [LBLS14] augmented their mid-air gestural interaction-based selection gestures with voice-recognition to create a VR annotation tool for 3D point cloud data.

The most commonly found hybrid paradigm for annotation and selection combine tactile and tangible interaction. For example, touch input can be used to sketch annotations or selection marks while tangible input is used to specify a 3D location or extrude 2D selections into 3D space.
# 4. Opportunities for future research

We would like to highlight three challenges and opportunities for future research that emerge from the survey.

## 4.1. A need to focus on visualization widgets

To facilitate discussions of the state of the art of 3D spatial visualization, we have gathered the approaches mentioned in our survey in Table 1, following the same classification as in Sect. 3 (an aggregated version was presented in Fig. 1). To obtain this table, we have included all past work in Sect. 3, excluding papers that do not focus on proposing a new technique for visualization.

Our survey highlights clear areas for future work and might also explain the lack of adoption of spatial 3D visualization techniques by domain experts, as pointed out by previous work [WBG*19]. In Table 1, we clearly see that HCI and VIS researchers have rightfully investigated several interaction paradigms, each of which is valuable, especially considering that experts in different domains have different needs. For instance, surgeons are more likely to be interested in mid-air gestural interaction, while researchers in geology, fluid dynamics, or archaeology can make use of props or screens. Our final report on hybrid interaction paradigms (see Sect. 3.4) and Table 1 also clearly highlight that researchers in VIS and HCI have been leveraging the potential of hybrid interaction paradigm for improving visual exploration of 3D datasets.

Despite these positive take-away messages, in Table 1 one can also clearly see that task groups 1 and 3—Volumetric view and object manipulation and 3D data selection or annotation—have been investigated by numerous authors, covering all the interaction paradigms we discussed. Task group 2 (defining, placing, and manipulating visualization widgets), however, has been barely investigated—to the best of our knowledge and considering the very large body of work we reviewed. Furthermore, Sect. 3 highlighted that the few approaches that did consider this task narrowly focus on one or two visualization widgets. These widgets have been identified by domain experts to be essential to conduct a proper 3D data analysis (e. g., [BIAI17a, PTSP02]). The lack of adoption of 3D spatial techniques developed by HCI and VIS researchers could, hence, be explained by the fact that most prototypes do not have all the widgets and features necessary to conduct a complete analysis. In addition, past work has highlighted the need to provide a link between HCI/VIS prototype and traditional analysis software (e. g., Matlab, Python) run on desktop computers [WBG*19]. On the other hand, research on 2D visualization widgets is not rare (see visual analytics contributions, e. g., [HS04, KHPA12, YS20]), which suggests that such research is valued in the visualization research community. The lack of focus on 3D visualization widgets could therefore be explained either by the fact that the challenges of manipulating virtual probes, cutting planes, and the like are still not addressed, or the fact that 3D visualization communities might not be interested or aware of the need for 3D visualization widgets beyond these. Either way, we hope that this survey will contribute to highlight this specific need and eventually foster more work that supports the manipulation of visualization widgets for 3D data.

Among the four spatial interaction paradigms we explored, we also found relatively few works on mid-air gestural input for all

three task groups. One possible reason is that, compared to other paradigms, mid-air gestural interaction is less precise due to unstable gestures. Another possibility is that tactile and tangible interaction paradigms feel more comfortable to users. In some special cases, however, the use of mid-air gestural interaction can be essential such as in operating rooms where hygiene is paramount (see, e. g., Sect. 2.1.3). Furthermore, mid-air gestural interaction also seem to be the de-facto expectation of some experts [WDB*20], making it a potentially more intuitive solution in some cases. Consequently, we believe that more research is needed in this direction.

## 4.2. Interaction in public spaces

Science communication is currently rapidly embracing the possibility of using 3D data visualization as a way of telling engaging data driven stories to a broad audience at public venues, such as science centers and museums. As intuitive, reliable and robust interaction is a hard requirement at public venues, research is needed to identify specific interaction challenges posed when dealing with large number of users exhibiting diversity in age, language, culture and knowledge. As previously postulated by Sunden et al. [SBJ*14], interaction techniques can help in the dissemination of scientific discoveries using interactive installations. Also, in large-scale immersive theaters such as planetariums, interaction becomes a central part of live programs, often with a facilitator [KHE*10, BHY18] (see Fig. 9a) carrying an interactive and non-linear narrative. Despite being already used for 2D data exploration tasks or discovery of ancient objects (e. g., [PPMW06, HdIRL*13, MSLF15]), the potential for interaction techniques in immersive visualization environments [BCD*18] to foster engagement and learning of complex 3D scientific subject matter remains under-explored and under-studied [YRA*16, YLT18] (see Fig. 9b). Here, we provide a few examples of the related work that does exist.

In public dissemination tactile interaction plays a central role, and there are numerous examples of installations utilizing tactile interfaces to navigate and interact with the data. A primary example is the Inside Explorer volumetric rendering with touch interfaces described by Ynnerman et. al. [YRA*16]. Jönsson et. al. also explored touch interfaces for simplified interaction with transfer functions, targeting novice users, using style galleries embedded in the 3D context [JFY16]. Ljung et al. [LKG*16] provide a general survey of transfer functions including interaction aspects.

Tangible interaction lends itself to natural use in public dissemination, and a multitude of examples are used on a regular basis. We do, however, find that many of these approaches are not documented in terms of research papers and there is room for systematic studies of tangible interaction in public spaces. One available example of is the use of physical objects to steer the behavior of visual objects on a touch surfaces [HCM*16]. Another interesting example used in public dissemination is the use of haptic interfaces for communication of complex molecular interfaces [PCT*07].

In public dissemination work, gesture-based interaction is commonly used by visitors to interact with content and select options in exploration scenarios. However, most of the developed techniques focus on interaction with 2D content or slide-based presentations (e. g., [CFM*12, RS13]). Schönborn et al. [SHLPF14] used gesture
<figure><img src="image_11.png" /><figcaption>Figure 9: Examples of 3D interaction for knowledge dissemination. From left to right: (a) live presentation in an immersive dome theater using the OpenSpace software [BHY18] for realtime 3D interaction (image © IEEE, used with permission), (b) three scenarios showing volumetric interaction with medical CT data using a touch surface [YRA*16], haptic feedback, and tangible input devices (image © ACM, used with permission), and (c) gesture based 3D interaction with nano tubes [SHLPF14] (reprinted with permission by IGI Global).</figcaption></figure>

based interaction with 3D representations of nanotubes together with stereoscopic viewing (see Fig. 9c). A recent study on different gesture paradigms for 3D interaction in mediated presentations was published be Krekhov et al. [KEBK17]. They highlight the lack of focus on presentation of 3D data and applications and propose three gesture sets for such presentations that they also evaluate.

Finally, hybrid interaction has been used in the context of cultural heritage visualization in public dissemination. For example, a combined touch interface and HMD installation was produced by Sundén et al. to visualize 3D reconstruction of ship wrecks [SLY17].

### 4.3. The challenge of evaluation

Our state of the art report did not detail how each of the techniques we surveyed has been evaluated, although doing so would also highlight how challenging the evaluation of such new techniques and interfaces is. Indeed, these techniques are designed mostly for experts and researchers in specific domains. Domain experts using data visualization for sense-making rarely evaluate visualizations based on completion time or error-based measures but rather on the potential to turn data into insights. The visualization community has, therefore, sought to develop more holistic strategies to assess the quality of interactive visualization techniques† However, as we found in this survey, it is still rare to see published work that integrates an evaluation not based on traditional metrics. Since 3D visualization techniques often target a specific audience of domain experts, the alternative evaluation strategies that are most common focus on feedback from these users. Recruiting experts can be difficult, but the feedback they provide in user studies is invaluable.

Looking at the surveyed papers, however, seems to be promising as some of these approaches are evaluated without relying on traditional statistical analyses of time and errors. Some past work has relied on case studies instead [SYG*16, JJB18] or reports of use by real end users (i. e., domain experts) [TFK*02, RRA*12, MCB*18]. Others have relied on qualitative feedback from a small pool of experts, e. g., [GJL10, LRF*11, SMP*17, BIAI17a, Koe18]. It would therefore seem that the 3D spatial visualization community is open

to such evaluation strategies. Despite these positive aspects, it remains that most evaluations are focused on understanding how the techniques work in a research lab environment. The lack of adoption of 3D spatial visualization techniques, as mentioned in Sect. 4.1, could also be explained by the lack of focus on how newly developed techniques can integrate within the workflows and environments used by experts, as indicated in previous surveys and position papers [MHWH17, WBG*19]. To degree to which experts adopt visualization tools and integrate them into their workflows is a clear measure of the impact of the work, but this is only sometimes brought to the forefront in research publications (e. g., [LRF*11, RRA*12, BIAI17a, MCB*18, WDB*20]).

While some of the papers surveyed are accompanied by open-source implementations, which help with adoption and follows current recommended practices [BPSS*20], it is not clear how many of these have been or will eventually be adopted outside of a lab environment. We know of some success stories [YRA*16, Hol19, KAM*08, CML*12], but others may also exist. 3D visualization interaction techniques are increasingly being adopted in public venues to disseminate scientific findings (see [YRA*16]), or in the planning of complex and news-worthy surgical procedures [Hol19]. These are highly visible outlets, but no specific 3D visualization research is cited. All in all, while we thoroughly examined all surveyed papers and tried to find which of the techniques have been adopted by domain experts, finding specific examples remains particularly challenging and one can question whether it is due to the fact that most techniques are not eventually adopted by experts or whether such adoption is not as relevant to the visualization community as the basic research. To disambiguate this in the future, the visualization community could try, much like case reports in the medical fields, to propose a venue or track to report on the successful (or failed) adoption of visualization techniques by domain experts.

### 4.4. The potential of hybrid interaction paradigms for 3D Visualization

Combining different interaction paradigms to leverage their inherent benefits and mitigate their limitations has been the focus of multiple research projects [SSD09, CSH*14, MBH14, LODI16, SvZP*16, BIAI17a, BAI17, BSY*19, Bes17]. Our survey highlighted that hy-

<footnote>† For example, see the publications in the BELIV workshop series.</footnote>
brid interaction techniques have successfully been used for 3D visualiza-
tion purposes and represent a promising area for future work.
Looking closely at the surveyed work, however, highlights that al-
most all of the hybrid approaches focus on combining touch and
tangible interaction (a detailed classification of all hybrid interaction
papers is available as supplementary material at https://osf.io/jtx75/).
Indeed only three papers focus on different combinations: Man-
dalika et al. [MCB*18] propose to combine a regular mouse and
keyboard with a tangible device (the zSpace device) for radiological
diagnosis; Tsang et al. [TFK*02] combine touch interaction, tangi-
ble interaction and voice input to facilitate annotations; and Lubos
et al. [LBLS14] combine mid-air gestural interaction with voice
recognition to create an annotation tool in VR. Research outside
of 3D visualization suggests a number of other possible combina-
tions: pressure and tactile interaction (e. g., [CVLB18]), pressure
and tangible interaction (e. g., [BAI17]), mid-air gestural interaction
and tactile interaction (e. g., [WB03, HIW*09]), mid-air gestural
interaction with tangible interaction (e. g., [SLM*03]). Our survey,
therefore, suggests that such other combinations should also be
investigated for use with 3D visualization.

Looking at Table 1, we can clearly see that tangible interaction
has been used more than other paradigms to provide Visualization
Widget placement and manipulation. As described in Sect. 4.1, addi-
tional studies with other paradigms are needed, but the current trend
could also be explained by the inherent and natural 3D positioning
that tangible interaction can provide to users. It is likely that there is
good reason to consider this interaction paradigm when implement-
ing a new 3D visualization technique, and to potentially combine it
with other interaction techniques.

Another take-away from this survey is the lack of focus on voice-
input for 3D visualization purposes. In particular, our survey only
revealed examples of voice input for annotation. While we have
noted in Sect. 2.1 that using voice for direct manipulations is gener-
ally discouraged [KI13], voice input has been and can be used for
visual analytic purposes. Natural language has been used to query
data via speech (e. g., [CGH*01, GDA*15, DMN*17, SS18, YS20])
and typing (e. g., [YS20]). Past work [SS18] has highlighted the
potential to combine natural language with other input modalities
and called for more research on this topic in visualization. Our
survey highlights that such work also has to be conducted in the
specific case of 3D visualization, beyond the more classical and
straightforward use of speech for capturing annotations.

5. Conclusion

In our report we have surveyed interaction techniques designed
to assist domain experts who rely on 3D spatial visualization. We
have discussed techniques relying on tactile, tangible, gestural in-
put or hybrid combinations of interaction paradigms, and classified
each technique based on the 3D visualization task it supports. We
used three high-level task groups: (1) Volumetric view and object
manipulation, (2) defining, placing, & manipulating visualization
widgets, and (3) 3D data selection & annotation. Our classifica-
tion highlighted the tremendous amount of effort put into two of
these tasks. In particular, *Volumetric view and object manipulation*
and *3D data selection* have been well covered with all interaction
paradigms. A possible explanation lies in the overlap of interest in

these tasks between the visualization community and other HCI and
VR communities. However, the tasks within visualization widget
manipulation, while also being essential (as highlighted in Sect. 2.2,
appear to have received less attention. Similarly, but perhaps to a
lesser extent, while we grouped annotation with 3D data selection,
we can see from our survey that it has been investigated less fre-
quently than 3D data selection. We hypothesize that placing an
increased emphasis on interactive techniques for manipulating vi-
sualization widgets and making annotations may lead to a better
adoption rate for visualization techniques.

Another pertinent take-away message from our report is the lack
of studies on how to best use the various interaction paradigms to
disseminate scientific knowledge in public venues. While much of
this literature has focused on the dissemination of and interaction
with 2D data or slide-based presentations (see our previous discus-
sion in Sect. 4), only a few studies have investigated these topics
with 3D data. We believe, however, that the potential of these inter-
action paradigms goes beyond supporting scientific discoveries to
also include engaged learning [XAM08, HCB12, BIAI17b].

Our survey also highlighted the benefits of each interaction
paradigm that are most applicable to addressing the challenges
of 3D spatial visualization. Our final report on hybrid interaction
paradigms (see Sect. 3.4) highlighted the potential of leveraging the
benefits of multiple interaction paradigms to address the challenges
of 3D Spatial data visualization. Moreover, it is essential to consider
the unique affordances of interaction paradigms in different scenar-
ios or contexts. For instance, when analyzing spatial data in a CAVE
(wearing a VR/AR HMD or a tracked pair of glasses) it is easy and
effective to change viewpoint to view the visualization from dif-
ferent perspectives [LBS14, BSB*18, MSD*18, WSS20]. However,
when the task is to analyze 3D volume data in the operating room,
the space limitation, the light condition as well as the convenience
of head-worn cameras need to be taken into consideration.

Our survey also discusses the challenges of evaluating 3D vi-
sualization techniques as well as the difficulty to knowing which
techniques have been successfully adopted by domain experts in
their workflows. To address this challenge, we recommend that the
visualization community creates additional opportunities for con-
tributing such reports, much like the tradition within the medical
research community of publishing case reports.

Finally, our survey can be used to support the evaluation of future
work: researchers can use our table to identify relevant related work
to compare to new approaches.

6. Acknowledgements

This work was supported in part by the NSF (IIS-1704604) and
by MIT/IAS at Linköping University. We wish to thank all our
reviewers and editors for their voluntary work and their helpful
feedback and comments on several versions of our manuscript. We
would like to thank Karin Ivarsson for her help in proof-reading this
manuscript. Finally, we thank all the authors we cited who provided
open-access version of their papers. Whenever possible, we have
linked to such open-access versions in the references.
## References

[AA09] ARGELAGUET F., ANDUJAR C.: Efficient 3D pointing selection in cluttered virtual environments. *IEEE Computer Graphics and Applications* 29, 6 (Nov./Dec. 2009), 34–43. Ⓔ Open Access. doi: 10.1109/MCG. 2009.117

[AA13] ARGELAGUET F., ANDUJAR C.: A survey of 3D object selection techniques for virtual environments. *Computers & Graphics* 37, 3 (May 2013), 121–136. Ⓔ Open Access. doi: 10.1016/j.cag.2012.12.003

[AAT08] ARGELAGUET F., ANDUJAR C., TRUEBA R.: Overcoming eye-hand visibility mismatch in 3D pointing selection. In *Proc. VRST* (2008), ACM, New York, pp. 43–46. Ⓔ Open Access. doi: 10.1145/1450579. 1450588

[AFT*04] ALLEN P., FEINER S., TROCCOLI A., BENKO H., ISHAK E., SMITH B.: Seeing into the past: Creating a 3D modeling pipeline for archaeological visualization. In *Proc. 3DPVT* (2004), IEEE Computer Society, Los Alamitos, pp. 751–758. Ⓔ Open Access. doi: 10.1109/TDPVT. 2004.1335391

[AGLW16] AMMA C., GEORGI M., LENZ T., WINNEN F.: Kinematic wave: A mobile freehand gesture and text-entry system. In *CHI Extended Abstracts* (2016), ACM, New York, pp. 3639–3642. doi: 10.1145/2851581. 2890263

[Ais79] AISH R.: 3D input for CAAD systems. *Computer-Aided Design* 11, 2 (Mar. 1979), 66–70. Ⓔ Open Access. doi: 10.1016/0010-4485(79)90098-8

[Ake06] AKERS D.: CINCH: A cooperatively designed marking interface for 3D pathway selection. In *Proc. UIST* (2006), ACM, New York, pp. 33–42. Ⓔ Open Access. doi: 10.1145/1166253.1166260

[AMR16] AL-MEGREN S., RUDDLE R. A.: Comparing tangible and multi-touch interaction for interactive data visualization tasks. In *Proc. TEI* (2016), ACM, New York, pp. 279–286. Ⓔ Open Access. doi: 10. 1145/2839462.2839464

[AS95] ANGUS I. G., SOWIZRAL H. A.: VRMosaic: WEB access from within a virtual environment. In *Proc. Visualization* (1995), IEEE Computer Society, Los Alamitos, pp. 59–64. doi: 10.1109/INFVIS.1995.528687

[AS96] AVILA R. S., SOBIERAJSKI L. M.: A haptic interaction method for volume visualization. In *Proc. Visualization* (1996), IEEE Computer Society, Los Alamitos, pp. 197–204. Ⓔ Open Access. doi: 10.1109/VISUAL. 1996.568108

[ATF12] AU O. K.-C., TAI C.-L., FU H.: Multitouch gestures for constrained transformation of 3D objects. *Computer Graphics Forum* 31, 2pt3 (May 2012), 651–660. Ⓔ Open Access. doi: 10.1111/j.1467-8659.2012. 03044.x

[AZ03] ALBINSON P.-A., ZHAI S.: High precision touch screen interaction. In *Proc. CHI* (2003), ACM, New York, pp. 105–112. Ⓔ Open Access. doi: 10.1145/642611.642631

[BAC*08] BONANNI L., ALONSO J., CHAO N., VARGAS G., ISHII H.: Handsaw: Tangible exploration of volumetric data by direct cut-plane projection. In *Proc. CHI* (2008), ACM, New York, pp. 251–254. Ⓔ Open Access. doi: 10.1145/1357054.1357098

[BAI17] BESANÇON L., AMMI M., ISENBERG T.: Pressure-based gain factor control for mobile 3D interaction using locally-coupled devices. In *Proc. CHI* (2017), ACM, New York, pp. 1831–1842. Ⓔ Open Access. doi: 10.1145/3025453.3025890

[Ban14] BANIC A.: Selection classification for interaction with immersive volumetric visualizations. In *Human Interface and the Management of Information. Information and Knowledge Design and Evaluation* (2014), Springer International, Cham, pp. 10–21. Ⓔ Open Access. doi: 10.1007/978-3-319-07731-4_2

[BBK*06] BORNIK A., BEICHEL R., KRUIJFF E., REITINGER B., SCHMALSTIEG D.: A hybrid user interface for manipulation of volumetric medical data. In *Proc. 3DUI* (2006), IEEE Computer Society, Los Alamitos, pp. 29–36. Ⓔ Open Access. doi: 10.1109/VR.2006.8

[BCD*18] BÜSCHEL W., CHEN J., DACHSELT R., DRUCKER S., DWYER T., GÖRG C., ISENBERG T., KERREN A., NORTH C., STUER-ZLINGER W.: Interaction for immersive analytics. In *Immersive Analytics*,

Marriott K., Schreiber F., Dwyer T., Klein K., Riche N. H., Itoh T., Stuer-zlinger W., Thomas B. H., (Eds.). Springer, Berlin/Heidelberg, 2018, ch. 4, pp. 95–138. Ⓔ Open Access. doi: 10.1007/978-3-030-01388-2_4

[BDP11] BIRR S., DACHSELT R., PREIM B.: Mobile interactive displays for medical visualization. In *Proc. DEXIS* (2011), INRIA, Le Chesnay, France, pp. 28–31. Ⓔ Open Access.

[Bes17] BESANÇON L.: An interaction Continuum for 3D Dataset Visualization. PhD thesis, Université Paris-Saclay, France, Dec. 2017. Ⓔ Open Access.

[BHMN95] BOLTER J., HODGES L. F., MEYER T., NICHOLS A.: Integrating perceptual and symbolic information in VR. *IEEE Computer Graphics and Applications* 15, 4 (July 1995), 8–11. Ⓔ Open Access. doi: 10.1109/38.391481

[BHWB07] BEYER J., HADWIGER M., WOLFSBERGER S., BÜHLER K.: High-quality multimodal volume rendering for preoperative planning of neurosurgical interventions. *IEEE Transactions on Visualization and Computer Graphics* 13, 6 (Nov. 2007), 1696–1703. Ⓔ Open Access. doi: 10.1109/TVCG.2007.70560

[BHY18] BOCK A., HANSEN C., YNNERMAN A.: OpenSpace: Bringing NASA missions to the public. *IEEE Computer Graphics and Applications* 38, 5 (Sep. 2018), 112–118. Ⓔ Open Access. doi: 10.1109/MCG.2018. 053491735

[BI05] BENKO H., ISHAK E. W.: Cross-dimensional gestural interaction techniques for hybrid immersive environments. In *Proc. VR* (2005), IEEE Computer Society, Los Alamitos, pp. 209–216, 327. Ⓔ Open Access. doi: 10.1109/VR.2005.27

[BIAI17a] BESANÇON L., ISSARTEL P., AMMI M., ISENBERG T.: Hybrid tactile/tangible interaction for 3D data exploration. *IEEE Transactions on Visualization and Computer Graphics* 23, 1 (Jan. 2017), 881–890. Ⓔ Open Access. doi: 10.1109/TVCG.2016.2599217

[BIAI17b] BESANÇON L., ISSARTEL P., AMMI M., ISENBERG T.: Mouse, tactile, and tangible input for 3D manipulation. In *Proc. CHI* (2017), ACM, New York, pp. 4727–4740. Ⓔ Open Access. doi: 10.1145/ 3025453.3025863

[BIH08] BUTLER A., IZADI S., HODGES S.: SideSight: Multi-"touch" interaction around small devices. In *Proc. UIST* (2008), ACM, New York, pp. 201–204. Ⓔ Open Access. doi: 10.1145/1449715.1449746

[BIRW19] BRUCKNER S., ISENBERG T., ROPINSKI T., WIEBEL A.: A model of spatial directness in interactive visualization. *IEEE Transactions on Visualization and Computer Graphics* 25, 8 (Aug. 2019), 2514–2528. Ⓔ Open Access. doi: 10.1109/TVCG.2018.2848906

[BJS*98] BRYSON S., JOHAN S., SCHLECHT L., GREEN B., KEN-WRIGHT D., GERALD-YAMASAKI M.: The virtual WindTunnel. In *Computational Fluid Dynamics Review*. World Scientific, Singapore, 1998, pp. 1113–1130. Ⓔ Open Access. doi: 10.1142/9789812812957_0062

[BKBN12] BLUM T., KLEEBERGER V., BICHLMEIER C., NAVAB N.: mirracle: An augmented reality magic mirror system for anatomy education. In *IEEE Virtual Reality Workshops* (2012), pp. 115–116. Ⓔ Open Access. doi: 10.1109/VR.2012.6180909

[BM13] BREHMER M., MUNZNER T.: A multi-level typology of abstract visualization tasks. *IEEE Transactions on Visualization and Computer Graphics* 19, 12 (Dec. 2013), 2376–2385. Ⓔ Open Access. doi: 10.1109/TVCG.2013.124

[BNB14] BACIM F., NABIYOUNI M., BOWMAN D. A.: Slice-n-swipe: A free-hand gesture user interface for 3D point cloud annotation. In *Proc. 3DUI* (2014), IEEE Computer Society, Los Alamitos, pp. 185–186. Ⓔ Open Access. doi: 10.1109/3DUI.2014.6798882

[BODVOGHV17] BOTERO-OSPINA A., DUQUE-VALLEJO S., OCHOA-GÓMEZ J., HERNÁNDEZ-VALDIVIESO A.: Touchless control module for diagnostic images at the surgery room using the Leap Motion system and 3D Slicer Software. *Revista Facultad de Ingeniería, Universidad de Antioquia* (Mar. 2017), 40–46. Ⓔ Open Access. doi: 10.17533/udea.redin. n82a05
[BPC19] BALOUP M., PIETRZAK T., CASIEZ G.: RayCursor: A 3D pointing facilitation technique based on raycasting. In Proc. CHI (2019), ACM, New York, pp. 101:1–101:12. ⑆ Open Access. doi: 10.1145/3290605.3300331

[BPSS*20] BESANÇON L., PEIFFER-SMADJA N., SEGALAS C., JIANG H., MASUZZO P., SMOUT C., DEFORET M., LEYRAT C.: *Open Science Saves Lives: Lessons from the COVID-19 Pandemic*. biorxiv preprint, 2020. ⑆ Open Access. doi: 10.1101/2020.08.13.249847

[BS19] BATMAZ A. U., STUERZLINGER W.: Effects of 3D rotational jitter and selection methods on 3D pointing tasks. In Proc. VR (2019), IEEE Computer Society, Los Alamitos, pp. 1687–1692. ⑆ Open Access. doi: 10.1109/VR.2019.8798038

[BSB*18] BACH B., SICAT R., BEYER J., CORDEIL M., PFISTER H.: The hologram in my hand: How effective is interactive exploration of 3D visualizations in immersive tangible augmented reality? *IEEE Transactions on Visualization and Computer Graphics* 24, 1 (Jan. 2018), 457–467. ⑆ Open Access. doi: 10.1109/TVCG.2017.2745941

[BSW19] BUTKIEWICZ T., STEVENS A. H., WARE C.: Multi-touch 3D positioning with the pantograph technique. In Proc. I3D (2019), ACM, New York, pp. 13:1–13:9. ⑆ Open Access. doi: 10.1145/3306131.3317014

[BSY*19] BESANÇON L., SERENO M., YU L., AMMI M., ISENBERG T.: Hybrid touch/tangible spatial 3D data selection. *Computer Graphics Forum* 38, 3 (June 2019), 553–567. ⑆ Open Access. doi: 10.1111/cgf.13710

[Bux95] BUXTON W.: Integrating the periphery and context: A new taxonomy of telematics. In Proc. Graphics Interface (1995), CHCCS, pp. 239–246. ⑆ Open Access. doi: 10.20380/GI1995.28

[BW10] BENKO H., WILSON A. D.: Multi-point interactions with immersive omnidirectional visualizations in a dome. In Proc. ITS (2010), ACM, New York, pp. 19–28. ⑆ Open Access. doi: 10.1145/1936652.1936657

[BW11] BUTKIEWICZ T., WARE C.: Multi-touch 3D exploratory analysis of ocean flow models. In Proc. OCEANS (2011), IEEE, Los Alamitos, pp. 746–755. ⑆ Open Access. doi: 10.23919/OCEANS.2011.6107079

[CBL*17] CORDEIL M., BACH B., LI Y., WILSON E., DWYER T.: Design space for spatio-data coordination: Tangible interaction devices for immersive information visualisation. In Proc. PacificVis (2017), IEEE Computer Society, Los Alamitos, pp. 46–50. ⑆ Open Access. doi: 10.1109/PACIFICVIS.2017.8031578

[CDH11] COHÉ A., DÈCLE F., HACHET M.: tBox: A 3D transformation widget designed for touch-screens. In Proc. CHI (2011), ACM, New York, pp. 3005–3008. ⑆ Open Access. doi: 10.1145/1978942.1979387

[CFH97] CUTLER L. D., FRÖHLICH B., HANRAHAN P.: Two-handed direct manipulation on the responsive workbench. In Proc. I3D (1997), ACM, New York, pp. 107–114. ⑆ Open Access. doi: 10.1145/253284.253315

[CFM*12] CUCCURULLO S., FRANCESE R., MURAD S., PASSERO I., TUCCI M.: A gestural approach to presentation exploiting motion capture metaphors. In Proc. AVI (2012), ACM, New York, pp. 148–155. doi: 10.1145/2254556.2254584

[CG17] CLERGEAUD D., GUITTON P.: Design of an annotation system for taking notes in virtual reality. In Proc. 3DTV-CON (2017), IEEE, Los Alamitos, pp. 1–4. ⑆ Open Access. doi: 10.1109/3DTV.2017.8280398

[CGH*01] COX K., GRINTER R. E., HIBINO S. L., JAGADEESAN L. J., MANTILLA D.: A multi-modal natural language interface to an information visualization environment. *International Journal of Speech Technology* 4, 3-4 (2001), 297–314. ⑆ Open Access. doi: 10.1023/A:1011368926479

[CGM*14] CHAKRABORTY A., GROSS R., MCINTEE S., HONG K. W., LEE J. Y., ST. AMANT R.: CAPTIVE: A cube with augmented physical tools. In CHI Extended Abstracts (2014), ACM, New York, pp. 1315–1320. doi: 10.1145/2559206.2581340

[CI09] CASSINELLI A., ISHIKAWA M.: Volume slicing display. In *SIGGRAPH ASIA Art Gallery & Emerging Technologies* (2009), p. 88. ⑆ Open Access. doi: 10.1145/1665137.1665207

[CJWC75] COLWELL JR WILLIAM C H. G. S.: Discriminating contact sensor, Oct. 7 1975. US Patent 3,911,215.

[CL17] CHAN L. K. Y., LAU H. Y. K.: MagicPad HD: The spatial user interface. In Proc. HCI (2017), Springer International, Cham, Switzerland, pp. 559–572. doi: 10.1007/978-3-319-58071-5_42

[CML*12] COFFEY D., MALBRAATEN N., LE T., BORAZJANI I., SOTIROPOULOS F., ERDMAN A. G., KEEFE D. F.: Interactive slice WIM: Navigating and interrogating volume datasets using a multi-surface, multi-touch VR interface. *IEEE Transactions on Visualization and Computer Graphics* 18, 10 (Oct. 2012), 1614–1626. doi: 10.1109/TVCG.2011.283

[CMS88] CHEN M., MOUNTFORD S. J., SELLEN A.: A study in interactive 3-D rotation using 2-D control devices. *ACM SIGGRAPH Computer Graphics* 22, 4 (Aug. 1988), 121–129. ⑆ Open Access. doi: 10.1145/378456.378497

[CON08] CHRISTIE M., OLIVIER P., NORMAND J.-M.: Camera control in computer graphics. *Computer Graphics Forum* 27, 8 (Dec. 2008), 2197–2218. ⑆ Open Access. doi: 10.1111/j.1467-8659.2008.01181.x

[COPG15] CORENTHY L., OTADUY M. A., PASTOR L., GARCIA M.: Volume haptics with topology-consistent isosurfaces. *IEEE Transactions on Haptics* 8, 4 (Oct. 2015), 480–491. ⑆ Open Access. doi: 10.1109/TOH.2015.2466239

[CPLCPFR14] CHAARAOUI A. A., PADILLA-LÓPEZ J. R., CLIMENT-PÉREZ F., FLÓREZ-REVUELTA F.: Evolutionary joint selection to improve human action recognition with RGB-D devices. *Expert Systems with Applications* 41, 3 (2014), 786–794. ⑆ Open Access. doi: 10.1016/j.eswa.2013.08.009

[CR96] CHUAH M. C., ROTH S. F.: On the semantics of interactive visualizations. In Proc. InfoVis (1996), IEEE Computer Society, Los Alamitos, pp. 29–36. ⑆ Open Access. doi: 10.1109/INFVIS.1996.559213

[CRR08] COUTURE N., RIVIÈRE G., REUTER P.: GeoTUI: A tangible user interface for geoscience. In Proc. TEI (2008), ACM, New York, pp. 89–96. ⑆ Open Access. doi: 10.1145/1347390.1347411

[CSD03] COURNIA N., SMITH J. D., DUCHOWSKI A. T.: Gaze-vs. hand-based pointing in virtual environments. In *CHI Extended Abstracts* (2003), ACM, New York, pp. 772–773. ⑆ Open Access. doi: 10.1145/765891.765982

[CSH*14] CHEN X. A., SCHWARZ J., HARRISON C., MANKOFF J., HUDSON S. E.: Air+touch: Interweaving touch & in-air gestures. In Proc. UIST (2014), ACM, New York, pp. 519–525. ⑆ Open Access. doi: 10.1145/2642918.2647392

[CSVBS15] COSTA SOUSA M., VITAL BRAZIL E., SHARLIN E.: Scalable and interactive visual computing in geosciences and reservoir engineering. *Geological Society, London, Special Publications* 406, 1 (2015), 447–466. ⑆ Open Access. doi: 10.1144/SP406.17

[CT05] COOK K. A., THOMAS J. J.: Illuminating the Path: The Research and Development Agenda for Visual Analytics. Department of Homeland Security, USA, 2005. ⑆ Open Access.

[CVLB18] CORSTEN C., VOELKER S., LINK A., BORCHERS J.: Use the force picker, Luke: Space-efficient value input on force-sensitive mobile touchscreens. In Proc. CHI (2018), ACM, New York, pp. 661:1–661:12. ⑆ Open Access. doi: 10.1145/3173574.3174235

[DD05] DARKEN R. P., DUROST R.: Mixed-dimension interaction in virtual environments. In Proc. VRST (2005), ACM, New York, pp. 38–45. ⑆ Open Access. doi: 10.1145/1101616.1101625

[DGHCM*03] DE GUZMAN E., HO-CHING F., MATTHEWS T., RAT-TENBURY T., BACK M., HARRISON S.: EEWWW!!: Tangible interfaces for navigating into the human body. In *CHI Extended Abstracts* (2003), ACM, New York, pp. 806–807. ⑆ Open Access. doi: 10.1145/765891.766003

[dHKP02] DE HAAN G., KOUTEK M., POST F. H.: Towards intuitive exploration tools for data visualization in VR. In Proc. VRST (2002), ACM, New York, pp. 105–112. ⑆ Open Access. doi: 10.1145/585740.585758

[dHKP05a] DE HAAN G., KOUTEK M., POST F. H.: IntenSelect: Using dynamic object rating for assisting 3D object selection. In Proc. EGVE (2005), Eurographics Association, Goslar, Germany, pp. 201–209. ⑆ Open Access. doi: 10.2312/EGVE/IPT_EGVE2005/201-209
[DHKP05b] DE HAAN G., KOUTEK M., POST F. H.: IntenSelect: Using dynamic object rating for assisting 3D object selection. In *Proc. IPT/EGVE* (2005), Eurographics Association, Goslar, Germany, pp. 201–209. Ⓘ *Open Access*. doi: 10.2312/EGVE/IPT_EGVE2005/201-209

[dIKRKOD08] DE LA RIVIÈRE J.-B., KERVÉGANT C., ORVAIN E., DIT-TLO N.: CubTile: A multi-touch cubic interface. In *Proc. VRST* (2008), ACM, New York, pp. 69–72. doi: 10.1145/1450579.1450593

[DMN*17] DHAMDHERE K., MCCURLEY K. S., NAHMIAS R., SUNDARARAJAN M., YAN Q.: Analyza: Exploring data with conversation. In *Proc. IUI* (2017), ACM, New York, pp. 493–504. Ⓘ *Open Access*. doi: 10.1145/3025171.3025227

[EAB*15] EIVAZI S., AFKARI H., BEDNARIK R., LEINONEN V., TUKIAINEN M., JÄÄSKELÄINEN J. E.: Analysis of disruptive events and precarious situations caused by interaction with neurosurgical microscope. *Acta Neurochirurgica* 157, 7 (July 2015), 1147–1154. Ⓘ *Open Access*. doi: 10.1007/s00701-015-2433-5

[EAG17] ESCALERA S., ATHITSOS V., GUYON I.: Challenges in multimodal gesture recognition. In * Gesture Recognition*, Escalera S., Guyon I., Athitsos V., (Eds.). Springer International, Cham, Switzerland, 2017, pp. 1–60. Ⓘ *Open Access*. doi: 10.1007/978-3-319-57021-1_1

[EJG73] EBELING F., JOHNSON R., GOLDHOR R.: Infrared light beam x-y position encoder for display devices, Nov. 27 1973. US Patent 3,775,560.

[FBZ*99] FRÖHLICH B., BARRASS S., ZEHNER B., PLATE J., GÖBEL M.: Exploring geo-scientific data in virtual environments. In *Proc. Visualization* (1999), IEEE Computer Society, Los Alamitos, pp. 169–173. Ⓘ *Open Access*. doi: 10.1109/Visual.1999.809884

[FCC*15] FUJISHIRO I., CHEN B.-Y., CHEN W., HONG S.-H., ITOH T., KOYAMADA K., ONO K., NONAKA J.: Top computational visualization R&D problems 2015: Panel. In *Proc. SIGGRAPH Asia Symposium on Visualization in High Performance Computing* (2015), ACM, New York, pp. 20:1–20:13. doi: 10.1145/2818517.2818545

[FDGS12] FLEURY C., DUVAL T., GOURANTON V., STEED A.: Evaluation of remote collaborative manipulation for scientific data analysis. In *Proc. VRST* (2012), ACM, New York, pp. 129–136. Ⓘ *Open Access*. doi: 10.1145/2407336.2407361

[FFF80] FRAZER J. H., FRAZER J. M., FRAZER P. A.: Intelligent physical three-dimensional modelling system. In *Proc. Computer Graphics* (1980), pp. 359–370.

[FFH13] FORBES A. G., FAST T., HÖLLERER T.: The natural materials browser: Using a tablet interface for exploring volumetric materials science datasets. In *IEEE Visualization Posters* (2013). Ⓘ *Open Access*.

[FG98] FUHRMANN A., GRÖLLER E.: Real-time techniques for 3D flow visualization. In *Proc. Visualization* (1998), IEEE Computer Society, Los Alamitos, pp. 305–312. Ⓘ *Open Access*. doi: 10.1109/Visual.1998.745317

[FGN10] FU C.-W., GOH W.-B., NG J. A.: Multi-touch techniques for exploring large-scale 3D astrophysical simulations. In *Proc. CHI* (2010), ACM, New York, pp. 2213–2222. Ⓘ *Open Access*. doi: 10.1145/1753326.1753661

[FH00] FOXLIN E., HARRINGTON M.: WearTrack: A self-referenced head and hand tracker for wearable computers and portable VR. In *Proc. ISWC* (2000), IEEE Computer Society, Los Alamitos, pp. 155–162. Ⓘ *Open Access*. doi: 10.1109/ISWC.2000.888482

[FHZ96] FORSBERG A., HERNDON K., ZELEZNIK R.: Aperture based selection for immersive virtual environments. In *Proc. UIST* (1996), ACM, New York, pp. 95–96. Ⓘ *Open Access*. doi: 10.1145/237091.237105

[Fit96] FITZMAURICE G. W.: Graspable User Interfaces. PhD thesis, University of Toronto, Canada, 1996. Ⓘ *Open Access*.

[FKK07] FREES S., KESSLER G. D., KAY E.: PRISM interaction for enhancing control in immersive virtual environments. *ACM Transactions on Computer-Human Interaction* 14, 1 (May 2007), 2:1–2:31. Ⓘ *Open Access*. doi: 10.1145/1229855.1229857

[FMHR87] FISHER S. S., MCGREEVY M., HUMPHRIES J., ROBINETT W.: Virtual environment display system. In *Proc. IED* (1987), ACM, New York, pp. 77–87. Ⓘ *Open Access*. doi: 10.1145/319120.319127

[Fre93] FREYMUTH P.: Flow visualization in fluid mechanics. *Review of Scientific Instruments* 64, 1 (1993), 1–18. doi: 10.1063/1.1144433

[FTW12] FREITAG G., TRÄNKNER M., WACKER M.: Enhanced feed-forward for a user aware multi-touch device. In *Proc. NordiCHI* (2012), ACM, New York, pp. 578–586. Ⓘ *Open Access*. doi: 10.1145/2399016.2399104

[FWSB07] FORLINES C., WIGDOR D., SHEN C., BALAKRISHNAN R.: Direct-touch vs. mouse input for tabletop displays. In *Proc. CHI* (2007), ACM, New York, pp. 647–656. Ⓘ *Open Access*. doi: 10.1145/1240624.1240726

[GAI*16] GROENEWALD C., ANSLOW C., ISLAM J., ROONEY C., PASSMORE P., WONG W.: Understanding 3D mid-air hand gestures with interactive surfaces and displays: A systematic literature review. In *Proc. HCI* (2016), BCS Learning & Development Ltd., Swindon, UK, pp. 43:1–43:13. Ⓘ *Open Access*. doi: 10.14236/ewic/HCI2016.43

[Gal13] GALLO L.: A study on the degrees of freedom in touchless interaction. In *SIGGRAPH Asia Technical Briefs* (2013), ACM, New York, pp. 28:1–28:4. Ⓘ *Open Access*. doi: 10.1145/2542355.2542390

[GB06] GROSSMAN T., BALAKRISHNAN R.: The design and evaluation of selection techniques for 3D volumetric displays. In *Proc. UIST* (2006), ACM, New York, pp. 3–12. Ⓘ *Open Access*. doi: 10.1145/1166253.1166257

[GBC13] GLESSER D., BÉRARD F., COOPERSTOCK J. R.: Overcoming limitations of the trackpad for 3D docking operations. In *CHI Extended Abstracts* (2013), ACM, New York, pp. 1239–1244. Ⓘ *Open Access*. doi: 10.1145/2468356.2468578

[GDA*15] GAO T., DONTCHEVA M., ADAR E., LIU Z., KARAHALIOS K. G.: Datatone: Managing ambiguity in natural language interfaces for data visualization. In *Proc. UIST* (2015), ACM, New York, pp. 489–500. Ⓘ *Open Access*. doi: 10.1145/2807442.2807478

[GHP*95] GOBLE J. C., HINCKLEY K., PAUSCH R., SNELL J. W., KASSELL N. F.: Two-handed spatial interface tools for neurosurgical planning. *Computer* 28, 7 (July 1995), 20–26. Ⓘ *Open Access*. doi: 10.1109/2.391037

[GJL10] GOMEZ S. R., JIANU R., LAIDLAW D. H.: A fiducial-based tangible user interface for white matter tractography. In *Advances in Visual Computing* (2010), Springer, Berlin, Heidelberg, pp. 373–381. Ⓘ *Open Access*. doi: 10.1007/978-3-642-17274-8_37

[GNBP11] GASTEIGER R., NEUGEBAUER M., BEUING O., PREIM B.: The flowlens: A focus-and-context visualization approach for exploration of blood flow in cerebral aneurysms. *IEEE Transactions on Visualization and Computer Graphics* 17, 12 (Dec. 2011), 2183–2192. Ⓘ *Open Access*.

[GPB99] GEE A., PRAGER R., BERMAN L.: Non-planar reslicing for freehand 3D ultrasound. In *Proc. MICCAI* (1999), Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 716–725. Ⓘ *Open Access*. doi: 10.1007/10704282.78

[GPC11] GALLO L., PLACITELLI A. P., CIAMPI M.: Controller-free exploration of medical image data: Experiencing the Kinect. In *Proc. CBMS* (2011), IEEE, Danvers, MA, USA, pp. 54:1–54:6. Ⓘ *Open Access*. doi: 10.1109/CBMS.2011.5999138

[GSBI20] GOSSET S., SERENO M., BESANÇON L., ISENBERG T.: Tangible Volumetric Brushing in Augmented Reality. IEEE VIS Posters, Oct. 2020. Ⓘ *Open Access*.

[GWB04] GROSSMAN T., WIGDOR D., BALAKRISHNAN R.: Multi-finger gestural interaction with 3D volumetric displays. In *Proc. UIST* (2004), ACM, New York, pp. 61–70. Ⓘ *Open Access*. doi: 10.1145/1029632.1029644

[Han97] HAND C.: A survey of 3D interaction techniques. *Computer Graphics Forum* 16, 5 (Dec. 1997), 269–281. Ⓘ *Open Access*. doi: 10.1111/1467-8659.00194
[HCB12] HORN M. S., CROUSER R. J., BERS M. U.: Tangible interaction and learning: The case for a hybrid approach. *Personal and Ubiquitous Computing* **16**, 4 (2012), 379–389. Ⓔ *Open Access*. doi: 10.1007/s00779-011-0404-2

[HCC07] HANCOCK M., CARPENDALE S., COCKBURN A.: Shallow-depth 3D interaction. In *Proc. CHI* (2007), ACM, New York, pp. 1147–1156. Ⓔ *Open Access*. doi: 10.1145/1240624.1240798

[HCM*16] HSUEH C., CHU J., MA K., MA J., FRAZIER J.: Fostering comparisons: Designing an interactive exhibit that visualizes marine animal behaviors. In *Proc. PacificVis* (2016), IEEE Computer Society, Los Alamitos, pp. 259–263. Ⓔ *Open Access*. doi: 10.1109/PACIFICVIS.2016.7465282

[HdIRL*13] HACHET M., DE LA RIVIÈRE J., LAVIOLE J., COHÉ A., CURSAN S.: Touch-based interfaces for interacting with 3D content in public exhibitions. *IEEE Computer Graphics and Applications* **33**, 2 (Mar. 2013), 80–85. Ⓔ *Open Access*. doi: 10.1109/MCG.2013.34

[Hib99] HIBBARD B.: Top ten visualization problems. *ACM SIGGRAPH Computer Graphics* **33**, 2 (May 1999), 21–22. Ⓔ *Open Access*. doi: 10.1145/326460.326485

[HIW*09] HILLIGES O., IZADI S., WILSON A. D., HODGES S., GARCIA-MENDOZA A., BUTZ A.: Interactions in the air: Adding further depth to interactive tabletops. In *Proc. UIST* (2009), ACM, New York, pp. 139–148. Ⓔ *Open Access*. doi: 10.1145/1622176.1622203

[HLN*17] HATSCHER B., LUZ M., NACKE L. E., ELKMANN N., MÜLLER V., HANSEN C.: Gazetap: Towards hands-free interaction in the operating room. In *Proc. ICMI* (2017), ACM, New York. Ⓔ *Open Access*. doi: 10.1145/3136755.3136759

[HMP*12] HERGHELEGIU P. C., MANTA V., PERIN R., BRUCKNER S., GRÖLLER E.: Biopsy planner – visual analysis for needle pathway planning in deep seated brain tumor biopsy. *Computer Graphics Forum* **31**, 3pt2 (2012), 1085–1094. Ⓔ *Open Access*. doi: 10.1111/j.1467-8659.2012.03101.x

[HMR*15] HETTIG J., MEWES A., RIABIKIN O., SKALEJ M., PREIM B., HANSEN C.: Exploration of 3D medical image data for interventional radiology using myoelectric gesture control. In *Proc. VCBM* (2015), Eurographics Association, Goslar, Germany, pp. 177–185. Ⓔ *Open Access*. doi: 10.2312/vcbm.20151221

[Hol19] HOLLEY P.: How doctors used virtual reality to save the lives of conjoined twin sisters, Apr. 2019.

[HPGK94] HINCKLEY K., PAUSCH R., GOBLE J. C., KASSELL N. F.: Passive real-world interface props for neurosurgical visualization. In *Proc. CHI* (1994), ACM, New York, pp. 452–458. Ⓔ *Open Access*. doi: 10.1145/191666.191821

[HPMD13] HÖTKER A. M., PITTON M. B., MILDENBERGER P., DÜBER C.: Speech and motion control for interventional radiology: requirements and feasibility. *International Journal of Computer Assisted Radiology and Surgery* **8**, 6 (Nov. 2013), 997–1002. doi: 10.1007/s11548-013-0841-7

[HPRB96] HARMON R., PATTERSON W., RIBARSKY W., BOLTER J.: The virtual annotation system. In *Proc. VRAIS* (1996), IEEE Computer Society, Los Alamitos, pp. 239–245. Ⓔ *Open Access*. doi: 10.1109/VRAIS.1996.490533

[HPSH00] HINCKLEY K., PIERCE J., SINCLAIR M., HORVITZ E.: Sensing techniques for mobile interaction. In *Proc. UIST* (2000), ACM, New York, pp. 91–100. Ⓔ *Open Access*. doi: 10.1145/354401.354417

[HRD*19] HURTER C., RICHE N. H., DRUCKER S. M., CORDEIL M., ALLIGIER R., VUILLEMOT R.: FiberClay: Sculpting three dimensional trajectories to reveal structural insights. *IEEE Transactions on Visualization and Computer Graphics* **25**, 1 (Jan. 2019), 704–714. Ⓔ *Open Access*. doi: 10.1109/TVCG.2018.2865191

[HS04] HOCHHEISER H., SHNEIDERMAN B.: Dynamic query tools for time series data sets: Timebox widgets for interactive exploration. Ⓔ *Open Access*. doi: 10.1145/993176.993177

[HTM10] HARRISON C., TAN D., MORRIS D.: Skinput: Appropriating the body as an input surface. In *Proc. CHI* (2010), ACM, New York, pp. 453–462. Ⓔ *Open Access*. doi: 10.1145/1753326.1753394

[HTP*97] HINCKLEY K., TULLIO J., PAUSCH R., PROFFITT D., KASSELL N.: Usability analysis of 3D rotation techniques. In *Proc. UIST* (1997), ACM, New York, pp. 1–10. Ⓔ *Open Access*. doi: 10.1145/263407.263408

[HWS02] HARDERS M., WILDERMUTH S., SZÉKELY G.: New paradigms for interactive 3D volume segmentation. *The Journal of Visualization and Computer Animation* **13**, 1 (Feb. 2002), 85–95. doi: 10.1002/vis.277

[IBIA16] ISSARTEL P., BESANÇON L., ISENBERG T., AMMI M.: A tangible volume for portable 3D interaction. In *Adjunct Proc. ISMAR* (2016), IEEE Computer Society, Los Alamitos, pp. 215–220. Ⓔ *Open Access*. doi: 10.1109/ISMAR-Adjunct.2016.0079

[IGA14a] ISSARTEL P., GUÉNIAT F., AMMI M.: A portable interface for tangible exploration of volumetric data. In *Proc. VRST* (2014), ACM, New York, pp. 209–210. Ⓔ *Open Access*. doi: 10.1145/2671015.2671130

[IGA14b] ISSARTEL P., GUÉNIAT F., AMMI M.: Slicing techniques for handheld augmented reality. In *Proc. 3DUI* (2014), IEEE Computer Society, Los Alamitos, pp. 39–42. Ⓔ *Open Access*. doi: 10.1109/3DUI.2014.6798839

[Ise16] ISENBERG T.: Interactive exploration of three-dimensional scientific visualizations on large display surfaces. In *Collaboration Meets Interactive Spaces*. Springer, Berlin/Heidelberg, 2016, ch. 6, pp. 97–123. Ⓔ *Open Access*. doi: 10.1007/978-3-319-45853-3 6

[Ish08] ISHII H.: Tangible bits: Beyond pixels. In *Proc. TEI* (2008), ACM, New York, pp. xv–xxv. Ⓔ *Open Access*. doi: 10.1145/1347390.1347392

[Iss17] ISSARTEL P.: New approaches for the portability, non-intrusiveness and accessibility of 3D manipulation interfaces. PhD thesis, Université Paris-Saclay, France, Apr. 2017. Ⓔ *Open Access*.

[IU97] ISHII H., ULLMER B.: Tangible bits: Towards seamless interfaces between people, bits and atoms. In *Proc. CHI* (1997), ACM, New York, pp. 234–241. Ⓔ *Open Access*. doi: 10.1145/258549.258715

[JCK12a] JACKSON B., COFFEY D., KEEFE D. F.: Force Brushes: Progressive data-driven haptic selection and filtering for multi-variate flow visualizations. In *EuroVis Short Papers* (2012), The Eurographics Association, Goslar, Germany, pp. 7–11. Ⓔ *Open Access*. doi: 10.2312/PE/EuroVisShort/EuroVisShort2012/007-011

[JCK12b] JACKSON B., COFFEY D., KEEFE D. F.: Force brushes: Progressive data-driven haptic selection and filtering for multi-variate flow visualizations. In *EuroVis Short Papers* (2012), The Eurographics Association, Goslar, Germany. Ⓔ *Open Access*. doi: 10.2312/PE/EuroVisShort/EuroVisShort2012/007-011

[JDF12] JANSEN Y., DRAGICEVIC P., FEKETE J.-D.: Tangible remote controllers for wall-size displays. In *Proc. CHI* (2012), ACM, New York, pp. 2865–2874. Ⓔ *Open Access*. doi: 10.1145/2207676.2208691

[JFY16] JÖNSSON D., FALK M., YNNERMAN A.: Intuitive exploration of volumetric data using dynamic galleries. *IEEE Transactions on Visualization and Computer Graphics* **22**, 1 (Jan. 2016), 896–905. Ⓔ *Open Access*. doi: 10.1109/TVCG.2015.2467294

[JGAK07] JORDÀ S., GEIGER G., ALONSO M., KALTENBRUNNER M.: The reacTable: Exploring the synergy between live music performance and tabletop tangible interfaces. In *Proc. TEI* (2007), ACM, New York, pp. 139–146. Ⓔ *Open Access*. doi: 10.1145/1226969.1226998

[JH13] JANKOWSKI J., HACHET M.: A survey of interaction techniques for interactive 3D environments. In *Eurographics State of the Art Reports* (2013), Goslar, Germany, pp. 65–93. Ⓔ *Open Access*. doi: 10.2312/conf/EG2013/stars/065-093

[JH14] JANKOWSKI J., HACHET M.: Advances in interaction with 3D environments. *Computer Graphics Forum* **34**, 1 (Feb. 2014), 152–190. Ⓔ *Open Access*. doi: 10.1111/cgf.12466
[JJB18] JACKSON B., JELKE B., BROWN G.: Yea big, yea high: A 3D user interface for surface selection by progressive refinement in virtual environments. In *Proc. VR* (2018), IEEE Computer Society, Los Alamitos, pp. 320–326. Ⓘ *Open Access*. doi: 10.1109/VR.2018.8447559

[JLS*13] JACKSON B., LAU T. Y., SCHROEDER D., TOUSSAINT K. C., KEEFE D. F.: A lightweight tangible 3D interface for interactive visualization of thin fiber structures. *IEEE Transactions on Visualization and Computer Graphics* 19, 12 (Dec. 2013), 2802–2809. Ⓘ *Open Access*. doi: 10.1109/TVCG.2013.121

[Joh65] JOHNSON E. A.: Touch display—A novel input/output device for computers. *Electronics Letters* 1, 8 (Oct. 1965), 219–220. doi: 10.1049/el.19650200

[Joh67] JOHNSON E. A.: Touch displays: A programmed man-machine interface. *Ergonomics* 10, 2 (1967), 271–277. doi: 10.1080/00140136708930868

[Joh04] JOHNSON C. R.: Top scientific visualization research problems. *IEEE Computer Graphics and Applications* 24, 4 (July/Aug. 2004), 13–17. Ⓘ *Open Access*. doi: 10.1109/MCG.2004.20

[JOR*19] JOHNSON S., ORBAN D., RUNESHA H. B., MENG L., JUHNKE B., ERDMAN A., SAMSEL F., KEEFE D. F.: Bento Box: An interactive and zoomable small multiples technique for visualizing 4D simulation ensembles in virtual reality. *Frontiers in Robotics and AI* 6 (2019), 61. Ⓘ *Open Access*. doi: 10.3389/frobt.2019.00061

[JSK12] JACKSON B., SCHROEDER D., KEEFE D. F.: Nailing down multi-touch: Anchored above the surface interaction for 3D modeling and navigation. In *Proc. of Graphics Interface* (2012), Canadian Information Processing Society, Toronto, Ont., Canada, Canada, pp. 181–184. Ⓘ *Open Access*.

[JW14] JACOB M. G., WACHS J. P.: Context-based hand gesture recognition for the operating room. *Pattern Recognition Letters* 36 (2014), 196–203. doi: 10.1016/j.patrec.2013.05.024

[KAM*08] KEEFE D. F., ACEVEDO D., MILES J., DRURY F., SWARTZ S. M., LAIDLAW D. H.: Scientific sketching for collaborative VR visualization design. *IEEE Transactions on Visualization and Computer Graphics* 14, 4 (July/Aug. 2008), 835–847. Ⓘ *Open Access*. doi: 10.1109/TVCG.2008.31

[KEBK17] KREKHOV A., EMMERICH K., BABINSKI M., KRÜGER J.: Gestures from the point of view of an audience: Towards anticipatable interaction of presenters with 3D content. In *Proc. CHI* (2017), ACM, New York, pp. 5284–5294. doi: 10.1145/3025453.3025641

[Kee08] KEEFE D. F.: Free-form VR interactions in scientific visualization. In *Proc. IEEE Visualization Workshop on Scientific Workflow with Immersive Interfaces for Visualization* (2008). Ⓘ *Open Access*.

[Kee10] KEEFE D. F.: Integrating Visualization and Interaction Research to Improve Scientific Workflows. *IEEE Computer Graphics and Applications* 30, 2 (Mar./Apr. 2010), 8–13. doi: 10.1109/MCG.2010.30

[KGDR09] KÖNIG W. A., GERKEN J., DIERDORF S., REITERER H.: Adaptive pointing – Design and evaluation of a precision enhancing technique for absolute pointing devices. In *Proc. INTERACT* (2009), Springer, Berlin/Heidelberg, pp. 658–671. Ⓘ *Open Access*. doi: 10.1007/978-3-642-03655-2_73

[KGP*12] KLEIN T., GUÉNIAT F., PASTUR L., VERNIER F., ISENBERG T.: A design study of direct-touch interaction for exploratory 3D scientific visualization. *Computer Graphics Forum* 31, 3 (June 2012), 1225–1234. Ⓘ *Open Access*. doi: 10.1111/j.1467-8659.2012.03115.x

[KH11] KNOEDEL S., HACHET M.: Multi-touch RST in 2D and 3D spaces: Studying the impact of directness on user performance. In *Proc. 3DUI* (2011), IEEE Computer Society, Los Alamitos, pp. 75–78. Ⓘ *Open Access*. doi: 10.1109/3DUI.2011.5759220

[KHE*10] KLASHED S., HEMINGSSON P., EMMART C., COOPER M., YNNERMAN A.: Uniview – Visualizing the universe. In *Eurographics Short papers* (2010), Eurographics Association, Goslar, Germany. Ⓘ *Open Access*. doi: 10.2312/eqa.20101005

[KHPA12] KEY A., HOWE B., PERRY D., ARAGON C.: Vizdeck: Self-organizing dashboards for visual analytics. In *Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data* (2012), SIGMOD '12, Association for Computing Machinery, New York, NY, p. 681–684. Ⓘ *Open Access*. doi: 10.1145/2213836.2213931

[KI13] KEEFE D. F., ISENBERG T.: Reimagining the scientific visualization interaction paradigm. *IEEE Computer* 46, 5 (May 2013), 51–57. Ⓘ *Open Access*. doi: 10.1109/MC.2013.178

[KJK*15] KIM K., JACKSON B., KARAMOUZAS I., ADEAGBO M., GUY S. J., GRAFF R., KEEFE D. F.: Bema: A multimodal interface for expert experiential analysis of political assemblies at the Pnyx in ancient Greece. In *Proc. 3DUI* (2015), IEEE Computer Society, Los Alamitos, pp. 19–26. Ⓘ *Open Access*. doi: 10.1109/3DUI.2015.7131720

[KK17] KERRACHER N., KENNEDY J.: Constructing and evaluating visualisation task classifications: Process and considerations. *Computer Graphics Forum* 36, 3 (2017), 47–59. Ⓘ *Open Access*. doi: 10.1111/cgf.13167

[KKKF18] KULIK A., KUNERT A., KEIL M., FROEHLICH B.: RST 3D: A comprehensive gesture set for multitouch 3D navigation. In *Proc. VR* (2018), IEEE Computer Society, Los Alamitos, pp. 363–370. Ⓘ *Open Access*. doi: 10.1109/VR.2018.8447554

[KL09] KRUSZYŃSKI K. J., LIERE R.: Tangible props for scientific visualization: Concept, requirements, application. *Virtual Reality* 13, 4 (Dec. 2009), 235–244. doi: 10.1007/s10055-009-0126-1

[KM76] KAPLOW R., MOLNAR M.: A computer-terminal, hardware/software system with enhanced user input capabilities: The enhanced-input terminal system (EITS). *SIGGRAPH Computer Graphics* 10, 2 (July 1976), 116–124. doi: 10.1145/965143.563297

[KMB18] KHADKA R., MONEY J., BANIĆ A.: Body-prop interaction: Augmented open discs and egocentric body-based interaction for exploring immersive visualizations. In *Proc. ISS* (2018), ACM, New York, pp. 327–332. doi: 10.1145/3279778.3281458

[KNEK05] KUKIMOTO N., NONAKA J., EBARA Y., KOYAMADA K.: Scientific visualization in collaborative virtual environment with PDA-based control and 3D annotation. *JSME International Journal Series B Fluids and Thermal Engineering* 48, 2 (2005), 252–258. Ⓘ *Open Access*. doi: 10.1299/jsmeb.48.252

[Koe18] KOEMAN L.: How many participants do researchers recruit? A look at 678 UX/HCI studies. Online. Last visited 06 January 2019, 2018.

[KR09] KRATZ S., ROHS M.: Hoverflow: Expanding the design space of around-device interaction. In *Proc. MobileHCI* (2009), ACM, New York, pp. 4:1–4:8. Ⓘ *Open Access*. doi: 10.1145/1613858.1613864

[KRG*12] KRATZ S., ROHS M., GUSE D., MÜLLER J., BAILLY G., NISCHT M.: PalmSpace: Continuous around-device gestures vs. multitouch for 3D rotation tasks on mobile devices. In *Proc. AVI* (2012), ACM, New York, pp. 181–188. Ⓘ *Open Access*. doi: 10.1145/2254556.2254590

[KRW*11] KIRMIZIBAYRAK C., RADEVA N., WAKID M., PHILBECK J., SIBERT J., HAHN J.: Evaluation of gesture based interfaces for medical volume visualization tasks. In *Proc. VRCAI* (2011), ACM, New York, pp. 69–74. Ⓘ *Open Access*. doi: 10.1145/2087756.2087764

[KTKT15] KATZAKIS N., TEATHER R. J., KIYOKAWA K., TAKEMURA H.: INSPECT: Extending plane-casting for 6-DOF control. *Human-centric Computing and Information Sciences* 5, 1 (2015), 22. Ⓘ *Open Access*. doi: 10.1186/s13673-015-0037-y

[KTY97] KIYOKAWA K., TAKEMURA H., YOKOYA N.: Manipulation aid for two-handed 3-D designing within a shared virtual environment. In *Proc. HCI International* (1997), pp. 937–940. Ⓘ *Open Access*.

[KV19] KOUTSABASIS P., VOGIATZIDAKIS P.: Empirical research in mid-air interaction: A systematic review. *International Journal of Human-Computer Interaction* 35, 18 (2019), 1747–1768. doi: 10.1080/10447318.2019.1572352

[KVLP04] KONRAD-VERSE O., LITTMANN A., PREIM B.: Virtual resection with a deformable cutting plane. In *SimVis* (2004), pp. 203–214. Ⓘ *Open Access*.
[KZL07] KEEFE D., ZELEZNIK R., LAIDLAW D.: Drawing on air: Input techniques for controlled 3D line illustration. *IEEE Transactions on Visualization and Computer Graphics* **13**, 5 (Sept. 2007), 1067–1081. doi: 10.1109/TVCG.2007.1060

[KZL08] KEEFE D. F., ZELEZNIK R. C., LAIDLAW D. H.: Tech-note: Dynamic dragging for input of 3D trajectories. In *Proc. 3DUI* (2008), IEEE Computer Society, Los Alamitos, pp. 51–54. ⚫ Open Access. doi: 10.1109/3DUI.2008.4476591

[LAFT12] LIU J., AU O. K.-C., FU H., TAI C.-L.: Two-finger gestures for 6DOF manipulation of 3D objects. *Computer Graphics Forum* **31**, 7 (2012), 2047–2055. ⚫ Open Access. doi: 10.1111/j.1467-8659.2012.03197.x

[LB13] LAHA B., BOWMAN D. A.: Volume cracker: A bimanual 3D interaction technique for analysis of raw volumetric data. In *Proc. SUI* (2013), ACM, New York, pp. 61–68. doi: 10.1145/2491367.2491368

[LBLS14] LUBOS P., BEIMLER R., LAMMERS M., STEINICKE F.: Touching the cloud: Bimanual annotation of immersive point clouds. In *Proc. 3DUI* (2014), IEEE Computer Society, Los Alamitos, pp. 191–192. ⚫ Open Access. doi: 10.1109/3DUI.2014.6798885

[LBS14] LAHA B., BOWMAN D. A., SOCHA J. J.: Effects of vr system fidelity on analyzing isosurface visualization of volume datasets. *IEEE Transactions on Visualization and Computer Graphics* **20**, 4 (Apr. 2014), 513–522. ⚫ Open Access. doi: 10.1109/TVCG.2014.20

[LCP*07] LUNDIN K., COOPER M., PERSSON A., EVESTEDT D., YNNERMAN A.: Enabling design and interactive selection of haptic modes. *Virtual Reality* **11**, 1 (Mar. 2007), 1–13. ⚫ Open Access. doi: 10.1007/s10055-006-0033-7

[LFO*13] LEITHINGER D., FOLLMER S., OLWAL A., LUESCHER S., HOGGE A., LEE J., ISHII H.: Sublimate: state-changing virtual and physical rendering to augment interaction with shape displays. In *Proc. CHI* (2013), ACM, New York, pp. 1441–1450. ⚫ Open Access. doi: 10.1145/2470654.2466191

[LFO115] LEITHINGER D., FOLLMER S., OLWAL A., ISHII H.: Shape displays: Spatial interaction with dynamic physical form. *IEEE Computer Graphics and Applications* **35**, 5 (Sept. 2015), 5–11. ⚫ Open Access. doi: 10.1109/MCG.2015.111

[LG94] LIANG J., GREEN M.: JDCAD: A highly interactive 3D modeling system. *Computers & Graphics* **18**, 4 (July/Aug. 1994), 217–222. doi: 10.1016/0097-8493(94)90062-0

[LGK*13] LEE J., GU H., KIM H., KIM J., KIM H., KIM H.: Interactive manipulation of 3D objects using kinect for visualization tools in education. In *Proc. ICCAS* (2013), IEEE Computer Society, Los Alamitos, pp. 1220–1222. doi: 10.1109/ICCAS.2013.6704175

[LGY15] LIU P., GONG J., YU M.: Visualizing and analyzing dynamic meteorological data with virtual globes: A case study of tropical cyclones. *Environmental Modelling & Software* **64** (Feb. 2015), 80–93. doi: 10.1016/j.envsoft.2014.11.014

[LJZD10] LUCCHI A., JERMANN P., ZUFFEREY G., DILLENBOURG P.: An empirical evaluation of touch and tangible interfaces for tabletop displays. In *Proc. TEI* (2010), ACM, New York, pp. 177–184. ⚫ Open Access. doi: 10.1145/1709886.1709917

[LK11] LAVIOLA J. J., KEEFE D. F.: 3D spatial interaction: Applications for art, design, and science. In *ACM SIGGRAPH 2011 Courses* (2011), ACM, New York, pp. 1:1–1:75. doi: 10.1145/2037636.2037637

[LKG*16] LJUNG P., KRÜGER J., GROLLER E., HADWIGER M., HANSEN C. D., YNNERMAN A.: State of the art in transfer functions for direct volume rendering. *Computer Graphics Forum* **35**, 3 (June 2016), 669–691. doi: 10.1111/cgf.12934

[LKM*17] LAVIOLA JR. J. J., KRUIJFF E., MCMAHAN R. P., BOWMAN D. A., POUPYREV I.: 3D User Interfaces: Theory and Practice, 2nd ed. Addison-Wesley, Boston, 2017.

[LODI16] LÓPEZ D., OEHLBERG L., DOGER C., ISENBERG T.: Towards an understanding of mobile touch navigation in a stereoscopic viewing environment for 3D data exploration. *IEEE Transactions on Visualization and Computer Graphics* **22**, 5 (May 2016), 1616–1629. ⚫ Open Access. doi: 10.1109/TVCG.2015.2440233

[LOM*11] LEVESQUE V., ORAM L., MACLEAN K., COCKBURN A., MARCHUK N. D., JOHNSON D., COLGATE J. E., PESHKIN M. A.: Enhancing physicality in touch interaction with programmable friction. In *Proc. CHI* (2011), ACM, New York, pp. 2481–2490. ⚫ Open Access. doi: 10.1145/1978942.1979306

[LPCP*07] LUNDIN (PALMERIUS) K., COOPER M., PERSSON A., EVESTEDT D., YNNERMAN A.: Enabling design and interactive selection of haptic modes. *Virtual Reality* (2007). ⚫ Open Access. doi: 10.1007/s10055-006-0033-7

[LPGY05] LUNDIN (PALMERIUS) K., GUDMUNDSSON B., YNNERMAN A.: General proxy-based haptics for volume visualization. In *Eurohaptics Conference, 2005 and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems*, 2005. (2005), pp. 557–560. doi: 10.1109/WHC.2005.62

[LPLCY06] LUNDIN (PALMERIUS) K., LUNDSTRÖM C., COOPER M., YNNERMAN A.: Enabling haptic interaction with volumetric mri data through knowledge-based tissue separation. In *Proceedings of Volume Graphics* (2006), pp. 75–78. ⚫ Open Access. doi: 10.2312/NG/NG06/075-078

[LPSCY05] LUNDIN (PALMERIUS) K., SILLÉN M., COOPER M., YNNERMAN A.: Haptic visualization of computational fluid dynamics data using reactive forces. In *International Conference on Visualization and Data Analysis, San Jose, USA* (2005), pp. 95–102. ⚫ Open Access. doi: 10.1117/12.587029

[LPYG02] LUNDIN PALMERIUS K., YNNERMAN A., GUDMUNDSSON B.: Proxy-based haptic feedback from volumetric density data. In *Proceedings of the Eurohaptics Conference* (2002), pp. 104–109. ⚫ Open Access.

[LRF*11] LUNDSTRÖM C., RYDELL T., FORSELL C., PERSSON A., YNNERMAN A.: Multi-touch table system for medical visualization: Application to orthopedic surgery planning. *IEEE Transactions on Visualization and Computer Graphics* **17**, 12 (Dec. 2011). ⚫ Open Access. doi: 10.1109/TVCG.2011.224

[LSG*16] LEXOW G. J., SCHURZIG D., GELLRICH N.-C., LENARZ T., MAJDANI O., RAU T. S.: Visualization, measurement and modelling of the cochlea using rotating midmodiolar slice planes. *International Journal of Computer Assisted Radiology and Surgery* **11**, 10 (Oct. 2016), 1855–1869. doi: 10.1007/s11548-016-1374-7

[LTM18] LAM H., TORY M., MUNZNER T.: Bridging from goals to tasks with design study analysis reports. *IEEE Transactions on Visualization and Computer Graphics* **24**, 1 (Jan. 2018), 435–445. ⚫ Open Access. doi: 10.1109/TVCG.2017.2744319

[Lue13] LUESCHER S.: *Beyond visualization: designing interfaces to contextualize geospatial data*. PhD thesis, Massachusetts Institute of Technology, 2013. ⚫ Open Access.

[LWS*13] LIANG H.-N., WILLIAMS C., SEMEGEN M., STUER-ZLINGER W., IRANI P.: An investigation of suitable interactions for 3D manipulation of distant objects through a mobile device. *International Journal of Innovative Computing, Information and Control* **9**, 12 (Dec. 2013), 4737–4752. ⚫ Open Access.

[LYS20] LU Y., YU C., SHI Y.: Investigating bubble mechanism for ray-casting to improve 3D target acquisition in virtual reality. In *Proc. VR* (2020), IEEE Computer Society, Los Alamitos, pp. 35–43. doi: 10.1109/VR46266.2020.00021

[Man01] MANTERE M.: Visualization of flow data in photo-realistic virtual environment. PhD thesis, Helsinki University of Technology, Espoo, Finland, 2001. ⚫ Open Access.

[MBH14] MARZO A., BOSSAVIT B., HACHET M.: Combining multi-touch input and device movement for 3D manipulations in mobile augmented reality environments. In *Proc. SUI* (2014), ACM, New York, pp. 13–16. ⚫ Open Access. doi: 10.1145/2659766.2659775

[MBJS97] MINE M. R., BROOKS JR F. P., SEQUIN C. H.: Moving objects in space: Exploiting proprioception in virtual-environment interaction. In *Proc. SIGGRAPH* (1997), ACM, New York, pp. 19–26. ⚫ Open Access. doi: 10.1145/258734.258747
[MCB*18] MANDALIKA V. B. H., CHERNOGLAZOV A. I., BILLINGHURST M., BARTNECK C., HURRELL M. A., DE RUITER N., BUTLER A. P., BUTLER P. H.: A hybrid 2D/3D user interface for radiological diagnosis. *Journal of Digital Imaging* **31**, 1 (Feb. 2018), 56–73. Ⓐ Open Access. doi: 10.1007/s10278-017-0002-6

[MCG10] MARTINET A., CASIEZ G., GRISONI L.: The effect of DOF separation in 3D manipulation tasks with multi-touch displays. In *Proc. VRST* (2010), ACM, New York, pp. 111–118. Ⓐ Open Access. doi: 10.1145/1889863.1889888

[MCG*19] MENDES D., CAPUTO F. M., GIACHETTI A., FERREIRA A., JORGE J.: A survey on 3D virtual object manipulation: From the desktop to immersive virtual environments. *Computer Graphics Forum* **38**, 1 (2019), 21–45. doi: 10.1111/cgf.13390

[MCN94] MEYER S., COHEN O., NILSEN E.: Device comparisons for goal-directed drawing tasks. In *Proc. CHI* (1994), ACM, New York, pp. 251–252. doi: 10.1145/259963.260468

[MEV*06] MLEJNEK M., ERMES P., VILANOVA A., VAN DER RIJT R., VAN DEN BOSCH H., GERRITSEN F., GRÖLLER M. E.: Application-oriented extensions of profile flags. In *Proc. EuroVis* (2006), Eurographics Association, Goslar, Germany, pp. 339–346. Ⓐ Open Access.

[MFF03] MANSSOUR I. H., FURUIE S. S., FREITAS C. M. D. S.: Inner structures visualization in multimodal volume data. In *Proc. XXIII Congress da Sociedade Brasileira de Computação* (2003), vol. 3, pp. 131–138. Ⓐ Open Access.

[MFH*09] MARSHALL P., FLECK R., HARRIS A., RICK J., HORNECKER E., ROGERS Y., YUILL N., DALTON N. S.: Fighting for control: Children’s embodied interactions when using physical and digital representations. In *Proc. CHI* (2009), ACM, New York, pp. 2149–2152. Ⓐ Open Access. doi: 10.1145/1518701.1519027

[MFOF02] MANSSOUR I. H., FURUIE S. S., OLABARRIAGA S. D., FREITAS C. M. D. S.: Visualizing inner structures in multimodal volume data. In *Proc. XV Brazilian Symposium on Computer Graphics and Image Processing* (2002), pp. 51–58. Ⓐ Open Access. doi: 10.1109/SIBGRA.2002.1167123

[MHWH17] MEWES A., HENSEN B., WACKER F., HANSEN C.: Touchless interaction with software in interventional radiology and surgery: a systematic literature review. *International Journal of Computer Assisted Radiology and Surgery* **12**, 2 (Feb. 2017), 291–305. Ⓐ Open Access. doi: 10.1007/s11548-016-1480-6

[MJGJ11] MARQUARDT N., JOTA R., GREENBERG S., JORGE J. A.: The continuous interaction space: Interaction techniques unifying touch and gesture on and above a digital surface. In *Proc. INTERACT* (2011), Springer, Berlin, Heidelberg, pp. 461–476. Ⓐ Open Access.

[MRH08] MENSMANN J., ROPINSKI T., HINRICHS K.: Interactive cutting operations for generating anatomical illustrations from volumetric data sets. 37–44. Ⓐ Open Access.

[MS05] MALKAWI A. M., SRINIVASAN R. S.: A new paradigm for human-building interaction: The use of CFD and augmented reality. *Automation in Construction* **14**, 1 (Jan. 2005), 71–84. doi: 10.1016/j.autcon.2004.08.001

[MS16] MOLDOVAN C. C., STARETU I.: Motion leap compared to data gloves in human hand tracking. In *Advances in Robot Design and Intelligent Control* (2016), Springer International, Cham, Switzerland, pp. 195–202. doi: 10.1007/978-3-319-21290-6_20

[MSD*18] MARRIOTT K., SCHREIBER F., DWYER T., KLEIN K., RICHE N. H., ITOH T., STUERZLINGER W., THOMAS B. H.: *Immersive Analytics*, vol. 11190. Springer, Berlin, Heidelberg, Germany, 2018. doi: 10.1007/978-3-030-01388-2

[MSLF15] MA J., SINDORF L., LIAO I., FRAZIER J.: Using a tangible versus a multi-touch graphical user interface to support data exploration at a museum exhibit. In *Proc. TEI* (2015), ACM, New York, pp. 33–40. doi: 10.1145/2677199.2680555

[MSR*16] MEWES A., SAALFELD P., RIABIKIN O., SKALEJ M.,

HANSEN C.: A gesture-controlled projection display for ct-guided interventions. *International Journal of Computer Assisted Radiology and Surgery* **11**, 1 (Jan. 2016), 157–164. Ⓐ Open Access. doi: 10.1007/s11548-015-1215-0

[Mun14] MUNZNER T.: Visualization Analysis and Design. CRC Press, Boca Raton, FL, USA, 2014. doi: 10.1201/b17511

[MVN06] MALMBERG F., VIDHOLM E., NYSTRÖM I.: A 3D live-wire segmentation method for volume images using haptic interaction. In *Proc. DGCI* (2006), Springer, Berlin/Heidelberg, pp. 663–673. Ⓐ Open Access. doi: 10.1007/11907350_56

[NHH*16] NESTOROV N., HUGHES P., HEALY N., SHEEHY N., OHARE N.: Application of natural user interface devices for touch-free control of radiological images during surgery. In *Proc. CBMS* (2016), IEEE Computer Society, Los Alamitos, pp. 229–234. doi: 10.1109/CBMS.2016.20

[NHK*03] NISHIKAWA A., HOSOI T., KOARA K., NEGORO D., HIKITA A., ASANO S., KAKUTANI H., MIYAZAKI F., SEKIMOTO M., YASUI M., MIYAKE Y., TAKIGUCHI S., MONDEN M.: Face Mouse: A novel human-machine interface for controlling the position of a laparoscope. *IEEE Transactions on Robotics and Automation* **19**, 5 (Oct. 2003), 825–841. doi: 10.1109/TRA.2003.817093

[NMT*19] NAM J. W., MCCULLOUGH K., TVEITE J., ESPINOSA M. M., PERRY C. H., WILSON B. T., KEEFE D. F.: Worlds-in-Wedges: Combining worlds-in-miniature and portals to support comparative immersive visualization of forestry data. In *Proc. VR* (2019), IEEE Computer Society, Los Alamitos, pp. 747–755. doi: 10.1109/VR.2019.8797871

[NMVB09] NYSTRÖM I., MALMBERG F., VIDHOLM E., BENGTSSON E.: Segmentation and visualization of 3D medical images through haptic rendering. In *Proc. PRIP* (2009), Belarusian State University Publishing Center, Minsk, Belarus, pp. 43–48. Ⓐ Open Access.

[OAWH11] OLSON I. C., ATRASH LEONG Z., WILENSKY U., HORN M. S.: "It's just a toolbar!" Using tangibles to help children manage conflict around a multi-touch tabletop. In *Proc. TEI* (2011), ACM, New York, pp. 29–36. Ⓐ Open Access. doi: 10.1145/1935701.1935709

[OBF03] OLWAL A., BENKO H., FEINER S.: Senseshapes: Using statistical geometry for object selection in a multimodal augmented reality. In *Proc. ISMAR* (2003), IEEE Computer Society, Los Alamitos, pp. 300–301. Ⓐ Open Access. doi: 10.1109/ISMAR.2003.1240730

[OF03] OLWAL A., FEINER S.: The flexible pointer: An interaction technique for selection in augmented and virtual reality. In *ACM UIST Poster Compendium* (2003), pp. 81–82. Ⓐ Open Access.

[OGS*14] O'HARA K., GONZALEZ G., SELLEN A., PENNEY G., VARNAVAS A., MENTIS H., CRIMINISI A., CORISH R., ROUNCEFIELD M., DASTUR N., CARRELL T.: Touchless interaction in surgery. *Communications of the ACM* **57**, 1 (Jan. 2014), 70–77. Ⓐ Open Access. doi: 10.1145/2541883.2541899

[OJMN*19] OELTZE-JAFRA S., MEUSCHKE M., NEUGEBAUER M., SAALFELD S., LAWONN K., JANIGA G., HEGE H.-C., ZACHOW S., PREIM B.: Generation and visual exploration of medical flow data: Survey, research trends and future challenges. *Computer Graphics Forum* **38**, 1 (2019), 87–125. Ⓐ Open Access. doi: 10.1111/cgf.13394

[OKKT12] OHNISHI T., KATZAKIS N., KIYOKAWA K., TAKEMURA H.: Virtual interaction surface: Decoupling of interaction and view dimensions for flexible indirect 3D interaction. In *Proc. 3DUI* (2012), IEEE Computer Society, Los Alamitos, pp. 113–116. doi: 10.1109/3DUI.2012.6184194

[OLC*04] OLOFSSON I., LUNDIN K., COOPER M., KJÄLL P., YNNERMAN A.: A haptic interface for dose planning in stereo-tactic radiosurgery. In *Proc. MediViz* (2004), pp. 200–205. Ⓐ Open Access.

[ONI05] OWADA S., NIELSEN F., IGARASHI T.: Volume catcher. In *Proc. 13D* (2005), ACM, New York, pp. 111–116. Ⓐ Open Access. doi: 10.1145/1053427.1053445

[Osa08] OSAWA N.: Two-handed and one-handed techniques for precise and efficient manipulation in immersive virtual environments. In *Proc. ISVC* (2008), Springer, Berlin, Heidelberg, pp. 987–997. doi: 10.1007/978-3-540-89639-5_94
[PBD*16] PLIMMER B., BROWN B., DIPROSE J., DU PREEZ S., LUXTON-REILLY A.: Tabletop 3D object manipulation with touch and tangibles. In *Collaboration Meets Interactive Spaces*. Springer International, Cham, Switzerland, 2016, pp. 11–32. doi: 10.1007/978-3-319-45853-5_2

[PBR07] PANCHAPHONGSAPHAK B., BURGKART R., RIENER R.: Three-dimensional touch interface for medical education. *IEEE Transactions on Information Technology in Biomedicine* **11**, 3 (May 2007), 251–263. doi: 10.1109/TITB.2006.884359

[PCE*17] PALOMAR R., CHEIKH F. A., EDWIN B., FRETLAND Å., BEGHADI A., ELLE O. J.: A novel method for planning liver resections using deformable Bézier surfaces and distance maps. *Computer Methods and Programs in Biomedicine* **144** (2017), 135–145. ⚙️ **Open Access**. doi: 10.1016/j.cmpb.2017.03.019

[PCR*19] PROUZEAU A., CORDEIL M., ROBIN C., ENS B., THOMAS B. H., DWYER T.: Scáptons and highlight-planes: Immersive interaction techniques for finding occluded features in 3D scatterplots. In *Proc. CHI* (2019), ACM, New York, pp. 325:1–325:12. ⚙️ **Open Access**. doi: 10.1145/3290605.3300555

[PCT*07] PERSSON P. B., COOPER M. D., TIBELL L. A. E., AINSWORTH S., YNNERMAN A., JONSSON B.: Designing and evaluating a haptic system for biomolecular education. In *Proc. VR* (2007), IEEE Computer Society, Los Alamitos, pp. 171–178. ⚙️ **Open Access**. doi: 10.1109/VR.2007.352478

[PDF14] PERIN C., DRAGICEVIC P., FEKETE J.-D.: Revisiting Bertin matrices: New interactions for crafting tabular visualizations. *IEEE Transactions on Visualization and Computer Graphics* (Nov. 2014), 2082–2091. ⚙️ **Open Access**. doi: 10.1109/TVCG.2014.2346279

[PHTP*10] PICK S., HENTSCHEL B., TEDJO-PALCZYNSKI I., WOLTER M., KUHLEN T.: Automated positioning of annotations in immersive virtual environments. In *Proc. JVRC* (2010), Eurographics Association, Goslar, Germany, pp. 1–8. ⚙️ **Open Access**. doi: 10.2312/EGVE/JVRC10/001-008

[PM03] POUPYREV I., MARUYAMA S.: Tactile interfaces for small touch screens. In *Proc. UIST* (2003), ACM, New York, pp. 217–220. ⚙️ **Open Access**. doi: 10.1145/964696.964721

[PMIoT76] PERLMAN R., MASSACHUSETTS INST. OF TECH. C. A. I. L.: *Using Computer Technology to Provide a Creative Learning Environment for Preschool Children*. AI Memo 360 [*microform*] / Radia Perlman. Distributed by ERIC Clearinghouse, Washington, D.C., 1976. ⚙️ **Open Access**.

[PMMG17] PFEUFFER K., MAYER B., MARDANBEGI D., GELLERSEN H.: Gaze + pinch interaction in virtual reality. In *Proc. SUI* (2017), ACM, New York, pp. 99–108. ⚙️ **Open Access**. doi: 10.1145/3131277.3132180

[POR*18] PAHUD M., OFEK E., RICHE N. H., HURTER C., GRUBERT J.: Mobiles as portals for interacting with virtual data visualizations. In *Proc. Data Visualization on Mobile Devices workshop at CHI* (2018). ⚙️ **Open Access**.

[PPMW06] PETRIDIS P., PLETINCKX D., MANIA K., WHITE M.: The EPOCH multimodal interface for interacting with digital heritage artefacts. In *Proc. Interactive Technologies and Sociotechnical Systems* (2006), Springer, Berlin, Heidelberg, pp. 408–417. doi: 10.1007/11890881_45

[PR10] PANEELS S., ROBERTS J. C.: Review of designs for haptic data visualization. *IEEE Transactions on Haptics* **3**, 2 (Apr. 2010), 119–137. ⚙️ **Open Access**. doi: 10.1109/TOH.2009.44

[PRI02] PIPER B., RATTI C., ISHII H.: Illuminating clay: A 3-D tangible interface for landscape analysis. In *Proc. CHI* (2002), ACM, New York, pp. 355–362. ⚙️ **Open Access**. doi: 10.1145/503376.503439

[PSOP01] PREIM B., SPINDLER W., OLDHAFER K. J., PEITGEN H.-O.: 3D-interaction techniques for planning of oncologic soft tissue operations. In *Proc. Graphics Interface* (2001), CIPS, Canada, pp. 183–190. ⚙️ **Open Access**. doi: 10.20380/GI2001.22

[PSZ*12] PRIETO P. A., SOTO F. D., ZÚNIGA M. D., QIN S.-F., WRIGHT D. K.: Three-dimensional immersive mixed-reality interface for structural design. *Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture* **226**, 5 (2012), 955–958. doi: 10.1177/0954405411432392

[PTH98] PFLESSER B., TIEDE U., HÖHNE K. H.: Specification, modelling and visualization of arbitrarily shaped cut surfaces in the volume model. In *Proc. MICCAI* (1998), Springer, Berlin, Heidelberg, pp. 853–860. ⚙️ **Open Access**. doi: 10.1007/Bf0056273

[PTSP02] PREIM B., TIETJEN C., SPINDLER W., PEITGEN H.-O.: Integration of measurement tools in medical 3D visualizations. In *Proc. IEEE Visualization* (2002), IEEE Computer Society, Los Alamitos, pp. 21–28. ⚙️ **Open Access**. doi: 10.1109/VISUAL.2002.1183752

[PTW98] POUPYREV I., TOMOKAZU N., WEGHORST S.: Virtual notepad: Handwriting in immersive VR. In *Proc. VRAIS* (1998), IEEE Computer Society, Los Alamitos, pp. 126–132. ⚙️ **Open Access**. doi: 10.1109/VRAIS.1998.658467

[PvW94] POST F. H., VAN WIJK J. J.: Visual representation of vector fields. In *Scientific Visualization: Advances and Challenges*, Rosenblum L. J., (Ed.). Academic Press, 1994, ch. 23, pp. 367–390. ⚙️ **Open Access**.

[PWHK16] PICK S., WEYERS B., HENTSCHEL B., KUHLEN T. W.: Design and evaluation of data annotation workflows for CAVE-like virtual environments. *IEEE Transactions on Visualization and Computer Graphics* **22**, 4 (Apr. 2016), 1452–1461. ⚙️ **Open Access**. doi: 10.1109/TVCG.2016.2518086

[RAW*15] RIND A., AIGNER W., WAGNER M., MIKSCH S., LAMMARSCH T.: Task cube: A three-dimensional conceptual space of user tasks in visualization design and evaluation. *Information Visualization* **15**, 4 (Dec. 2015), 288–300. doi: 10.1177/1473871615621602

[RBP*19] RO H., BYUN J.-H., PARK Y. J., LEE N. K., HAN T.-D.: AR pointer: Advanced ray-casting interface using laser pointer metaphor for object manipulation in 3D augmented reality environment. *Applied Sciences* **9**, 15 (July 2019), 3078:1–3078:18. ⚙️ **Open Access**. doi: 10.3390/app9153078

[RCDD13] REN L., CUI J., DU Y., DAI G.: Multilevel interaction model for hierarchical tasks in information visualization. In *Proc. VINCI* (2013), ACM, New York, pp. 11–16. ⚙️ **Open Access**. doi: 10.1145/2493102.2493104

[RCR08] REUTER P., COUTURE N., RIVIÈRE G.: Démonstration: ArcheoTUI. In *Proc. IHM* (2008), ACM, New York, pp. 221–222. ⚙️ **Open Access**. doi: 10.1145/1512714.1512761

[RDH09] REISMAN J. L., DAVIDSON P. L., HAN J. Y.: A screen-space formulation for 2D and 3D direct manipulation. In *Proc. UIST* (2009), ACM, New York, pp. 69–78. ⚙️ **Open Access**. doi: 10.1145/1622176.1622190

[REM11] RÖTTGER D., ENGELHARDT S., MÜLLER S.: Multimodal visualizations for pre-operative neurosurgical planning. In *Proc. INFORMATIK* (2011), Springer, Berlin, Heidelberg, p. 438. ⚙️ **Open Access**.

[RGB*10] RAYNAL M., GAUFFRE G., BACH C., SCHMITT B., DUBOIS E.: Tactile camera vs. tangible camera: Taking advantage of small physical artefacts to navigate into large data collection. In *Proc. NordiCHI* (2010), ACM, New York, pp. 373–382. ⚙️ **Open Access**. doi: 10.1145/1868914.1868958

[Rhe02] RHEINGANS P.: Are we there yet? Exploring with dynamic visualization. *IEEE Computer Graphics and Applications* **22**, 1 (Jan. 2002), 6–10. ⚙️ **Open Access**. doi: 10.1109/38.974511

[RRA*12] RUPPERT G. C. S., REIS L. O., AMORIM P. H. J., DE MORAES T. F., DA SILVA J. V. L.: Touchless gesture user interface for interactive image visualization in urological surgery. *World Journal of Urology* **30**, 5 (Oct. 2012), 687–691. ⚙️ **Open Access**. doi: 10.1007/s00345-012-0879-0

[RRRP08] RIEDER C., RITTER F., RASPE M., PEITGEN H.-O.: Interactive visualization of multimodal volume data for neurosurgical tumor treatment. *Computer Graphics Forum* **27**, 3 (2008), 1055–1062. ⚙️ **Open Access**. doi: 10.1111/j.1467-8659.2008.01242.x

[RS13] ROTH S., SCHULZ C.: A pattern-based approach to presentations using slide facilities. In *Proc. EuroPLoP* (2013), ACM, New York, pp. 15:1–15:12. doi: 10.1145/2739011.2739026

[RSBB06] REITINGER B., SCHMALSTIEG D., BORNIK A., BEICHEL R.: Spatial analysis tools for virtual reality-based surgical planning. In *Proc. 3DUI* (2006), IEEE Computer Society, Los Alamitos, pp. 37–44. ⚙️ **Open Access**. doi: 10.1109/VR.2006.121
[RvKC*11] RICK T., VON KAPRI A., CASPERS S., AMUNTS K., ZILLES K., KUHLEN T.: Visualization of probabilistic fiber tracts in virtual reality. In *Medicine Meets Virtual Reality 18* (2011), IOS Press, Amsterdam, pp. 486–492. doi: 10.3233/978-1-60750-706-2-486

[RWP*04] RATTI C., WANG Y., PIPER B., ISHII H., BIDERMAN A.: Phoxel-space: An interface for exploring volumetric data with physical voxels. In *Proc. DIS* (2004), ACM, New York, pp. 289–296. doi: 10.1145/1013115.1013156

[SAIB16] SERENO M., AMMI M., ISENBERG T., BESANÇON L.: Tangible brush: Performing 3D selection with portable and position-aware devices. In *IEEE VIS Poster Compendium* (2016). doi: 10.1145/3299896.3299900

[SAL06] SUBRAMANIAN S., ALIAKSEYEU D., LUCERO A.: Multi-layer interaction for digital tables. In *Proc. UIST* (2006), ACM, New York, pp. 269–272. doi: 10.1145/1166253.1166295

[SBB*06] SCHMIDT G., BAILLOT Y., BROWN D. G., TOMLIN E. B., SWAN J.: Toward disambiguating multiple selections for frustum-based pointing. In *Proc. 3DUI* (2006), IEEE Computer Society, Los Alamitos, pp. 87–94. doi: 10.1109/VR.2006.133

[SBG09] SCHMIDT D., BLOCK F., GELLERSEN H.: A comparison of direct and indirect multi-touch input for large surfaces. In *Proc. INTERACT* (2009), Springer, Berlin, Heidelberg, pp. 582–594. doi: 10.1007/978-3-642-03655-2_65

[SBJ*14] SUNDÉN E., BOCK A., JÖNSSON D., YNNERMAN A., ROPINSKI T.: Interaction techniques as a communication channel when presenting 3D visualizations. In *Proc. 3DVis* (2014), IEEE Computer Society, Los Alamitos, pp. 61–65. doi: 10.1109/3DVis.2014.7160102

[SBM92] SPRINGMEYER R. R., BLATTNER M. M., MAX N. L.: A characterization of the scientific data analysis process. In *Proc. Visualization* (1992), IEEE Computer Society, Los Alamitos, pp. 235–242. doi: 10.1109/MISUAL.1992.235203

[SBPM98] SADARJOEN A., BOER A. J. D., POST F. H., MYNETT A. E.: Particle tracing in $\Sigma$-transformed grids using tetrahedral 6-decomposition. In *Proc. VisSym* (1998), Eurographics Association, Goslar, Germany, pp. 71–80. doi: 10.2312/vissym19981006

[Sch07] SCHAEFFER S.: An augmented haptic interface as applied to flow visualization. Master's thesis, University of New Hampshire, Durham, 2007. doi: 10.1145/2299806.2299810

[SFL*04] SOBEL J. S., FORSBERG A. S., LAIDLAW D. H., ZELEZNIK R. C., KEEFE D. F., PIVKIN I., KARNIADAKIS G. E., RICHARDSON P., SWARTZ S.: Particle flurries. *IEEE Computer Graphics and Applications* 24, 2 (Mar. 2004), 76–85. doi: 10.1109/MCG.2004.1274065

[SG15] SIMEONE A. L., GELLERSENY H.: Comparing indirect and direct touch in a stereoscopic interaction task. In *Proc. 3DUI* (2015), IEEE Computer Society, Los Alamitos, pp. 105–108. doi: 10.1109/3DUI.2015.7131733

[SGF*11] SONG P., GOH W. B., FU C.-W., MENG Q., HENG P.-A.: WYSIWYF: Exploring and annotating volume data with a tangible handheld device. In *Proc. CHI* (2011), ACM, New York, pp. 1333–1342. doi: 10.1145/1978942.1979140

[SGH03] SAROUL L., GERLACH S., HERCH R. D.: Exploring curved anatomic structures with surface sections. In *Proc. Visualization* (2003), IEEE Computer Society, Los Alamitos, pp. 27–34. doi: 10.1109/MISUAL.2003.1250351

[SGH*12] SONG P., GOH W. B., HUTAMA W., FU C.-W., LIU X.: A handle bar metaphor for virtual object manipulation with mid-air interaction. In *Proc. CHI* (2012), ACM, New York, pp. 1297–1306. doi: 10.1145/2207676.2208585

[SHLPF14] SCHÖNBORN K., HÖST G., LUNDIN PALMERIUS K., FLINT J.: Development of an interactive immersion environment for engineering understanding about nanotechnology: Concept, construction, and implementation. *International Journal of Virtual and Personal Learning Environments 5* (Apr. 2014), 40–56. doi: 10.4018/ijvple.2014040104

[Shn83] SHNEIDERMAN B.: Direct manipulation: A step beyond programming languages. *Computer 16*, 8 (Aug. 1983), 57–69. doi: 10.1109/MC.1983.1654471

[Shn96] SHNEIDERMAN B.: The eyes have it: A task by data type taxonomy for information visualizations. In *Proc. VL* (1996), IEEE Computer Society, Los Alamitos, pp. 336–343. doi: 10.1109/VL.1996.545307

[SHSK08] STEINICKE F., HINRICHS K., SCHÖNING J., KRÜGER A.: Multi-touching 3D data: Towards direct interaction in stereoscopic display environments coupled with mobile devices. In *Proc. AVI Workshop on Designing Multi-Touch Interaction Techniques for Coupled Public and Private Displays* (2008), pp. 46–49. doi: 10.1109/TVCG.2008.13

[SIS04] SCHKOLNE S., ISHII H., SCHRODER P.: Immersive design of DNA molecules with a tangible interface. In *Proc. Visualization* (2004), IEEE Computer Society, Los Alamitos, pp. 227–234. doi: 10.1109/MSUAL.2004.47

[SKR*15] SHARP T., KESKIN C., ROBERTSON D., TAYLOR J., SHOTTON J., KIM D., RHEMANN C., LEICHTER I., VINNIKOV A., WEI Y., FREEDMAN D., KOHLI P., KRUPKA E., FITZGIBBON A., IZADI S.: Accurate, robust, and flexible real-time hand tracking. In *Proc. CHI* (2015), ACM, New York, pp. 3633–3642. doi: 10.1145/2702123.2702179

[SLC*15] SHEN E., LI S., CAI X., ZENG L., WANG W.: SAVE: Saliency-assisted volume exploration. *Journal of Visualization 18*, 2 (May 2015), 369–379. doi: 10.1007/s12650-014-0237-y

[SLM*03] STARNER T., LEIBE B., MINNEN D., WESTYN T., HURST A., WEEKS J.: The perceptive workbench: Computer-vision-based gesture tracking, object tracking, and 3D reconstruction for augmented desks. *Machine Vision and Applications 14*, 1 (Apr. 2003), 59–71. doi: 10.1007/s00138-002-0096-8

[SLY17] SUNDÉN E., LUNDGREN I., YNNERMAN A.: Hybrid virtual reality touch table – An immersive collaborative platform for public explanatory use of cultural objects and sites. In *Proc. Eurographics Workshop on Graphics and Cultural Heritage* (2017), Eurographics Association, Goslar, Germany, pp. 109–113. doi: 10.2312/gch.20171300

[SMP*17] SOUSA M., MENDES D., PAULO S., MATELA N., JORGE J., LOPES D. S. O.: VRRRRoom: Virtual reality for radiologists in the reading room. In *Proc. CHI* (2017), ACM, New York, pp. 4057–4062. doi: 10.1145/3025453.3025566

[SNHS13] SCHULZ H., NOCKE T., HEITZLER M., SCHUMANN H.: A design space of visualization tasks. *IEEE Transactions on Visualization and Computer Graphics* 19, 12 (Dec. 2013), 2366–2375. doi: 10.1109/TVCG.2013.120

[SP04] STEED A., PARKER C.: 3D selection strategies for head tracked and non-head tracked operation of spatially immersive displays. In *Proc. Intern. Immersive Projection Technology Workshop* (2004), pp. 163–170. doi: 10.1145/1008106.1008107

[SS91] SEARS A., SHNEIDERMAN B.: High precision touchscreens: Design strategies and comparisons with a mouse. *International Journal of Man-Machine Studies* 34, 4 (1991), 593–613. doi: 10.1016/0020-7373(91)90037-8

[SS99] SCHMALSTIEG D., SCHAUFLER G.: Sewing worlds together with seams: A mechanism to construct complex virtual environments. *Presence* 8, 4 (Aug. 1999), 449–461. doi: 10.1162/105474699566332

[SS18] SRINIVASAN A., STASKO J.: Orko: Facilitating multimodal interaction for visual exploration and analysis of networks. *IEEE Transactions on Visualization and Computer Graphics* 24, 1 (Jan. 2018), 511–521. doi: 10.1109/TVCG.2017.2745219

[SSD09] SPINDLER M., STELLMACH S., DACHSELT R.: PaperLens: Advanced magic lens interaction above the tabletop. In *Proc. ITS* (2009), ACM, New York, pp. 69–76. doi: 10.1145/1731903.1731920

[SSSS11] SULTANUM N., SOMANATH S., SHARLIN E., SOUSA M. C.: “Point it, split it, peel it, view it”: Techniques for interactive reservoir
visualization on tabletops. In Proc. ITS (2011), ACM, New York, pp. 192–201. ⌜ Open Access . doi: 10.1145/2076354.2076390

[Ste00] STEINMAN D. A.: Simulated pathline visualization of computed periodic blood flow patterns. Journal of Biomechanics 33, 5 (2000), 623–628. doi: 10.1016/S0021-9290(99)00205-5

[Ste06] STEED A.: Towards a general model for selection in virtual environments. In Proc. 3DUI (2006), IEEE Computer Society, Los Alamitos, pp. 103–110. doi: 10.1109/VR.2006.134

[Sut64] SUTHERLAND I. E.: Sketch Pad: A man-machine graphical communication system. In Proc. of the SHARE Design Automation Workshop (1964), ACM, New York, pp. 6.329–6.346. ⌜ Open Access . doi: 10.1145/800265.810742

[Sut65] SUTHERLAND E. I.: The ultimate display. In Proceedings of IFIP Congress (1965), Macmillan and Co, London, pp. 506–508.

[Sut66] SUTHERLAND I. E.: Ten unsolved problems in computer graphics. Datamation 12, 5 (1966), 22. ⌜ Open Access .

[SVBCS13] SULTANUM N., VITAL BRAZIL E., COSTA SOUSA M.: Navigating and annotating 3D geological outcrops through multi-touch interaction. In Proc. ITS (2013), ACM, New York, pp. 345–348. ⌜ Open Access . doi: 10.1145/2512349.2512396

[SvZP*16] SOLLICH H., VON ZADOW U., PIETZSCH T., TOMANCAP P., DACHSELT R.: Exploring time-dependent scientific data using spatially aware mobiles and large displays. In Proc. ISS (2016), ACM, New York, pp. 349–354. ⌜ Open Access . doi: 10.1145/2992154.2996779

[SXL*14] SHAN G., XIE M., LI F., GAO Y., CHI X.: Interactive visual exploration of halos in large-scale cosmology simulation. Journal of Visualization 17, 3 (Aug. 2014), 145–156. ⌜ Open Access . doi: 10.1007/s12650-014-0206-5

[SYG*16] SONG P., YAN X., GOH W. B., CHEN A. Q., FU C.-W.: Hand-posture-augmented multitouch interactions for exploratory visualization. In SIGGRAPH ASIA Technical Briefs (2016), ACM, New York, pp. 27:1–27:4. ⌜ Open Access . doi: 10.1145/3005358.3005363

[TEC*16] TONG X., EDWARDS J., CHEN C., SHEN H., JOHNSON C. R., WONG P. C.: View-dependent streamline deformation and exploration. IEEE Transactions on Visualization and Computer Graphics 22, 7 (July 2016), 1788–1801. ⌜ Open Access . doi: 10.1109/TVCG.2015.2502583

[TFK*02] TSANG M., FITZMAURICE G. W., KURTENBACH G., KHAN A., BUXTON B.: Boom chameleon: Simultaneous capture of 3D viewpoint, voice and gesture annotations on a spatially-aware display. In Proc. UIST (2002), ACM, New York, pp. 111–120. ⌜ Open Access . doi: 10.1145/571985.572001

[TGM*17] TOMINSKI C., GLADISCH S., KISTER U., DACHSELT R., SCHUMANN H.: Interactive lenses for visualization: An extended survey. Computer Graphics Forum 36, 6 (2017), 173–200. ⌜ Open Access . doi: 10.1111/cgf.12871

[THA10] TOP A., HAMARNEH G., ABUGHARBIEH R.: Spotlight: Automated confidence-based user guidance for increasing efficiency in interactive 3D image segmentation. In Proc. MCV (2010), Springer, Berlin/Heidelberg, pp. 204–213. ⌜ Open Access . doi: 10.1007/978-3-642-18421-5_20

[TJ00] TANRIVERDI V., JACOB R. J.: Interacting with eye movements in virtual environments. In Proc. CHI (2000), ACM, New York, pp. 265–272. doi: 10.1145/332040.332443

[TKI10] TUDDENHAM P., KIRK D., IZADI S.: Graspables revisited: Multi-touch vs. tangible input for tabletop displays in acquisition and manipulation tasks. In Proc. CHI (2010), ACM, New York, pp. 2223–2232. ⌜ Open Access . doi: 10.1145/1753326.1753662

[TKSI07] TERRENghi L., KIRK D., SELLEN A., IZADI S.: Affordances for manipulation of physical versus digital media on interactive surfaces. In Proc. CHI (2007), ACM, New York, pp. 1157–1166. ⌜ Open Access . doi: 10.1145/1240624.1240799

[TLN17] THEART R. P., LOOS B., NIESLER T. R.: Virtual reality assisted microscopy data visualization and colocalization analysis. BMC Bioinformatics 18, 2 (2017), 64:1–64:16. ⌜ Open Access . doi: 10.1186/s12859-016-1446-2

[TLS17] TONG X., LI C., SHEN H.-W.: Glyphslens: View-dependent occlusion management in the interactive glyph visualization. IEEE Transactions on Visualization and Computer Graphics 23, 1 (Jan 2017), 891–900. doi: 10.1109/TVCG.2016.2599049

[TM04] TORY M., MÖLLER T.: Human factors in visualization research. IEEE Transactions on Visualization and Computer Graphics 10, 1 (2004), 72–84. ⌜ Open Access . doi: 10.1109/TVCG.2004.1260759

[TO10] TAWARA T., ONO K.: A framework for volume segmentation and visualization using augmented reality. In Proc. 3DUI (2010), IEEE Computer Society, Los Alamitos, pp. 121–122. ⌜ Open Access . doi: 10.1109/3DUI.2010.5444707

[TRC*93] TAYLOR R., ROBINETT W., CHI V., BROOKS F., WRIGHT W., WILLIAMS R., SNYDER E.: The nanomanipulator: A virtual-reality interface for a scanning tunneling microscope. In Proc. SIGGRAPH (1993), ACM, New York, pp. 127–134. doi: 10.1145/166117.166133

[TTSI*12] TUNTAKURN A., THONGVIGITMANEE S. S., SA-ING V., MAKHANOV S. S., HASEGAWA S.: Natural interaction on 3D medical image viewer software. In The 5th 2012 Biomedical Engineering International Conference (2012), pp. 1–5. ⌜ Open Access . doi: 10.1109/BMEICon.2012.6465424

[UK12] ULLRICH S., KUHLEN T.: Haptic palpation for medical simulation in virtual environments. IEEE Transactions on Visualization and Computer Graphics 18, 4 (Apr. 2012), 617–625. doi: 10.1109/TVCG.2012.46

[VC14] VEIT M., CAPOBIANCO A.: Go’Then’Tag: A 3-D point cloud annotation technique. In Proc. 3DUI (2014), IEEE Computer Society, Los Alamitos, pp. 193–194. ⌜ Open Access . doi: 10.1109/3DUI.2014.6798886

[VCB09] VEIT M., CAPOBIANCO A., BECHMANN D.: Influence of degrees of freedom’s manipulation on performances during orientation tasks in virtual reality environments. In Proc. VRST (2009), ACM, New York, pp. 51–58. ⌜ Open Access . doi: 10.1145/1643928.1643942

[VGC07] VANACKEN L., GROSSMAN T., CONINX K.: Exploring the effects of environment density and target visibility on object selection in 3D virtual environments. In Proc. 3DUI (2007), IEEE Computer Society, Los Alamitos. ⌜ Open Access . doi: 10.1109/3DUI.2007.340783

[vPOBB*11] VAN PELT R., OLIVAN BESCOS J., BREEUWER M., CLOUGH R. E., GRÖLLER M. E., TER HAAR ROMENIJ B., VILANOVA A.: Interactive virtual probing of 4D MRI blood-flow. IEEE Transactions on Visualization and Computer Graphics 17, 12 (Dec. 2011), 2153–2162. ⌜ Open Access . doi: 10.1109/TVCG.2011.215

[vRBKB03] VAN REIMERSDAHL T., BLEY F., KUHLEN T., BISCHOF C. H.: Haptic rendering techniques for the interactive exploration of CFD datasets in virtual environments. In Proc. Workshop on Virtual Environments (2003), ACM, New York, pp. 241–246. ⌜ Open Access . doi: 10.1145/769953.769981

[VSB*10] VALKOV D., STEINICKE F., BRUDER G., HINRICHS K. H., SCHÖNING J., DAIBER F., KRÜGER A.: Touching floating objects in projection-based virtual reality environments. In Proc. EGVE/EuroVR/VEC (2010), Eurographics Association, Goslar, Germany, pp. 17–24. ⌜ Open Access . doi: 10.2312/EGVE/JVRC10/017-024

[VSBH11] VALKOV D., STEINICKE F., BRUDER G., HINRICHS K.: 2D touching of 3D stereoscopic objects. In Proc. CHI (2011), ACM, New York, pp. 1353–1362. ⌜ Open Access . doi: 10.1145/1978942.1979142

[Wan10] WANG S.: Comparing Tangible and Multi-Touch Interfaces for a Spatial Problem Solving Task. Master’s thesis, Simon Fraser University, Canada, 2010. ⌜ Open Access .

[WB03] WU M., BALAKRISHNAN R.: Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays. In Proc. UIST (2003), ACM, New York, pp. 193–202. ⌜ Open Access . doi: 10.1145/964696.964718

[WBAI17] WANG X., BESANÇON L., AMMI M., ISENBERG T.: Augmenting tactile 3D data exploration with pressure sensing. In IEEE VIS Posters (2017). ⌜ Open Access .

[WBAI19] WANG X., BESANÇON L., AMMI M., ISENBERG T.: Augmenting tactile 3D data navigation with pressure sensing. Computer