# A Design Study of Direct-Touch Interaction for Exploratory 3D Scientific Visualization

Tijmen Klein, Florimond Guéniat, Luc R. Pastur, Frédéric Vernier, Tobias Isenberg

► To cite this version:

Tijmen Klein, Florimond Guéniat, Luc R. Pastur, Frédéric Vernier, Tobias Isenberg. A Design Study of Direct-Touch Interaction for Exploratory 3D Scientific Visualization. Computer Graphics Forum, 2012, 31 (3), pp.1225-1234. 10.1111/j.1467-8659.2012.03115.x. hal-00781293

HAL Id: hal-00781293

https://inria.hal.science/hal-00781293

Submitted on 25 Jan 2013

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
# A Design Study of Direct-Touch Interaction for Exploratory 3D Scientific Visualization

Tijmen Klein<fnref n="1" />,<fnref n="3" />

Florimond Guéniat<fnref n="3" />,<fnref n="5" />

Luc Pastur<fnref n="3" />,<fnref n="5" />

Frédéric Vernier<fnref n="3" />,<fnref n="5" />

Tobias Isenberg<fnref n="1" />,<fnref n="2" />,<fnref n="3" />,<fnref n="4" />

## Abstract

We present an interaction design study of several non-overlapping direct-touch interaction widgets, postures, and bi-manual techniques to support the needs of scientists who are exploring a dataset. The final interaction design supports navigation/zoom, cutting plane interaction, a drilling exploration, the placement of seed particles in 3D space, and the exploration of temporal data evolution. To ground our design, we conducted a requirements analysis and used a participatory design approach throughout development. We chose simulations in the field of fluid mechanics as our example domain and, in the paper, discuss our choice of techniques, their adaptation to our target domain, and discuss how they facilitate the necessary combination of visualization control and data exploration. We evaluated our resulting interactive data exploration system with seven fluid mechanics experts and report on their qualitative feedback. While we use flow visualization as our application domain, the developed techniques were designed with generalizability in mind and we discuss several implications of our work on further development of direct-touch data exploration techniques for scientific visualization in general.

Categories and Subject Descriptors (according to ACM CCS): I.3.m [Computer Graphics]: Miscellaneous—Scientific visualization; 3D interaction; direct-touch interaction; large displays; flow visualization.

## 1. Introduction

Due to its many benefits (e. g., [RDLT06, HMDR08, KAD09]) and the increasing availability of supporting display hardware, direct-touch interaction rightfully receives a lot of attention in many domains. In particular in the field of human-computer interaction, researchers develop and evaluate novel interaction techniques for multi-touch surfaces. In the field of visualization interest has largely been concentrated in information visualization and visual analytics (e. g., [FHD09, IF09, NDL*09, TIC09]).

Scientific visualization, however, has many specific constraints [Ise11]. It focuses in many cases on spatially explicit three-dimensional data for which the mapping from 2D touch input to 3D manipulations is not obvious. Moreover, scientific visualization often requires multiple different exploration techniques for the same type of dataset. For example, one may need to navigate, place cutting planes, seed particles, and probe data values. These interactions often even have to share the same interaction space on the screen, which can be difficult to achieve with any single 3D interaction technique developed for manipulating 3D objects. It is thus

imperative to investigate interaction techniques specifically tailored to the needs of scientific visualization [Kee10].

To address the mentioned challenges, we present a design study of an encompassing direct-touch exploration system in our example domain of fluid flow visualization (see the resulting interface in Fig. 1). We tackle the challenges of combining general 7 DOF navigation, 3 DOF cutting plane

<figure><img src="image_1.png" /><figcaption>**Figure 1:** *Our direct-touch fluid flow exploration tool in use.*</figcaption></figure>

<footnote n="1">¹ University of Groningen</footnote>

<footnote n="2">² DIGITEO</footnote>

<footnote n="3">³ LIMSI-CNRS</footnote>

<footnote n="4">⁴AVIZ-INRIA</footnote>

<footnote n="5">⁵ Université Paris-Sud</footnote>
interaction, additional 2 DOF drilling exploration, the use of
both volumetric visualization and iso-surface specification,
the placement of seed particles in 3D space (5 DOF or more),
and the exploration of temporal data aspects. Several of
the included interaction techniques share the same 2D input
space by employing dedicated postures [IH12] or by making
use of bi-manual interaction. In addition, we report on obser-
vations from interaction sessions with a representative num-
ber of domain experts, with consistent results. We chose this
study methodology because qualitative observations are par-
ticularly appropriate to provide holistic insight into complex
processes such as exploratory visualization [Car08, GB08].

One contribution of our work is thus an encompassing
direct-touch system for exploring data in the domain of
fluid flow visualization. More importantly, however, we con-
tribute a better understanding of how different 3D touch-
based interaction techniques can be integrated with each
other and the insight that different touch-based interaction
techniques (3D and 2D ones) can and need to be combined in
a single visualization system. Moreover, we show that touch-
based visualization can hugely benefit from its support of di-
rect manipulation as well as from the fluidity of interaction.
Finally, our observations indicate that collaborative visual-
ization can not only be facilitated by multi-touch settings
on large displays but that it is also desired by experts, even
when using vertical display configurations. The results of our
work inform the creation of direct-touch interfaces for other
domains since many of our supported exploration techniques
are core to scientific visualization in general.

The remainder of the paper is structured as follows. We
start by discussing related work in Section 2. Next, we de-
scribe our participatory design process in Section 3 and ex-
plain our design decisions in detail. In Section 4 we describe
our qualitative evaluation with five domain experts and de-
scribe and discuss our observations. We conclude the paper
in Section 5 and mention possibilities for future work.

## 2. Related Work

Work related to our own can be found in two major fields:
existing solutions that employ direct-touch interaction in the
context of scientific visualization and interaction with 3D
data or objects using direct-touch input in general.

### 2.1. Direct-Touch Interaction with 3D Data or Objects

In this latter group, several approaches have been pub-
lished. While a more complete list of approaches can be
found elsewhere (e. g., [Ise11]), here we only mention a
few selected techniques that are particularly relevant to our
own work. Generally, touch-based interaction (3D manip-
ulation/3D navigation) depends on the nature of the con-
trolled objects. For example, for manipulating individual and
medium-sized objects, techniques such as Hancock et al.'s
three-finger [HCC07] or sticky tools [HtCC09] can be em-
ployed, or a set of dedicated gestures as in Eden [KMB*11].

Using an constrained energy minimization approach, Reis-
man et al. [RDH09] facilitate intuitive rotate-scale-translate
(RST) interaction—well suited, in particular, for manipulat-
ing planar surfaces in 3D space. In all these cases, however,
objects that can be touched and which constrain the interac-
tion are necessary. If these are not available, e. g., in the case
of particle clouds or volumetric data, navigation of the dis-
played space by means of widgets such as Yu et al.'s FI3D
technique [YSI*10] or Cohe et al.'s tBox [CDH11] can be
used, as well as techniques to control the camera [HDKG08].

### 2.2. Direct-Touch Interaction in Scientific Visualization

Despite these general techniques, some research has also
specifically addressed touch-based interaction with scien-
tific visualization. In addition to some 2D techniques [FS05,
IEGC08], we are specifically interested in those approaches
that support interaction with 3D scientific data. Here we can
find two general approaches: those that develop techniques
for a *general type of data* and those which develop or inte-
grate techniques for a *specific application domain*. An exam-
ples for the former group is the previously mentioned FI3D
technique [YSI*10] that facilitates interaction with the visu-
alization space, for example for particle cloud data. A sec-
ond example is Coffey et al.'s Slice WIM [CML*11] that
supports the exploration of datasets in a virtual reality con-
text, using touch interaction with 2D projections of a stereo-
scopically projected miniature version of the dataset (i. e.,
avoiding the conflict between touch input and stereoscopic
projection) to control specific objects and the visualization
in general. Finally, Fu et al.'s powers-of-10 ladder [FGN10]
facilitates the interaction with a dataset's scale rather than
space and is applicable for most 3D visualizations.

Other work investigates very specific visualization do-
mains. For example, Sultanum et al. [SSS*10, SSSS11] de-
scribe techniques to support the exploration of potential
oil deposits. They employ, beyond ‘standard’ touch-based
navigation, posture-based techniques including dedicated lo-
cal “probing,” axis-aligned “cutting,” sub-object distortion
(“peeling”), and (assisted by tangibles) focus+context views
that remove parts of the dataset that lie between the fo-
cal point and the camera. In an application for orthopedic
surgery planning, Lundström et al. [LRF*11] developed a
multi-touch table setup that provides—due to their specific
application domain—a constrained set of 6 DOF general
3D navigation techniques. They combine these techniques
with specific exploration tools including “movable alternator
pucks” to specify exploration modalities and “natural size
zoom” for maintaining similarity to the clinical reality. Fi-
nally, Butkiewicz and Ware [BW11] describe an application
for stereoscopic oceanographic flow visualization to analyze
ocean currents. Unique about their setup is that the dataset is
relatively shallow and that it is projected using a tilted con-
figuration on a similarly tilted 3D display. This makes the in-
interaction with the stereoscopic data intuitive because there is
an obvious touch surface (the ocean’s surface) which is simi-
larly arranged in physical space (due to the tilted display) as
in virtual space (due to the tilted projection). Based on this
setup, they also employ precise 3D positioning techniques
using a two-finger pantograph selection.

Both the domain-specific setups and also the data-driven
approaches pay attention to the ability to combine different
interaction techniques in such a way that they can be em-
ployed in the same interaction space, an aspect that is also
essential in our approach. Mode switching is often enabled
using postures, while Sultanum et al. [SSSS11] also use tan-
gibles and Lundström et al. [LRF*11] virtual objects for the
same purpose. Some techniques essential in visualization—
beyond our ‘normal’ navigation and zooming [YSI*10]—
have been studied before. For example, cutting [SSSS11]
and cutting planes [CML*11], probing [SSSS11], particle
placement [BW11], and combinations of 2D and 3D inter-
action [CML*11] have previously been discussed individu-
ally. Our application domain of fluid mechanics, however,
like many other visualization fields requires all of these and
other techniques to be accessible, while some (e. g., cutting
planes) also need to be controllable with more DOF. We ad-
dress this problem with our design described next.

3. Multi-Touch Interaction with 3D Visualizations

Based on this understanding of the related work, we set out
to design a tool for exploratory visualization that facilitates
more than a simple 3D data navigation. We started with a
detailed analysis of the requirements of our chosen applica-
tion domain, flow visualization, but most apply to 3D scien-
tific visualization in general. The resulting constraints were
derived from several conversations with two fluid mechanics
experts (co-authors of this paper) at our research lab to under-
stand their current work practices as well as to extract needs
for a new touch-driven interface. Based on this input (Sec-
tion 3.1), we followed a participatory design process [SN93]
in which we closely involved the two experts in the new sys-
tem’s development. During the five-month development, we
met with them once every one to two weeks for a total of
approximately 15 meetings. We describe this process in Sec-
tions 3.2–3.6, and outline how the requirements as well as
other decisions influenced the interaction design process.

**3.1. Flow Visualization Interaction Requirements**

From initial interactions with two fluid mechanics experts we derived a list of requirements to guide our development. Additional requirements stem from our own past experience developing direct-touch interactions and scientific visualizations. We now describe how requirements R1–R11 were derived and list a summary at the end of this section.

Data and Temporal Exploration

The specific application case we used for our design (data
provided by the experts) is the exploration of time-dependent
(R1) fluid flow simulations that consist of a scalar 3D1C field

<figure><img src="image_2.png" /><figcaption>**Figure 2:** *Slice of a scalar FTLE field in Matlab, showing material divergence between close-by particles. The flow moves from the left to the right over the cavity; red lines indicate material frontiers through which mass fluxes are null.*</figcaption></figure>

and a vector 3D3C field (R2) [Hal01]. While the vectors en-
code the flow’s direction and velocity, the scalar field (Fig. 2)
is a Finite Time Lyapunov Exponent (FTLE) field that repre-
sents the rate of divergence of neighboring material particles.
The peaks of such a FTLE field represent Lagrangian Coherent
Structures (LCS) which act as a material frontier in the
flow. LCS frontiers separate areas of particles with different
behavior because the particles cannot cross the frontiers.

Scalar Field Exploration

Visualizations based on these scalar fields provide informa-
tion, for example, about how the mixing in a physical flow
happens—a problem relevant in practical applications. Other
kinds of important scalar fields in mechanics such as temper-
ature or concentration could also easily be used in the ap-
plication through trivial modifications, as would any vectors
fields such as vorticity or the magnetic field. As means for ex-
ploring the scalar field, the experts required both iso-surface
adjustment and a drilling interaction (R3, R8).

Exploration Using Cutting Planes

Our collaborating fluid mechanics experts reported that
they traditionally explore their simulations using two-
dimensional cuts through a three-dimensional dataset, for
example to analyze the flow behavior in and around cavi-
ties. Therefore, they requested the ability to use 2D visual-
ization techniques (R5) in the new interface, in addition to
3D interaction (R4). This 2D visualization should be sup-
ported through freely positionable cutting planes (R6)—this
positioning should be free in 3D space, axis-aligned cutting
planes are not sufficient (R6).

Vector Data Exploration

Moreover, it needs to be possible to also explore the vector
aspects of the dataset. A way to achieve this traditionally is
to seed the 3D3C field at any given timestep and to look at
the interaction between the particles and the field such as
by following their trajectories. This interaction should also
be possible for the new 3D visualization (R1, R7). For this
purpose it needs to be possible to flexibly define regions of
various sizes in the 3D space where particles are placed and
from which streamlines emerge (R7).

Control and Modes

From our own experience with touch-based scientific visual-
ization and related work in HCI, we chose a design which
focused on interaction techniques which avoid semaphoric
gestures (R9) to aid memory of supported interactions.
The needed modalities should be specified in the form of
spring-loaded, user-controlled modes [Bux86, SKB92] that
let people actively invoke certain interactions without risk-
ing modal errors. For example, one can use bi-manual in-
teraction, hand postures [IH12], and non-obtrusive widgets.
These widgets, however, should never cover the center of the
exploration (R10) to always be able to focus on the data.

Hardware Setup

Our physical display setup was inspired by our own past ex-
periences as well as by setups described in the literature,
specifically Lundström et al.’s Medical Visualization Table
(MVT) [LRF*11]. Similar to the MVT, we based out design
on a 1920 × 1080 tiltable display, but used a larger one (55
inch diagonal) to support more than one person comfortably
(R11). In contrast to the MVT, our setup (Fig. 8) is height-
adjustable and is primarily used in a vertical or close-to-
vertical orientation, guided by the preferences of the fluid
mechanics experts. The multi-touch input is provided by a
PQLabs Multi-Touch G³ Plus overlay which provides the
touch events via a TUIO [KBBC05] interface.

In summary, we used the following eleven requirements
to guide the development process:

R1: Temporal exploration.

R2: Visualization and exploration of a combination of vec-
tor 3D3C and scalar 3D1C data, using volumetric and
iso-surface modalities for the scalar field.

R3: Creation, adjustment, and removal of iso-surfaces.

R4: 3D navigation and zoom.

R5: 2D visualization and interaction.

R6: Cutting planes: free placement, orientation, translation.

R7: Definition of seed regions in 3D space from which streamlines emerge to show the vector field.

R8: Drilling to explore the scalar values along a 3D line.

R9: Using the interface's visual appearance to indicate function, i. e. use of bi-manual, postural, and widget-based methods.

R10: The data not obstructed by interaction widgets,
touches on the data limited to essential ones.

R11: The physical setup of the touch-based visualization
supporting at least two people comfortably and flex-
ibly configurable and adjustable.

**3.2. Data Visualization and Navigation**

Before we commenced with the project we decided on a sci-
entific visualization environment on which to base the de-
velopment. Our collaborators were most familiar with VTK

and ParaView, but ParaView's interface is heavily based on
a traditional menu-centered approach. VTK, however, pro-
vides access to basic visualization techniques without man-
dating a specific interaction paradigm and we consequently
decided to use VTK as our foundation. As our basic vi-
sualization techniques we chose—after discussion with the
fluid mechanics experts—to support both volumetric (us-
ing VTK's GPU-implemented volume ray casting) and iso-
surface visualizations of the scalar field (R2). While being
aware that rainbow-like color maps are not considered to be
ideal [BT07], we used the Jet color map for both techniques
because it is the one reported to be most often employed in
the flow visualization context. In the volume rendering, we
used a linear opacity ramp in which larger scalar values re-
ceive a larger opacity, based on what our fluid mechanics
experts commonly use (in ParaView).

For the interactive exploration of the 3D dataset at least
7 degrees of freedom (DOF) need to be supported: trans-
lation along three axes, orientation/rotation with respect to
three axes, and uniform zoom (R4). To realize this interac-
tion with direct-touch input several options exist as outlined
in the related work. The fundamental decision we needed
to make was whether to treat the data volume as a dedi-
cated object that can be used to constrain the interaction
or whether we rather wanted to support interaction with the
data space instead. Techniques that focus on 3D interaction
with smaller objects (e. g., [CDH11, HCC07, HtCC09]) were
less suitable for our purpose because we wanted to support
zooming into the dataset which leads to the data volume be-
coming large. Therefore, based on the same motivation as
presented by Yu et al. [YSI*10] for their interaction with par-
ticle data, we chose their FI3D widget as our most basic in-
teraction paradigm. FI3D also only uses at maximum two si-
multaneous touch points and, thus, configurations with more
than two fingers can be mapped to additional interactions.
Specifically, we chose their second interaction mapping—
one finger for rotation, perpendicular-to-frame interaction
for translation—because in our case rotations were expected
to be much more common interactions than translations, and

<figure><img src="image_3.png" /><figcaption>**Figure 3:** *General interface design with two data views, showing the iso-surface and volumetric visualizations of the scalar FTLE field using the domain-specific Jet colormap.*</figcaption></figure>
<figure><img src="image_4.png" /><figcaption>**Figure 4:** Cutting plane interaction with bi-manual control.</figcaption></figure>

since in most cases the whole dataset (with an inherent center) would be shown. As in the initial FI3D technique, the frame interaction was combined with a two-finger RST interaction [RDH09] to support intuitive integrated rotation-scale-translate interactions.

To show both the volumetric and the iso-surface visualizations at the same time (R2), we split the display into two main regions, each with an FI3D widget to control the regular 3D navigation (Fig. 3). Both views can be used independently or in a connected manner in which the view (projection settings) of one widget is mirrored in the other widget, while the selection of visualization techniques (volume rendering, iso-surfaces, both, or none of them) can be changed independently. In addition, the right side of the interface contains a view of the Jet color scale in which iso-values (represented as dots) can be added (by tapping on an empty spot), adjusted (by dragging a dot up or down), or removed (by dragging a dot out of the scale) to satisfy R3. Manipulation of the dots produces a new iso-surface rendering in real-time.

### 3.3. Cutting Plane & Drilling Interaction

In addition to the regular 3D interaction with the dataset we needed to also support the manipulation of cutting planes as a fundamental exploration technique (R6). This means had to integrate the 3DOF cutting plane interaction with 7DOF interaction with the dataset/data space—ideally both being invocable in the same interaction space. With the FI3D widget in use for the latter, three or more finger techniques can be used for reorienting or translating cutting planes. This makes it possible to employ, e. g., Reisman et al.'s screen-space technique [RDH09] for touch interaction with planes. For our specific application, we used Reisman et al.'s approach as an inspiration—in an abstracted form. We wanted to enable precise control and, using a similar interaction, be able to translate the cutting plane along its normal. We decided to use a bi-manual technique and to specify an axis on the cutting plane using two fingers of the non-dominant hand and use the motion of a third finger to specify an angular rotation around this axis (Fig. 4). The symmetry of the chosen

<figure><img src="image_5.png" /><figcaption>**Figure 5:** Drilling interaction (left) with bi-manual control; the 'drilling core' with its iso-values is shown both in the 3D view as well as in an undistorted way on its left side.</figcaption></figure>

bi-manual technique makes it always possible for the user to choose the non-dominant hand to specify the axis. This configuration by itself can still invoke RST manipulations if the fingers are moved. To enable translation, we use the same two-finger posture to specify the interaction with the cutting plane, but use one of the FI3D widget's frames to specify a translation. Here, the frame is used as an auxiliary control. To minimize obstruction of the data during both interactions (R10) the non-dominant hand can be released while the previously enabled interaction is continued as long as the third finger maintains contact. Our experts initially asked for non-planar cutting surfaces to be integrated. Together with the experts, however, we began with an interaction design for planar ones to restrict interaction complexity. Non-planar cutting planes would require the specification of arbitrary cutting surfaces and an interaction paradigm of its own.

We also implemented a drilling operation (R8) that is invoked bi-manually using a dedicated activation region on the left (non-dominant) frame of the FI3D widget. The dominant hand, in this interaction, specifies a point on the cutting plane, perpendicular to which a 'drilling core' is shown which displays the scalar values sampled along its path (Fig. 5). This is shown both as the 'drilling core' in the 3D view and as a 2D representation on the left side of the 3D view. Initially we had implemented a drilling interaction along the view ray defined by the dominant hand's touch, but after seeing it in action the experts suggested our current approach with the core perpendicular to the cutting plane.

### 3.4. 2D Visualization and Seed Point Interaction

Based on the traditional use of 2D visualizations by our collaborating experts we also wanted to support two-dimensional data exploration strategies (R5). We based these techniques on the cutting planes already specified in the interface, but added an additional 2D view that shows the undistorted intersection of the currently defined cutting plane (Fig. 6). In this 2D cut view we implemented techniques to place, in particular, seeds to be able to integrate streamlines in the 3D view (R7) as an essential and fundamental way to
<figure><img src="image_6.png" /><figcaption>**Figure 6:** Specifying seeds in the 2D cut view (two touches/hands, right) to interactively place 3D streamlines (left).</figcaption></figure>

explore the characteristics of the dataset’s vector field (R2), such as investigating the divergence of the particle trajectories when searching for LCS’ in our specific application. The combination of 3D cutting plane specification and 2D seeding allowed us to solve the otherwise difficult problem of specifying a seeding region in 3D space, despite only being able to interact on a 2D surface.

By extending previous 2D, posture-based particle placement techniques [IEGC08], we decided to employ one-finger, two-finger, and three-or-more-finger techniques to define regions in which to place seed particles that are used to integrate streamlines. The specific design of these techniques, however, was heavily guided by discussions with the fluid mechanics experts. For the one-finger interaction we first considered to place a single seed, but based on the feedback we received we later decided to place seeds in a small spherical region centered around the touch point to be able to see minute changes of the vector field in a small region. Similarly, for two fingers we initially only placed one seed each for the touch points, but based on feedback changed this to place seeds along the line connecting the two touch points. Three or more fingers can be used to define a larger sphere whose size can be adjusted. Initially we used this larger sphere to only seed on its surface, but based on feedback decided to seed evenly inside its volume.

In all three cases the experts finally reported that they require control over how densely seeds are placed, so we added such a control by means of using the 2D view's frame as an auxiliary slider control: dragging a finger of the non-dominant hand along this frame (depending on the dragging direction) increases or decreases the number of seeds (Fig. 7). This technique lets people transition, for example, between our initially envisioned single seed per touch point in the one- and two-finger interactions to a dense seeding pattern. Moreover, we also change from 'normal' streamlines to a ribbon representation (which require more space) when only few seed points are employed because these provide more information about the dataset's characteristics. The orientation of these ribbons is derived locally from vorticity of the flow vectors. To provide even more information during

<figure><img src="image_7.png" /><figcaption>**Figure 7:** Bi-manual control of the seeding density.</figcaption></figure>

the seeding exploration we also provide the numeric value of the scalar field for single-finger seeding interaction. Finally, as a seeding technique related to the two-finger technique, we also generate seed points along the drilling core when the drilling interaction is being used.

### 3.5. Temporal Exploration

We provide typical timeline-based interaction to support exploration of the dataset’s temporal behavior with a timeline placed at the bottom of the interface (Fig. 3). The time step displayed in the interface can be controlled by dragging the timeline’s time cursor. Playing the whole simulation is started by flip-dragging the cursor or briefly tapping it, and stopping by dragging the cursor out of the timeline. The temporal interaction takes the previously mentioned streamline seeding into account for both the posture-based and the drilling-based seed points. We store their position in space and time and based on this information show their motion over time. Finally, when other interactions are being performed, we stop a possibly running temporal animation due to performance reasons, and continue the animation when the interactions are completed.

### 3.6. Support for Co-located Collaboration

While we did not intentionally develop our interface to support it, the presence of two configurable views and the support of multi-touch interaction with up to 32 independent points facilitates the simultaneous collaborative interaction by at least two people. In particular, we allowed each of the two FI3D-based widgets to be switched independently between 3D view and 2D cut view, and in a 3D view the type of visualization (volumetric and iso-surfaces) can independently be enabled or disabled. Moreover, if both widgets show the 3D visualization, their views can be used independently or in a coordinated fashion. In the latter case, navigation interactions are replicated in the respective opposite widget so that both always show the same view.

This setup facilitates a range of different individual or collaborative exploration scenarios. Aside from individual
<figure><img src="image_8.png" /><figcaption>**Figure 8:** *Co-located collaboration between two experts.*</figcaption></figure>

work, groups of two people can work independently, each with their own view by switching between 2D- and 3D-based exploration. Alternatively, they can also choose closely coupled work by using the whole interface in a joint fashion (e. g., Fig. 8), for example one person controlling seed point placement in the 2D view while the other manipulates the 3D view on the dataset. While we did receive some insight on such work practices in our evaluation session on which we report next, more dedicated design for collaborative use of the visualizations would be necessary.

## 4. Evaluation and Feedback

In addition to the feedback we received during the participatory design process we also wanted to get a more thorough understanding of the interface toward the end of our development. For this purpose we asked our collaborating experts as well as five additional fluid mechanics researchers (who were previously not involved) to participate in longer observational evaluation sessions. We specifically chose a qualitative, observational study methodology [Car08] because we were interested in studying the *integration of several direct-touch interaction techniques for exploratory visualization*—not in whether one specific technique outperforms another one which could be studied with a quantitative controlled experiment. After all, from a single technique's performance we cannot learn how people approach a whole system and how they use the techniques in combination, in particular when studying a complex process such as exploratory visualization [Car08, GB08]. Moreover, specific performance and feedback benefits of direct-touch interaction techniques have been established in the past [RDLT06, KAD09].

We thus split the seven participants (6 male, 1 female, ages 23–57 years, median 39 years) into four groups, three groups with two participants (G1, G2, G4) and one person by himself (G3). One group (G1) comprised the two researchers who also participated in the design process and who are co-authors of this paper, thus we specifically point out and separate their feedback if it differs from the other three groups.

A complete evaluation session took about 1.5–2 hours and all sessions followed the same procedure. After having been informed about the topic of the evaluation, participants were first given a tutorial of the visualization and interaction tech-

niques using a simple dataset. Participants were allowed to try out the interactions themselves as well as to ask questions. Next, a larger dataset (used throughout this paper) was loaded and participants were asked to explore it, to look for interesting or unexpected aspects, or to confirm aspects they were familiar with. All participants were familiar with the used data type, with groups G1 & G3 and one person in G2 being most familiar with the specific data, while group G4 and the other person in G2 were familiar with similar fluid simulations. During this exploration phase, participants were asked to think-aloud, could ask questions to the experimenter, and were sometimes asked about their actions or were reminded of functionality (to reduce learning effects). Afterward, we conducted a semi-structured interview to discuss their experience in more detail, including differences between their traditional analysis process and the touch-based one and specific feedback about the direct-touch exploration tool. Finally, we asked the participants to fill out a questionnaire to inquire about demographics, background, and numeric ratings for their opinions about specific aspects (using 5-point Likert scales). Results were captured using videotaping as well as note-taking by the experimenter.

The findings from this expert evaluation can roughly be categorized into four groups: usability issues, interaction concepts, extensions of the tool, and implications for the design of other direct-touch scientific data exploration settings.

### Usability Issues

We naturally found a number of *smaller usability issues* with the current implementation of our interface. These include, e. g., some inconsistencies with the implementation of the timeline interaction, the lack of remembering the view settings when toggling between 2D and 3D views, the lack of indication of the streamlines’ direction, etc. These are all problems that can easily be addressed through small changes in the application. We also found usability issues which stemmed from the tested interaction design. These, in particular, relate to the interplay of 3D navigation and interaction with cutting planes in the same interaction space (i. e., interaction specified with postures). While all participants in G1 & G4 were able to deal with the interplay of techniques, one person in G2 and the person in G3 mentioned that it may require some learning to be used effectively. The other person in G2, however, found the interaction design for moving the cutting plane “unintuitive” and evidently had trouble to obtain the view settings and cutting plane configurations he desired. The observation that both experts in G1 were better able to use it than the other participants seems to further indicate that learning may help people better use the technique. The cutting plane orientation problem was, nevertheless, considered to be the most severe usability issue by all.

### Interaction Concepts

The challenges of cutting plane interaction were indicative of more fundamental requirements of touch interaction with three-dimensional scientific visualizations: the *need for pre-
cise interaction techniques and the need to isolate interac-
tions. The fact that initiating a cutting plane interaction al-
ways led to small changes in the view led to some visible
frustration during interaction. One person suggested to use
explicit system-controlled modes which he was familiar with
in his traditional tools. However, problems with moded in-
terfaces are well known and we believe it to be possible to
find interaction techniques with improved usability that do
not rely on system-controlled modes. Related to this inter-
action isolation issue, participants mentioned several times
that they need to be able to achieve specific configurations
of the cutting plane—such as planes parallel to the sides of
the data box—to be able to make precise measurements. Nev-
ertheless, the interface as a whole was not considered to be
imprecise as may be assumed from the ‘fat finger problem.’
Reasons for this reaction that were mentioned by the par-
ticipants include that the specific type of data had a lower
resolution than the size of pixels on the screen, that other
interactions (e. g., streamline exploration) were considered
to be precise, and that the supported exploratory visualiza-
tion was mentioned to have a more “qualitative” character in
which pixel precision was less important.

Extensions

Despite the above mentioned usability challenges, partici-
pants viewed the application as very useful for exploratory
visualization and asked for a number of additional features
to make the tool even more useful in practice. Numerous
ideas were mentioned and we, thus, highlight a selection of
features participants asked for—some of them easy to realize
while others are more challenging. For example, participants
asked the ability to apply transformations to the 2D view, not
only panning and zooming but also flipping to reduce men-
tal mapping problems if the 3D view and its cutting plane
are rotated by more than 90° around some axis. Also, related
to the issue of precise interaction, participants asked for de-
fault views that are frequently used in exploration and which
would allow them to easily return to a well-known configu-
ration. Similarly, to ease navigation, participants asked for
more specific information about the displayed data in form
of detailed numeric read-outs, axis labels, small 3D (and pro-
jected 2D) coordinate system axes, etc. To be able to explore
the vector properties more easily, the experts suggested plac-
ing small vectors (rather than streamlines) on the cutting
plane or along a line (like our drilling core), placing sev-
eral instances of streamlines to be able to compare cyclic be-
havior, and specifying locations that continuously emit par-
ticles. Naturally, they also requested the ability to toggle be-
tween several different scalar properties as well as to use
iso-surfaces with a transparency. A very interesting sugges-
tion, yet more difficult to realize, is to capture a history of
interactions that would allow researchers to reproduce the
exploration, to get images of specific encountered views or
configurations, or to run similar explorations with other data.

Implications for Design

Finally, we learned from our experts about direct-touch inter-

action with 3D scientific data in general. In particular, three
main aspects were mentioned to be the most important ad-
vantages of the presented interaction combined with direct-
touch input: direct manipulation and fluidity of interaction,
the combination of 2D with 3D exploration, and the sup-
port of collaboration. Participants mentioned that the first
of these aspects—the ability to directly manipulate the data,
the rapid access to exploration facilities, the fluidity of the
interaction, and the immediate feedback—was essential for
their effective exploration of the dataset. One person men-
tioned that the latency both from the touch processing and
the demanding rendering of the visualization may start to
become an issue if it became greater, but that the feedback
that was provided helped to alleviate the problem. Compared
to their traditional tools, the experts also named as an ad-
vantage that it was not necessary to build dedicated visual-
ization pipelines. For touch-based scientific visualization we
can thus learn that the ability to directly manipulate and the
interaction fluidity are essential and should be maintained.

However, as we approach more realistic visualization set-
tings, the interfaces will necessarily become more complex.
This means that touch-based interaction cannot always re-
main completely intuitive but in some aspects may have to
be learned. In our observations and in the experts’ responses
we saw that this fact is accepted by the people. They men-
tioned that interactions can be learned by exploring the in-
terface while exploring the data at the same time, one hence
needs to provide an explorable and exploratory interface.

The second aspect concerns the combination of 2D and
3D exploration techniques. It was evident that providing 2D
exploration tools that are inter-connected with the 3D visual-
ization is essential, in particular since several of the experts
traditionally use 2D exploration tools for fluid flow research.
While the placement of seed points by means of a cutting
plane may be less direct than with, e. g., 3D tracking in VR,
the technique at least allowed a reasonably flexible 3D seed-
ing of streamlines with control over the shape and charac-
teristics of the seed region. This worked well for 5 of the 7
participants (2 x 'liked it' and 3 x 'definitely liked it'), the re-
maining 2 rated it 'neutral.' In fact, one of the participants
(G3) reported a previous experience of fluid flow visualiza-
tion in a VR environment using a Phantom for specifying
seed point placement directly in 3D space. He stated that he
liked the direct-touch interaction much better than the VR
setting and that he wants to use our tool in the future to ex-
plore his own data. In this context he also mentioned that
stereoscopic displays should be investigated but that he is
not sure that it would be better than a monoscopic one.

The final aspect relates to the ability to collaboratively
exploring a visualization with an interface based on direct-
touch control. In our (interaction and evaluation) setting
we specifically supported and observed the collaboration be-
tween two colleagues discussing a common problem. This
type of collaboration was liked by G1, G2, and G4 (working
together worked ‘well’ 3× and ‘definitely well’ 3×) and G3 mentioned that he would really want to collaborate with others using the tool (without being asked about this fact specifically). All mentioned that 2–3 people would be best, with 2 people being the ideal configuration. The experts specifically mentioned the issue of awareness of the other’s actions being a benefit, that “collaboration came naturally,” and that they “complemented each other” in their work. However, G1, G2, and G4 exhibited very different collaboration styles. G1 closely collaborated, at many times worked simultaneously, and did not notice any interference while doing that. G2, in contrast, were explicit about taking turns and mentioned that usability issues kept them from working at the same time due to fears that an additional person interacting would cause (more) problems with the work of the first person. G4’s style was somewhat in-between that of G1 and G2, with G4’s participants taking turns without being explicit about it but discussing while one person was interacting.

This preference for collaboration using our vertical display setting to some degree contradicts Rogers and Lindley’s findings [RL04] that vertical displays are “difficult and awkward to collaborate around.” We hypothesize that this is due to the specific type of data we used: 3D data that is thought to have an inherent orientation (with an up and a down). We specifically asked our participants after the session whether they would have preferred a horizontal setup (and demonstrated that the display can be turned), and all reported that the vertical or a slightly tilted setting would have been best. Of course, we cannot derive statistically significant findings from these few observations and a dedicated study would need to investigate the suitability of both horizontal and vertical settings for 3D data analysis, but we saw indications that vertical or slightly tilted setups are not as bad for scientific visualization (for small groups of 2–3 people, e. g., discussions of colleagues) as some literature [RL04] suggests.

## 5. Conclusion and Future Work

We presented a design study for supporting exploratory visualization on a direct-touch platform, using fluid mechanics as our example domain. We combined several interaction techniques that permit both the use of different exploration techniques in the same interaction space (the 3D view) as well as the combination of 3D and 2D exploration techniques. As part of both, we developed a technique to be able to place seed points into the 3D visualization space in various configurations, despite only using input from a 2D touch surface.

We closely worked with two domain experts to build a tool that is useful for them in practice for their work. Its usefulness and power was validated in an observational study with five additional domain experts—one of which, on his own initiative, expressed that he wants to continue and explore interaction with his own data. While many possibilities exist to improve the current tool, we showed that systems entirely controlled using multi-touch interaction can

be very useful for interactive exploratory scientific visualization. The expert participants in our observational study said that tools like ours are good to get a first impression about an unknown dataset and to identify interesting aspects to investigate in detail, before proceeding with other, more specialized tools. Participants suggested that it would be good to alternate between traditional and touch setups to test ideas.

Our study also revealed that our initial approach did not yield a 100% ideal solution. Ways to alternate between view control and cutting plane manipulation, as well as the specific technique for interacting with the cutting plane, need further improvement. However, our study also showed that with learning people can master our technique and that it is possible to integrate several different 3D interaction techniques (view navigation, cutting plane interaction, drilling interaction) in the same interaction space (3D view widget).

The most important findings from our evaluation and thus this paper’s conclusions, however, are those that relate to touch interaction with 3D data in general. We saw that direct manipulation and interaction fluidity are essential to permit rapid idea exploration, that combinations of 3D and 2D visualization and exploration techniques are useful, in particular, if 2D techniques are frequently used traditionally, and that collaboration using vertical display settings can be suitable and desired in exploratory scientific visualization. In addition to the mentioned improvements to the specific tool and the exploration of other domains, it will thus be important to study different display orientations for scientific visualization, to come up with guidelines on how and when vertical or tilted setups support collaboration in small groups, and to understand how collaboration can be further encouraged and supported. Moreover, it would be necessary to develop an integrated interaction toolkit that better supports the effective and intuitive combinations of multiple different 3D interaction techniques with different visualization tools [Ise11].

## Acknowledgments

We thank all study participants for their input as well as Dorine Caqueret and Jean-Marc Vézien for building the display and Petra Isenberg for her feedback on the paper.

## References

[BT07] BORLAND D., TAYLOR II R. M.: Rainbow Color Map (Still) Considered Harmful. *IEEE Computer Graphics and Applications* 27, 2 (2007), 14–17. doi> 10.1109/MCG.2007.323435

[Bux86] BUXTON W.: Chunking and Phrasing and the Design of Human-Computer Dialogues. In *Proc. IFIP World Computer Congress* (1986), pp. 475–480.

[BW11] BUTKIEWICZ T., WARE C.: Exploratory Analysis of Ocean Flow Models with Stereoscopic Multi-Touch. In *IEEE Visualization Posters* (2011).

[Car08] CARPENDALE S.: Evaluating Information Visualizations. In *Information Visualization: Human-Centered Issues and Perspectives*, vol. 4950 of LNCS. Springer-Verlag, Berlin/Heidelberg, 2008, pp. 19–45. doi> 10.1007/978-3-540-70956-5_2
[CDH11] COHÉ A., DÈCLE F., HACHET M.: tBox: A 3D Trans-formation Widget Designed for Touch-Screens. In *Proc. CHI* (2011), ACM, New York, pp. 3005–3008. doi> 10.1145/1978942.1979387

[CML*11] COFFEY D., MALBRAATEN N., LE T., BORAZJANI I., SOTIROPOULOS F., KEEFE D. F.: Slice WIM: A Multi-Surface, Multi-Touch Interface for Overview+Detail Exploration of Volume Datasets in Virtual Reality. In *Proc. I3D* (2011), ACM, New York, pp. 191–198. doi> 10.1145/1944745.1944777

[FGN10] FU C.-W., GOH W.-B., NG J. A.: Multi-Touch Techniques for Exploring Large-Scale 3D Astrophysical Simulations. In *Proc. CHI* (2010), ACM, New York, pp. 2213–2222. doi> 10.1145/1753326.1753661

[FHD09] FRISCH M., HEYDEKORN J., DACHSELT R.: Investigating Multi-Touch and Pen Gestures for Diagram Editing on Interactive Surfaces. In *Proc. ITS* (2009), ACM, New York, pp. 149–156. doi> 10.1145/1731903.1731933

[FS05] FORLINES C., SHEN C.: DTLens: Multi-User Tabletop Spatial Data Exploration. In *Proc. UIST* (2005), ACM, New York, pp. 119–122. doi> 10.1145/1095034.1095055

[GB08] GREENBERG S., BUXTON B.: Usability Evaluation Considered Harmful (Some of the Time). In *Proc. CHI* (2008), ACM, New York, pp. 111–120. doi> 10.1145/1357054.1357074

[Hal01] HALLER G.: Distinguished Material Surfaces and Coherent Structures in Three-Dimensional Fluid Flows. *Physica D: Nonlinear Phenomena* 149, 4 (Mar. 2001), 248–277. doi> 10.1016/S0167-2789(00)00199-8

[HCC07] HANCOCK M., CARPENDALE S., COCKBURN A.: Shallow-Depth 3D Interaction: Design and Evaluation of One-, Two- and Three-Touch Techniques. In *Proc. CHI* (2007), ACM, New York, pp. 1147–1156. doi> 10.1145/1240624.1240798

[HDKG08] HACHET M., DECLE F., KNÖDEL S., GUITTON P.: Navidget for Easy 3D Camera Positioning from 2D Inputs. In *Proc. 3DUI* (2008), IEEE Computer Society, Los Alamitos, pp. 83–89. doi> 10.1109/3DUI.2008.4476596

[HMDR08] HORNECKER E., MARSHALL P., DALTON N. S., ROGERS Y.: Collaboration and Interference: Awareness of Mice or Touch Input. In *Proc. CSCW* (2008), ACM, New York, pp. 167–176. doi> 10.1145/1460563.1460589

[HtCC09] HANCOCK M., TEN CATE T., CARPENDALE S.: Sticky Tools: Full 6DOF Force-Based Interaction for Multi-Touch Tables. In *Proc. ITS* (2009), ACM, New York, pp. 145–152. doi> 10.1145/1731903.1731930

[IEGC08] ISENBERG T., EVERTS M., GRUBERT J., CARPENDALE S.: Interactive Exploratory Visualization of 2D Vector Fields. *Computer Graphics Forum* 27, 3 (May 2008), 983–990. doi> 10.1111/j.1467-8659.2008.01233.x

[IF09] ISENBERG P., FISHER D.: Collaborative Brushing and Linking for Co-located Visual Analytics of Document Collections. *Computer Graphics Forum* 28, 3 (Mar. 2009), 1031–1038. doi> 10.1111/j.1467-8659.2009.01444.x

[IH12] ISENBERG T., HANCOCK M.: Gestures vs. Postures: ‘Gestural’ Touch Interaction in 3D Environments. In *Proc. 3DCHI* (2012). To appear.

[Ise11] ISENBERG T.: Position Paper: Touch Interaction in Scientific Visualization. In *Proc. ITS Workshop on Data Exploration on Interactive Surfaces—DEXIS* (2011).

[KAD09] KIN K., AGRAWALA M., DEROSE T.: Determining the Benefits of Direct-Touch, Bimanual, and Multifinger Input on a Multitouch Workstation. In *Proc. Graphics Interface* (2009), Canadian Information Processing Society, Toronto, pp. 119–124.

[KBBC05] KALTENBRUNNER M., BOVERMANN T., BENCINA R., COSTANZA E.: TUIO: A Protocol for Table-Top Tangible User Interfaces. In *Proc. Workshop on Gesture in Human-Computer Interaction and Simulation* (2005).

[Kee10] KEEFE D. F.: Integrating Visualization and Interaction Research to Improve Scientific Workflows. *IEEE Computer Graphics and Applications* 30, 2 (Mar./Apr. 2010), 8–13. doi> 10.1109/MCG.2010.30

[KMB*11] KIN K., MILLER T., BOLLENSDORFF B., DEROSE T., HARTMANN B., AGRAWALA M.: Eden: A Professional Multitouch Tool for Constructing Virtual Organic Environments. In *Proc. CHI* (2011), ACM, New York, pp. 1343–1352. doi> 10.1145/1978942.1979141

[LRF*11] LUNDSTRÖM C., RYDELL T., FORSELL C., PERS-SON A., YNNERMAN A.: Multi-Touch Table System for Medical Visualization: Application to Orthopedic Surgery Planning. *IEEE Transactions on Visualization and Computer Graphics* 17, 12 (Dec. 2011), 1775–1784. doi> 10.1109/TVCG.2011.224

[NDL*09] NORTH C., DWYER T., LEE B., FISHER D. ISEN-BERG P., ROBERTSON G., INKPEN K.: Understanding Multi-Touch Manipulation for Surface Computing. In *Proc. INTERACT* (2009), Springer-Verlag, Berlin/Heidelberg, pp. 236–249. doi> 10.1007/978-3-642-03658-3_31

[RDH09] REISMAN J. L., DAVIDSON P. L., HAN J. Y.: A Screen-Space Formulation for 2D and 3D Direct Manipulation. In *Proc. UIST* (2009), ACM, New York, pp. 69–78. doi> 10.1145/1622176.1622190

[RDLT06] ROBLES-DE-LA-TORRE G.: The Importance of the Sense of Touch in Virtual and Real Environments. *IEEE MultiMedia* 13, 3 (July–Sept. 2006), 24–30. doi> 10.1109/MMUL. 2006.69

[RL04] ROGERS Y., LINDLEY S.: Collaborating Around Vertical and Horizontal Large Interactive Displays: Which Way Is Best? *Interacting with Computers* 16, 6 (Dec. 2004), 1133–1152. doi> 10.1016/j.intcom.2004.07.008

[SKB92] SELLEN A. J., KURTENBACH G. P., BUXTON W. A. S.: The Prevention of Mode Errors Through Sensory Feedback. *Human Computer Interaction* 7 (June 1992), 141–164. doi> 10.1207/s15327051hci0702_1

[SN93] SCHULER D., NAMIOKA A. (Eds.): *Participatory Design: Principles and Practices*. CRC/Lawrence Erlbaum Associates, Hillsdale, NJ, USA, 1993.

[SSS*10] SULTANUM N., SHARLIN E., SOUSA M. C., MIRANDA-FILHO D. N., EASTICK R.: Touching the Depths: Introducing Tabletop Interaction to Reservoir Engineering. In *Proc. ITS* (2010), ACM, New York, pp. 105–108. doi> 10.1145/1936652.1936671

[SSSS11] SULTANUM N., SOMANATH S., SHARLIN E., SOUSA M. C.: “Point it, Split it, Peel it, View it”: Techniques for Interactive Reservoir Visualization on Tabletops. In *Proc. ITS* (2011), ACM, New York, pp. 192–201. doi> 10.1145/2076354.2076390

[TIC09] TOBIASZ M., ISENBERG P., CARPENDALE S.: Lark: Coordinating Co-located Collaboration with Information Visualization. *IEEE Transactions on Visualization and Computer Graphics* 15, 6 (Nov./Dec. 2009), 1065–1072. doi> 10.1109/TVCG.2009.162

[YSI*10] YU L., SVETACHOV P., ISENBERG P., EVERTS M. H., ISENBERG T.: FI3D: Direct-Touch Interaction for the Exploration of 3D Scientific Visualization Spaces. *IEEE Transactions on Visualization and Computer Graphics* 16, 6 (Nov./Dec. 2010), 1613–1622. doi> 10.1109/TVCG.2010.157