ISSN 2186-7437

NII Shonan Meeting Report

No. 2016-2

Immersive Analytics: A new
multidisciplinary initiative to explore
future interaction technologies for data
analytics
Takayuki Itoh
Kim Marriott
Falk Schreiber
Uwe Wössner
February 15–18, 2016

National Institute of Informatics
2-1-2 Hitotsubashi, Chiyoda-Ku, Tokyo, Japan

Immersive Analytics: A new multidisciplinary
initiative to explore future interaction
technologies for data analytics
Organizers:
Takayuki Itoh (Ochanomizu University, Tokyo, Japan)
Kim Marriott (Monash University, Melbourne, Australia)
Falk Schreiber (Monash University, Melbourne, Australia
and University of Konstanz, Germany)
Uwe Wössner (HLRS Stuttgart, Germany)
February 15–18, 2016

1

Introduction

There is a tidal wave of big and complex data across many domains. Several research
directions are concerned with the development of methods to support the analysis ofsuch data, including machine learning, information and data visualisation, data analytics and human computer interaction. However, there is still a major gap to fill: how
can we immerse analysts in the data in order to provide more natural ways for data
exploration, data analysis and collaboration?
In recent years we have seen significant advances in the development of new technologies for human-computer interfaces. Immersive environments like a CAVE makeuse of ultra-high resolution technology, and combine 2D and 3D visualisations to allow users to immerse themselves into computer generated scenes, e.g., in the virtual
reality created in a flight simulator. Driven by the entertainment industry, tools andmethods like voice- and gesture-based control as well as 3D VR and AR visualisation
are becoming more and more part of everyday life. This has meant that technologies
for natural user interfaces, virtual and augmented reality devices have progressed very
quickly to provide very engaging and immersive experiences, and they are rapidly becoming available and affordable for small businesses and the general public.
In response to these new technologies, we propose a new facet of data analytics
research that we call Immersive Analytics. This new research direction explores how
these new display and interaction technologies can be used to create more engaging
experiences and seamless work-flows for data analysis applications. The environments
we envision that will be developed by immersive analytics researchers should be usable by experts and analysts to help in the detailed analysis of complex data sets, butwill also be accessible to decision makers, that is, the managers who spend more timeworking face-to-face than in front of a desktop computer, and to the everyday public tohelp them in tasks like balancing nutrition, social and ethical concerns and price whenbuying groceries.

1

The ability to move inside the data with new display devices, to interact with thedata representation in various direct and embodied ways allows the user to create aricher experience that may lead to better understanding and finally deeper insight. For
that to happen, the visualisation and interaction concepts need to allow a faithful representation of the data and provide intuitive and efficient orientation and navigation in
the data space. We expect that to create productive tools many gaps will need to be
filled.
Immersive analytics is inherently multi-disciplinary and involves researchers fromhuman computer interaction, data sciences, visual analytics, augmented reality, dataand information visualisation, as well as experts from application areas.
The aim of the seminar is to bring together experts in these fields to• define a new research field, ‘Immersive Analytics’,• create a community of researchers into immersive analytics,• take the first steps in the creation of a research manifesto for the new field.

2

Program and Schedule

The seminar’s schedule reflected that the main goals of the meeting were different fromusual meeting goals.
Experts from quite diverse fields did participate and thus we needed introductory
talks to give an overview on the research fields and application areas.
• Wolfgang Stuerzlinger gave an overview talk on a VR perspective for Visual
Analytics,• Christopher Collins gave an overview talk on an Information Visualisation perspective for Visual Analytics.
Tim Dwyer, Bruce H. Thomas, Yang Cheng, Lingling Shen, Oliver Kohlbacher and
Tom Chandler gave examples from application areas.
However, most time was devoted to discussions in small groups focusing to investigate the research questions, potential, and challenges of immersive analytics, and toprepare the research manifesto. The participants agreed on the idea to prepare a bookpublication for the manifesto.
Initially, groups were formed to propose a definition for immersive analytics. First
discussions were inconclusive, but by the end of the meeting the participants agreed on
the following working definition of immersive analytics.
Immersive Analytics studies how to engage people with data and with eachother in data analysis and decision making with the aim of improving theoutcomes. Engagement may be through the use of immersive virtual environment technologies or responsive interactive analytics but is not limited
to these technologies.
After the introductory and talks sessions on the first morning, a general discussion lead to the formulation of challenges and research questions for working groups.
The rest of the seminar was spent in working groups, except for a few feedback andreporting sessions. The meeting’s schedule is given in Table 1.

2

Sunday:
Day 1:
09:00:
10:00:
10:30:
11:00:
12:00:
13:30:
14:00:
15:00:
15:30:
16:00:
17:00:
18:00
Day2:
09:00:
09:15:
10:00:
10:30:
11:00:
11:30:
12:00:
12:15:
13:30:
15:30:
16:00:
17:00:
18:00
Day3:
09:00:
09:15:
10:30:
11:00:
11:30:
12:00:
13:30:
19:00:
Day4:
09:00:
09:15:
10:30:
11:00:
11:30:
12:00:

Welcome Banquet
Seminar start, welcome, introductory presentations, and working
groupsOpening and individual introductions
Introduction to Immersive Analytics and goals of the meeting(Kim Marriott)
Break
Invited Talks session (Wolfgang Stuerzlinger and Chris Collins)
Lunch
Group discussions
Working group discussions – IA research agenda
Group reports
Break
Book discussion: Definition of book chapters and general book structure
Definition of working groups according to book chapters
DinnerWorking groups, discussions, presentations
Recap from Monday
Applications presentations
Group discussion: How does IA map onto those areas?
Coffee break
Working groups
Progress reports
Group photo
Lunch
Working groups
BreakWorking groups
Progress reports
DinnerWorking groups and excursion
Recap from Tuesday
Working groups
BreakWorking groups
Progress reports
Lunch
Excursion
BanquetWorking groups, discussions, planning beyond the seminar
Recap from Wednesday
Working groups
BreakWorking groups
Progress reports and wrap up
Lunch – end of the seminar
Table 1: The meeting schedule3

3Working Group Reports

3.1

Applications in Life & Health Sciences

Participants: Yang Cheng, Tobias Czauderna, Oliver Kohlbacher, Karsten Klein,
Falk Schreiber, Lingling Shen
Summary: The group “Applications in Life & Health Sciences” was responsible
to outline the chapter about Immersive Analytics applications in the life and healthsciences domain. It started off their discussions together with the group “Exploring Immersive Analytics For Built Environments” about a general outline of chapters discussing applications in Immersive Analytics. Eventually the two groups decided to break down the application chapters into the four sub-chapters “Introduction”,“Overview Application Examples”, “Application Examples in Detail”, and “Challenges
and Outlook”. The ongoing discussions took place in the individual group with a particular focus on the life and health sciences domain and the goal to outline the subchapters in more detail.
The sub-chapter “Introduction” will illustrate which part of the application area iscovered, will review state-of-the-art Visual Analytics and Immersive Analytics in thelife and health sciences domain, and will introduce data types as well as discuss thecomplexity of biomedical data. Sub-chapter “Overview Application Examples” will
briefly showcase application examples covering the different omics technologies aswell as application examples from healthcare and medicine to provide the reader witha broader overview of possible applications. In “Application Examples in Detail” anumber of applications will be presented in more detail.
During the discussion the group came up with a common structure for the examples
being presented. For each example different questions should be addressed, e.g., a)
What is the problem to be solved?, b) Which approaches/methods have been used toaddress this problem?, c) What worked and what didnt work?, and d) How can or did
methods from Immersive Analytics help to solve the problem?
Currently, the list of detailed examples includes four examples covering quite diverse topics: one example exploring an omics (genomics) data dataset, one example
from structural biology, one example discussing an image analysis related problem, andone example detailing a healthcare/medical application in particular electronic healthrecords.
Finally, the sub-chapter “Challenges and Outlook” will discuss challenges suchas the amount of data, the integration of data, and problems with existing approaches.
Furthermore an outlook to further developments of applications in Immersive Analytics
will be given.

3.2

Exploring Immersive Analytics For Built Environments

Participants: Tom Chandler, Uwe Wössner, Torsten Kuhlen
Summary: In simulations of the built environment, 3D is much more than a graphic
gimmick; it is crucial to discern essential information such as line of sight issues, material flows and intersections of objects, all which are best remedied by a full 3D, dynamic
reconstruction. The utility of 3D models has been known to architects, engineers andurban planners for some time, but new Immersive Analytic interfaces, augmentations

4

(a) Crowd simulation for archeological sites(b) Architectural design(c) Factory planning

Figure 1: Built environments
and overlays promise to significantly extend the interpretation and exploration of thesevirtual environments.
In this chapter we will overview ongoing research into three prototypical built environments ranging from production planning, architectural design to the reconstructionof heritage listed landscapes and archaeological sites, see Figure 1.
Outlook and Challenges All of the applications we have discussed above have thepotential to be deployed as Augmented Reality applications in the field.
Annotation and Provenance tracking these immersive 3D environments are visually and spatially rich, and they also include a wide range of text overlays, - a general
problem is that we are mostly navigating within the immersive environment and thechallenge is to embed documentation that can be viewed consistently within the environment without loosing immersion, occlusion issues, and unnecessarily complicatinguser experience
A minor point - in our field, aspects like motion sickness and latency are morecritical than compared to, for example, medical visualisations and neuroscience where
the graphics are more centred and not presented as scale models of real spaces careful
design of the interfaces is important in dealing with challenges like hiding latency.

3.3

Immersive Analytics: Human in the Loop

Participants: Jian Chen, Christopher Collins, Tim Dwyer, Andreas Kerren, Gerik
Scheuermann, Wolfgang Stuerzlinger
Summary: We consider the space of human engagement with data analysis tools.
Machine learning techniques are rapidly growing in power in their ability to find pat5

terns in data, yet they remain dumb in the sense that a human is still required to interpretthe implications of these patterns. To make tools that support people in this collaboration with machines we need to first understand how people form insights about data
and the limits of their cognition and memory. Then, we need to consider the technical issue of how automatic algorithms used in data analysis must be instrumented
to support close supervision and control by humans. Further, we consider interfacesthat can support such interaction with these analytics engines providing, for example,
controlling a number of simultaneous analysis, differential views, and so on. Finally,
we consider future directions using advanced human physiological input and biofeedback to create enhanced adaptive and seamless integration of human supervision withmachine analytics.

3.4

Immersive Interactive Storytelling

Participants: Masahiko Inami, Harald Reiterer, Bruce Thomas, Chris Weaver
Summary: Immersive environments hold great potential for interactive wandering,
investigation, interpretation, and narrative. We investigated how people inhabit virtual
data spaces to create, examine, explore, interpret, explain, and share information andknowledge. Immersion offers a richly structured design space to target different degrees of immersion, dimensions of data, forms of immersion, modes of transition andnavigation, scopes of audience and user roles, and kinds of narrative. We used sidebar stories from our own experiences to share several current examples and discuss theimmersive interactive storytelling needs of domain experts.
Motivated by these stories, we laid out a framework for defining how immersive
spaces enable location-oriented, scale-oriented, path-oriented, and rate-oriented modesof entry, movement, and exit through visualized data. We focused on how the wide variety of navigation, selection, annotation, and composition interactions of existing virtual environments and visualization tools (and well-known taxonomies of these) carryover into interaction in immersive spaces. We also examined the effects on immersion of coordinated and collaborative styles of interaction that engage multiple visualperspectives simultaneously.
The framework aims to support a wide variety of narrative styles ranging fromexplanatory (highly constrained by a designer) to exploratory (moderately constrainedby a designer for a wandering user) to situational (lightly constrained by a user onthe-fly within a shaped world) to ambient (minimally constrained through pursuit ofaffective directions). Based on this framework, we sketched out design guidelines andrecommended best practices for integrating visual interactions with path, pace, andperspective into tools that support genuinely immersive storytelling.

3.5

Reconsidering the use of 3D for abstract data visualisation

Participants: Marcel Hlawatsch, Takayuki Itoh, Kim Marriott, Miguel Nacenta, Guido
Reina
Summary: During the last decade information visualisation researchers have havebeen very cautious about the use of immersive VR and AR for abstract data representation. This conservatism is quite deliberate. In a recent popular text on data visualization, Munzner [1] cautions against use of 3D for representation of abstract data. The

6critique is posed as a reaction to “unbridled enthusiasm” for 3D in the 90’s and cites
several studies that did not find any benefit to particular 3D mappings for abstract data.
In this chapter we review the evidence for and against the use of 3D for abstract
data visualisation and suggest that the answer is not clear cut and that in some situations
and tasks it may be preferable to 2D. The chapter is organised as follows• Perception of 3D• Limitations of 3D for abstract data representation• Potential of 3D for abstract data representation: examples and studies• A novel design space framework for abstract data visualisation that maps onto
different medium (paper, standard display, VR, AR, physical model)• Using the framework• Research questions

References[1] Tamara Munzner. Visualization Analysis and Design. CRC Press, 2014.

4

Participants• Takayuki Itoh, Ochanomizu University• Kim Marriott, Monash University• Falk Schreiber, Monash University• Uwe Wössner, University of Stuttgart• Tom Chandler, Monash University• Jian Chen, University of Maryland• Yang Cheng, China Novartis Biomedical Research Institute• Christopher Collins, University of Ontario• Tobias Czauderna, Monash University• Tim Dwyer, Monash University• Marcel Hlawatsch, University of Stuttgart• Jason Hideyo Haga, AIST
• Masahiko Inami, Keio University• Andreas Kerren, Linnaeus University• Karsten Klein, Monash University• Oliver Kohlbacher, University of Tuebingen7• Torsten Kuhlen, RWTH Aachen• Shen Lingling, China Novartis Biomedical Research Institute• Miguel Nacenta, University of St Andrews• Guido Reina, Stuttgart University• Harald Reiterer, Konstanz University• Gerik Scheuermann, Leipzig University• Wolfgang Stuerzlinger, Simon Fraser University• Bruce Thomas, University of South Australia• Chris Weaver, The University of Oklahoma

8Overview of Talks
There were seven invited talks, two overview talks and five talks on applications.

Information Visualization Overview and the State of 3D Visualization
Christopher Collins, CSIRO & University of Ontario Institute of Technology
This talk reviewed the basics of information visualization theory and practice, including how to construct visualizations and evaluate their effectiveness, following a
number of theoretical framings of visualization, including the nested model of Tamara
Munzner and the visual variables model of Jacques Bertin. This was intended to help
participants who are in other fields such as virtual reality learn a bit about the visualization side of immersive analytics. The talk also reviewed the systemic resistanceto using 3D rendering in information visualization, and exposed the criticisms of 3D
which are prevalent in the research community. I discussed the potential of virtual reality to improve visualization research and why some of these arguments against 3D
and VR may no longer be valid as new interaction devices and display technologies are
becoming more readily available.

Virtual Reality + Visual Analytics
Wolfgang Stuerzlinger, Simon Fraser University
My presentation initially reviews fundamental properties of 3D graphics, stereo
display systems, as well as the spatial and interaction abilities of humans. Based on
this, I discuss various ways in which visual analytics systems can support immersion as
well increase engagement of the user with the content. Finally, I briefly introduce the
V4-Space system, which uses eight very large touch-sensitive high-resolution displays
in portrait mode to support seamless interaction with big data.

Software Engineering
Tim Dwyer, Monash University
In software engineering visual analytics can help with both high-level understanding of connectivity between major components and also low-level dependencies involved in particular aspects of functionality or bugs. In this talk we discuss some
practical commercial tools that have been successfully deployed to support this kindof visual analysis but also consider what an immersive future may hold for software
engineers.

Situated Analytics
Bruce H. Thomas, University of South Australia
In this presentation explored the research question of how to support visual analytics reasoning by embedding the visual representations and interaction of the resulting
data in the physical environment using augmented reality. Situated analytics is this new
research area at the intersection of visual analytics and augmented reality.
9Multi-dimensional Informatics Platform for Genomic Research
Yang Cheng and Lingling Shen, NIBR Informatics, Novartis
Next-generation Sequencing (NGS) platforms play a prominent role in biologicalresearch. Large amounts of sequencing data generated from extensive R&D initiatives
and the demand for storage and analysis are expected to grow dramatically. However,turning this data in biological relevant and actionable results has not been as easy yet.
To use NGS at maximum compatibility, we must address IT tools of data management,
storage, and analysis. We have implemented several web-based applications from different angels to help researchers to track real-time business workflow and manage thelifecycle of a sequencing service request, easily conduct precise quality check on sequencing data and integrate heterogeneous data from different sequencing technologies, as well as facilitate their data mining effort to provide interactive visualizations.

Visual Analytics for (Bio)Medical Data
Oliver Kohlbacher, Universität Tübingen
Biomedical data has changed drastically in recent years. With the availability ofhigh-throughput image and molecular data (genomics, transcriptomics, proteomics,etc.), data analysis has become an indispensible component in biomedical research.
In some cases, visual analytics can support the data exploration process, however, careneeds to be taken how to employ these techniques to make optimal use of human cognitive capabilities.

Simulating a day in the medieval life of Angkor Wat
Tom Chandler, Monash University
Constructed in the 12th century, the world famous heritage site Angkor Wat is thelargest religious monument on earth. Recent archaeological estimates suggest that nine
centuries ago, the temple was supported by a population of up to 125,000 people, andstaffed by a regular workforce of 25,000. Using an extensive library of evidence based
3D models and immersive analytic overlays, this simulation explores how the complex
might have operated over a 24 hour cycle; a day in the life of medieval Angkor.

Acknowledgments The organizers would like to thank all participants, NII, and the
Shonan staff for all their contributions. We hope that the research during the seminar,
as well as the exchange between the participants, will lead to new successful collaborations and research findings in the future.

10