# Visualisation Immersive et Interactive de Données 3D
Temporelles

Gwendal Fouché

<figure><img src="image_1.png" /></figure>

To cite this version:

Gwendal Fouché. Visualisation Immersive et Interactive de Données 3D Temporelles. Synthèse d'image et réalité virtuelle [cs.GR]. INSA de Rennes, 2022. Français. NNT: 2022ISAR0035 . tel-04529934

HAL Id: tel-04529934

https://theses.hal.science/tel-04529934

Submitted on 2 Apr 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# THÈSE DE DOCTORAT DE

L'INSTITUT NATIONAL DES
SCIENCES APPLIQUÉES RENNES

ÉCOLE DOCTORALE N° 601

Mathématiques et Sciences et Technologies
de l'Information et de la Communication

Spécialité : Informatique

Par

Gwendal FOUCHÉ

Immersive and Interactive Visualization of Temporal 3D Data

Thèse présentée et soutenue à Rennes, le 05 Décembre 2022

Unité de recherche : Inria, Centre Inria de l'Université de Rennes (Inria-Rennes)

ISAR 34 / D22 - 34

## Rapporteurs avant soutenance :

Christophe HURTER Professeur, ENAC, Toulouse, France

Tobias ISENBERG Directeur de Recherche, Inria Saclay, Paris, France

## Composition du Jury :

Président : Emmanuel DUBOIS

Examineurs :
Nancy RODRIGUEZ
Christophe HURTER

Tobias ISENBERG
Directeur de Recherche, Inria Saclay, Paris, France

Dir. de thèse : Charles KERVRANN

Encadr. de thèse :
Ferran ARGELAGUET
Emmanuel FAURE

Professeur, Université de Toulouse, France

MCF, Univ. Montpellier, France
Professeur, ENAC, Toulouse, France

Directeur de Recherche, Inria Saclay, Paris, France
Directeur de Recherche, Inria, Rennes, France

CRCN Inria, Rennes, France
CRCN CNRS, LIRMM, Montpellier, France
<figure><img src="image_3.png" /></figure>
ACKNOWLEDGEMENT

Ainsi ces trois ans et quelques mois sont déjà passés. Trois ans à approfondir des problé-
matiques scientifiques et techniques avancées, mais aussi trois ans de progression personnelle
dans des domaines qui m'anime, et trois ans de travaux qui seront résumés dans ce manuscrit.
Bien sûr, ce parcours, ces travaux, je ne les ai pas faits seul, et je profite de cette préface pour
remercier tous ceux qui m'ont encadré et accompagné pendant cette longue, et pourtant si courte
période.

Tout d'abord, je souhaite remercier les membres du jury : Pr. Emmanuel Dubois, qui m'a fait l'honneur de présider le jury, Pr. Christophe Hurter et Dr. Tobias Isenberg pour avoir accepté le rôle de rapporteur, ainsi que Nancy Rodriguez pour avoir accepté le rôle d'examinatrice pour ma soutenance. Je vous remercie pour votre lecture attentive de mon manuscrit ainsi que vos retours et questions, intéressantes et pertinentes, auxquelles j'ai sincèrement apprécié être confronté lors de ma soutenance.

Je souhaite ensuite remercier chacun de mes encadrants. Merci Charles de m'avoir accordé
ta confiance pour travailler sur le projet Naviscope, qui fut pour moi une occasion de travailler à
l'intersection entre la réalité virtuelle et la biologie. Merci Emmanuel de m'avoir prêté ton logiciel
MorphoNet, grâce auquel nous avons pu prototyper et expérimenter chacune des méthodes
conçues lors de cette thèse. Merci à vous deux de m'avoir permis de travailler avec des biologistes,
pour apporter un sens plus concret à ces travaux. Enfin et surtout, merci Ferran. Du moment où
tu es venu me chercher dans une salle en haut de Supélec pour voir ce que je bricolais en projet
de fin d'étude sur de la réalité virtuelle, jusqu'à cette dernière répétition, le dimanche la veille de
la soutenance, ces réunions régulières aux discussions captivantes, ambitieuses ou rassurantes
quand il le fallait, tu as toujours fait preuve d'une très forte implication et de soutien dans mon
encadrement. Encore une fois, merci à vous trois, vous m'avez guidé tout au long de ce parcours,
vous avez été parfaits !

Mes remerciements s'adressent ensuite à l'ensemble des collaborateurs biologistes et bioinfor-
maticiens avec qui j'ai eu le plaisir de travailler. Tout d'abord, je tiens à remercier les chercheurs
du CRBM, en particulier Christophe Lemaire et Benjamin Gallean, qui m'a énormément aidé
lors des expériences. J'aimerais également exprimer ma gratitude envers l'équipe Salamero de
l'Institut Curie, composée de Jean Salamero, Cesar Augusto Valades et Ludovic Leconte, pour
leur apport scientifique ainsi que ces grandes réussites qu'ont été les journées de démonstrations de réalité virtuelle. Enfin, je souhaite remercier tous les autres collaborateurs qui ont participé aux expériences et apporté leurs retours pertinents.

Je remercie également mes collègues qui m'ont entouré pendant cette thèse, et m'ont également beaucoup soutenu par ce contact que nous avons pu garder pendant des années COVID particulières. Je remercie tout d'abord Anatole Lécuyer, chef de l'équipe Hybrid, acteur dans cette cohésion d'équipe tant professionnellement que relationnellement, et Nathalie Denis, notre assistante d'équipe, qui a parfaitement géré tous nos besoins pour les déplacements et rencontres avec toutes les personnes impliquées dans les travaux, répartis un peu partout en France. Je souhaite aussi remercier tout particulièrement mes deux collègues de bureau, Adrien Reuzeau et Alexandre Audinot, qui étaient présents pour m'aider à me former sur des points techniques poussés, ainsi que pour tous les moments de frustration que ça a pu engendrer. Je remercie aussi Thomas Howard, Víctor Rodrigo Mercado et Anthony Mirabile, avec qui nous avons pu faire preuve de beaucoup de créativité en de nombreuses occasions, et même parfois au travail ! A eux, ainsi que l'ensemble des collègues que j'ai pu rencontrer pendant ces trois ans, stagiaires, doctorants, ingénieurs, post-doc et permanent, à travers les réunions d'équipe et les discussions techniques, mais aussi à travers les pauses, les visios, les sorties, les soirées jeux, un grand merci à vous tous, je garde de précieux souvenirs de cette vie dans l'équipe.

Je tiens également à remercier ma famille, qui a été un pilier pour moi pendant ces trois années. Mes parents, qui m'ont permis de poursuivre de longues études dans d'excellentes conditions et ont été tout particulièrement présents durant ces années thèse. Je remercie ma sœur pour sa présence, notamment ces soirées jeux qui ont été une bouffée d'oxygène et une pause bienvenue ; mes frères, en particulier Gaëtan, qui a été d'un grand soutien moral, et Eymeric avec qui nous avons enchainé quelques solides marathons de gaming qui m'ont tout autant laissé de bons souvenirs et été des moments de détente. Je voudrais remercier ceux qui étaient plus éloignés mais dont je sais qu'ils ont pensé à moi, notamment ma marraine Marie-Laure. Merci d'avoir été là pour moi, comme vous l'avez toujours été !

Je remercie finalement tous mes amis, ceux proches que je voyais régulièrement pour des soirées de jeux et de détente, Lilian, Catherine, Marie-Christine, Didinski, Barnabé, Ludovic ; ceux plus éloignés mais quand même présents malgré cette distance, Nic-nic, Julien, Louis, Romain, Bélis, Son Dorian, Deli. Je te mets à part Vincent, le meilleur pour la fin, un grand, grand merci à toi, pour ta présence inconditionnelle, ces appels à des heures et aux durées indéterminées, bref, merci d'être la personne sur qui je peux compter depuis maintenant 13 ans...

Encore une fois, merci à vous tous !
# TABLE OF CONTENTS

## 1 Introduction 9

1.1 Why this thesis? 9

1.2 Context of the thesis 11

1.3 Research issues 12

1.4 Scientific Challenges 13

1.5 Approach and Contributions 14

## 2 Related Works 17

2.1 Definitions and Background 17

2.1.1 Visualization and Visual Analytics 18

2.1.2 Immersive Analytics: Virtual Reality and 3D Interaction 21

2.1.3 3D Temporal Data: Towards the Notion of 4D Objects 27

2.2 Visualization Techniques for Temporal Data 32

2.2.1 High Dimensional Data Visualization 32

2.2.2 Temporal Visualization: Dynamic vs Static Approaches 33

2.2.3 Space-Time Cube Visualization 35

2.2.4 Timeline Visualization 37

2.3 Immersive Visualization for Temporal Data 40

2.3.1 Immersion for Scientific Visualization 40

2.3.2 Immersion for Information Visualization 41

2.3.3 Collaborative Mixed Reality 43

2.4 Conclusion 48

## 3 Immersive and interactive visualization of 3D spatio-temporal data using a Space Time Hypercube applied to biological imaging data 49

3.1 Space-Time Hypercube Visualization 51

3.1.1 Generalizing the STC in 4D 51

3.1.2 Extracting a 3D STC from a 4D STH 52

3.1.3 Visualization: 3D Rendering and Quantitative Data 53
TABLE OF CONTENTS

<table><tbody><tr><td>3.1.4</td><td>Interacting with the STH and STC</td><td>55</td></tr><tr><td>3.1.5</td><td>Generation and Interaction Performance</td><td>59</td></tr><tr><td>3.2</td><td>Use Case: Application for Morphogenesis</td><td>59</td></tr><tr><td>3.2.1</td><td>VR Visualization Application</td><td>59</td></tr><tr><td>3.2.2</td><td>Use Cases</td><td>60</td></tr><tr><td>3.3</td><td>Evaluation</td><td>62</td></tr><tr><td>3.3.1</td><td>Tasks and Hypotheses</td><td>63</td></tr><tr><td>3.3.2</td><td>Apparatus and Participants</td><td>64</td></tr><tr><td>3.3.3</td><td>Experiment</td><td>65</td></tr><tr><td>3.3.4</td><td>Results</td><td>66</td></tr><tr><td>3.3.5</td><td>Discussion</td><td>67</td></tr><tr><td>3.4</td><td>Expert Feedback</td><td>68</td></tr><tr><td>3.5</td><td>Additional examples</td><td>70</td></tr><tr><td>3.6</td><td>Conclusion</td><td>70</td></tr><tr><td>4</td><td>Timeline Design Space for Immersive Exploration of Spatial 3D Temporal Data</td><td>73</td></tr><tr><td>4.1</td><td>A Timeline in a 3D Environment</td><td>74</td></tr><tr><td>4.1.1</td><td>Extending the 2D Timeline Design Space</td><td>75</td></tr><tr><td>4.1.2</td><td>Interacting with 3D Timelines</td><td>81</td></tr><tr><td>4.1.3</td><td>Choosing a Timeline Design</td><td>83</td></tr><tr><td>4.2</td><td>Evaluation</td><td>84</td></tr><tr><td>4.2.1</td><td>Dataset and Task Design</td><td>84</td></tr><tr><td>4.2.2</td><td>Hypotheses</td><td>85</td></tr><tr><td>4.2.3</td><td>Apparatus and Participants</td><td>86</td></tr><tr><td>4.2.4</td><td>Independent and Dependent Variables</td><td>86</td></tr><tr><td>4.2.5</td><td>Experimental Protocol</td><td>87</td></tr><tr><td>4.2.6</td><td>Results</td><td>87</td></tr><tr><td>4.2.7</td><td>Discussion</td><td>91</td></tr><tr><td>4.3</td><td>3D Timelines to Explore 3D Temporal Imaging</td><td>91</td></tr><tr><td>4.3.1</td><td>Application and Timeline Design</td><td>92</td></tr><tr><td>4.3.2</td><td>Finding Regions of Interests in Large Datasets</td><td>93</td></tr><tr><td>4.3.3</td><td>Comparing Objects</td><td>95</td></tr><tr><td>4.3.4</td><td>Fusion and Fission</td><td>95</td></tr><tr><td>4.3.5</td><td>Expert Feedback</td><td>96</td></tr></tbody></table>
<table><tr><td>4.4 Conclusion</td><td>97</td></tr></table>

5 Space-Then-Time: Toward Selection Methods for Punctual Events in 3D Temporal Data through 3D Interaction 99

<table><tr><td>5.1 Time-then-Space or Space-then-Time?</td><td>100</td></tr></table>

<table><tr><td>5.2 Visualizing the Right Data</td><td>100</td></tr></table>

<table><tr><td>5.3 Selecting 3D Information</td><td>102</td></tr></table>

<table><tr><td>5.4 Disambiguation in Time</td><td>103</td></tr></table>

<table><tr><td>5.5 Implementation Details</td><td>105</td></tr></table>

<table><tr><td>5.6 Conclusion</td><td>105</td></tr></table>

6 Conclusion 107

<table><tr><td>6.1 Contributions</td><td>107</td></tr></table>

<table><tr><td>6.2 Perspectives</td><td>109</td></tr></table>

<table><tr><td>6.2.1 Short-Term Perspectives</td><td>109</td></tr></table>

<table><tr><td>6.2.2 Long-term Perspectives</td><td>110</td></tr></table>

<table><tr><td>6.3 About Visualization in Bioinformatics</td><td>111</td></tr></table>

<table><tr><td>6.4 Concluding word</td><td>113</td></tr></table>

A Author's Publications 115

B Résumé en Français 117

Bibliography 137
<figure><img src="image_4.png" /></figure>
# INTRODUCTION

## 1.1 Why this thesis?

This thesis, entitled "Immersive and Interactive Visualization of Temporal 3D Data", presents the research contributions we proposed to tackle issues around immersion, visualization, and interaction with data from other scientific domains. In this first section, we introduce the main parts of this thesis title, provides with elements of definition for each of those terms and illustrate the inspiration that led us to explore the topic.

**Why 3D temporal data?** Datasets are growing in size and dimensions, and more and more analysts have difficulties to handle their data. A whole research field, the Big Data, notably explores methods for the analysis of datasets composed of huge amounts of data entries. We chose to focus on the emerging issue of 3D temporal datasets. 3D temporal data is getting adopted in numerous fields, as biological microscopy imagery, medical imagery or macro scale geographical information. As the technology to handle 3D temporal data is maturing, the number of existing datasets of this type is increasing, and the interest toward this topic with it. The size of the datasets, measurable sometimes in terabytes, can now be acquired, stored, and displayed with hardware and software techniques developed for the past few decades. These datasets are rich in information and could give a new point of view in many subjects we thought were mastered already. Yet, the development of methods to properly analyze such data is still quite a challenge.

**Why Visualization?** Sight is the most important sense, providing more than 80% of information perceived, according to Rosemblum [1]. People always want to see, as it is the easiest way to understand objects, mechanisms, phenomenon, basically most things happening and existing around us. Indeed, seeing is actually a complex task, involving observation, extraction of elements of interest, analyzing them, their physical characteristics, their activity, and their link to the surrounding context. Many games are designed around the difficulty of this task, from "Where's Waldo" to shooter games, or even when looking for a Lego piece in a huge box
(see Figure 1.1). Visualization is the domain that tries to formalize and tackle these everyday issues relying on vision, in the context of the analysis of data. As mentioned above, datasets are growing in size and complexity, so that analysts from various domains have trouble handling their data, ultimately creating new challenges in Visualization.

<figure><img src="image_5.png" /><figcaption>Figure 1.1 – The struggle of finding that one specific piece.</figcaption></figure>

**Why interaction?** Interaction constitutes all the actions of moving, grabbing or pushing around items. Such actions can support the observation. For instance, to find that missing Lego piece, people rummage through the box, until that specific piece is not hidden, in a position and orientation that your eye can easily discern. Interactions allow developing strategies to find relevant elements in a large set of information. Yet, seeing an element of interest is only the first step. Interaction also helps to understand the nature of this particular element. Turning it around to discern its shape, making it move, applying pressure, feeling the surface, throwing it around... So many actions that a baby would do when discovering a new toy. Therefore, interacting with the data is crucial to support the various steps of visualization, as much as it is natural.

**Why Immersion?** Immersion is the ability to adapt to a different environment than usual, with different rules and references. Humans adapt very fast to new environments, and notably simulated ones. The notions of presence, i.e. the feeling of “being there”, and embodiment, i.e. perceiving an avatar as oneself, are thoroughly studied with virtual environments. In the context of data analysis, immersion can help us understanding models of objects of very different scales, from a planetary system to entities moving in a cell. Technologies supporting immersive
technologies also matured a lot in the recent years. Head Mounted Displays are now of high
resolution, high frame rate and low cost. Such stereoscopic displays show two 2D images, one
for each of our eyes, which simulates the same “3D” we are used to see. Tracking devices follow
our head, hands, overall movements with high precision and low latency. Overall, immersion
simulates visuals in a similar way than with our eyes, and also natural and supernatural interaction.
This has potential to support visualization.

## 1.2 Context of the thesis

The fields of biology and medical imaging have been demonstrating significant improvements
in the recent years, especially around issue of acquisition and curation of high dimensional and
3D temporal datasets. Ground-breaking live imaging techniques such as the Selective Plane
Illumination Microscopy (SPIM) and Lattice Light Sheet Microscopy (LLSM) are now able to
create 3D temporal images at a rate and resolution sufficient to observe and detect important
temporal features, would it be general behaviors or punctual events. The automatic or semi-
automatic analysis of the data got a lot of attention in the respective communities, and methods
based on GPU computation or deep-learning algorithms emerged. These methods produce
additional information, such as segmentation, tracking, or numerical and categorical information
regarding the imaged entity, resulting in complex high dimensional 3D temporal datasets. Yet,
there is a lack in the detection of regions of interest and visualization in these complex data, such
that specialized scientists have difficulties exploring their images and can miss key information.
This thesis was written in the context of the Inria Challenge Naviscope. This project aims at
creating a comprehensive set of methods for the analysis of multidimensional and multi-valued
images, especially in the context of biological and medical imagery. Such methods include
techniques of navigation, manipulation and visualization of volumetric images, enable semi-
automatic analysis, and put a particular focus on the investigation of temporal features in the
data. A system implementing such techniques would be a tremendous support for scientists to
better understand the massive amounts of information and extract most relevant elements.

Similar challenges appear in various research domains, as high dimensional, 3D or 3D
temporal datasets are getting more and more common. The temporal dimension in the latter
introduces even more difficulties, in representation, display and interaction with the data. The
need for adapted tools helping analysts to understand and extract relevant information from these
datasets is becoming crucial. Among the research domains trying to tackle these issues, the field
of Visualization develops solutions to help users observe and explore the data. Such methods
rely on adapted visual representation, that emphasizing or filtering out some part of the data,
compromising between the quantity of information to display and the capacity of the analyst
to interpret it. Furthermore, the field of Visual Analytics attempts to complete Visualization
methods using concepts from adjacent domains of data science. It includes techniques from
data management to human-computer interaction, in order to give additional tools to analysts.
Immersive Analytics, another research domain deriving from Visual Analytics, tries to tackle
the same issues using immersive technologies. The methods from this field rely on stereoscopic
displays and high degrees-of-freedom viewpoint and devices to exploit efficiently large 3D
environments. Methods to enhance navigation in a 3D space, selection and manipulation of
objects in various types of 3D environments can come efficient in solving classic visualization
tasks. Regarding 3D temporal data, the size and number of dimensions, and the capacities
of interaction of users become a center of interest to approach new visualization challenges.
Therefore, Immersive Analytics can probably show its most potential.

**1.3 Research issues**

In order to approach the issues regarding the visualization of 3D temporal data introduced in
the previous section, we address three main research questions:

**Research Question 1: How to represent 3D temporal data?** The first question focuses on the representation of 3D temporal data, and especially its fourth dimension, time. Both the representations of time in lower dimension data [2–4] and of 3D data [5–8] are a well-explored challenges in the visualization community. Yet, as 3D temporal datasets are getting more and more common, abstractions required to properly define 3D temporal features are lacking. An approach to characterize and formalize the various 3D temporal features in necessary to support adapted visualization and interaction methods.

**Research Question 2: How to visualize different types of 3D temporal regions of interest?**

The visualization of 3D temporal datasets is limited first, by the number of dimensions available
to display the whole data, second, by the capacity of the user to interpret large number of
dimensions and information. 3D temporal data suffers from the issues of displaying both temporal
information and 3D data. On one hand, 3D data suffers from occlusion issues, and the density and
implied visual clutter further limits the human perception of the information. Such limitations call
for solutions including adequate rendering or navigation features. On the other hand, temporal
information representation often involves obstacles in displaying of the information, as the mapping costs an additional dimension. Adapted visualizations for 3D temporal data need either to compensate for these drawbacks or to compromise in an optimal way to highlight the main features of the dataset.

**Research Question 3: How to interact with different types of 3D temporal regions of interest?** Interaction is often reported as a key element to support the limitations of visualization. Various methods were developed to enhance the interaction in 3D environments and notably to explore 3D data. Interactive navigation in time is also recommended to compromise between using an additional Euclidean axis and the cognitive workload of animations. The selection, manipulation and exploration of 3D temporal regions of interest involves both of these constraints. By analogy with the limitations of human perception for visualization, human capacity needs to be taken into account for interaction, both in terms of physical capacity and precision, and of cognitive ability. Consequently, interaction must be adapted either to map efficiently and naturally the input of the user to an action on the 3D temporal region of interest, in order to facilitate its analysis.

## 1.4 Scientific Challenges

The research questions presented before leads to several challenges. We identified four challenges that needs to be tackled to answer our research questions.

**Challenge 1: Approaching various types of regions of interest** Depending on the entity observed or represented, the methods for generation or acquisition, 3D temporal data will present varied characteristics, which can require a potentially different approach for representation and visualization. Variations in the nature of the data (e.g. volumetric, point clouds, etc.), density or visual clutter can add obstacles to the visualization. On the other hand, regions of interest can be of different nature (e.g. trajectory, objects evolving over time), or have sizes of different orders of magnitudes. These differences in size are relevant not only in space, i.e. small vs large objects, but also in time, punctual vs extended events.

These variations in context and regions of interest will have impact on the perception and exploration of the data by the user. The design of visualization and interaction methods must take into account this obstacle.
**Challenge 2: Navigating in time** Time is often represented either spatially or temporally. Representing time as a Euclidean axis increases the complexity of the visualization, and is especially difficult for spatial 3D data. Controlled animations reduce the number of dimensions, mapping the temporal information on a “rescaled” time axis, yet it collides with the slogan “Eyes beats Memory”, i.e. the difficulty of the user to interpret and compare data that are not directly visible. Using any of these approaches, navigating in time and exploring temporal information remains challenging. First, interactions need to map the input of the user, usually defined spatially, to coherent actions on temporal information. Second, navigation methods should optimize access to relevant information.

**Challenge 3: Managing the view for visualization and interaction in 3D and time** View management is a classic issue when exploring 3D environments, and therefore when visualizing 3D data as well. First, it includes managing occlusion [9] and visual clutter, which are caused or increased with the density of the data. Second, the work space needs to be organized correctly, notably when using multiple juxtaposed views. Occlusion issues can notably be compensated with adequate manipulation and navigation methods. Yet again, when confronted with 3D temporal data, such issues tend to be even more prominent, notably depending on the method used to visualize the time dimension. As such, occlusion management will need particular attention when designing visualizations and interactions.

**Challenge 4: Encoding large high dimensional dataset** As mentioned previously, 3D temporal datasets often also include numerical and categorical dimensions. They can be crucial for the exploration and analysis of the data observed, through means of filtering and comparison. Therefore, these dimensions also need to be displayed with the visualization of the 3D temporal data. Several channels can be used to display such information. More commonly, the information is encoded on the color channel through color maps. Depending on the nature of the information, it can also be encoded through sound, haptic feedback, with labels or juxtaposed views. The encoding of the information must take into account the analytical purpose, but it also should avoid cluttering the visualization or involve too much mental workload for the user.

**1.5 Approach and Contributions**

As mentioned through the scientific challenges, the temporal dimension is a strong constraint for both visualization and human-computer interaction. Exploiting recent progression in those
two research domains, the field of Immersive Analytics leverages the benefits of immersive tech-
nology to support visualization and interaction. Issues regarding navigation in the temporal data,
observation of 3D data and overall view management could be facilitated by methods developed
in this field. As such, we decided to involve immersive environments in our visualization and
interaction approaches.

The strong variation in 3D temporal data highlighted in Challenge 1 led us to consider a
global approach of our main research questions through different types of data. As such, the
first step of our approach is the characterization of 3D temporal data and the features we will
be confronted with. Furthermore, the constraints in the representation of time will call for more
effort on either visualization or interaction. The remaining steps of our approach will try to tackle
the main research questions in these different situations. The outline of this thesis is as follows:

**Chapter 2** presents literature report describing the objectives and methods developed in the main research domains surrounding the thesis subject. We present first the domains of Visualization, Visual Analytics and Immersive Analytics, and approach the question of representation of 3D temporal data in data analysis context, with illustrations from the field of biology imaging. Then, we focus on methods to represent and visualize the temporal dimension in general data, focusing on specific static visualization, namely the Space-Time Cube and Timelines. Finally, we present applications and methods from Immersive Analytics to approach visualization issues, notably regarding high dimension data, and conclude with an opening of collaborative visualization methods.

**Chapter 3** describes the Space-Time Hypercube representation, which extends the classic Space-Time Cube representation to 3D temporal data. We propose a method create visualizations from the Space-Time Hypercube, defining extraction operators based on a cutting plane defined by the user. The resulting visualization is a Space-Time Cube displaying the evolution of a cut-away view through time, highlighting major events in the dataset or the general evolution of shapes or values. Classic 3D interaction tools come to support the exploration and analysis of the complex 3D structure. We evaluated this method through a user-study on a biological imaging dataset, and also reported feedback from domain experts who experimented with the method.

Following these evaluations, we conclude that this visualization method helps emphasize 3D
temporal features of significant spatial size and their evolution over large intervals of time.
**Chapter 4** presents a design space for the creation of 3D timelines visualization. This design space extends the classic visualization of timelines and explores ways to display them in a 3D environment. As such, 3D timelines can support the visualization of 3D temporal datasets. Leveraging the enhanced interaction and exploration of virtual environment, we developed interaction processes to exploit the 3D timeline. Using navigation, selection and filtering tools, the user can define and refine efficiently zones of interest in both space and time. A user-study was designed to assess the benefits of 3D timelines regarding usual visualization tasks using a procedurally generated dataset. We also report the feedback of biology experts, who tested the method on different types of biological imaging data.

The results of these evaluations led us to conclude that this visualization method, accompanied by the adequate interaction processes we proposed, helps define local regions of interest in both space and time, with short to medium temporal range.

**Chapter 5** explores a method of selection of events relying on a Space-Then-Time process. We focus on the selection of punctual events, that are challenging to access using traditional visualizations of temporal information. We propose that instead of exploring the temporal dimension, then the 3D space, to find and select a region of interest, an analyst could identify first the 3D location and then select the region of interest in time. We discuss an interaction process relying on three most important points: which information should be directly available to the user; how to select the region of interest in space; how to disambiguate the selection over time.

We suggest that an interaction method based on this concept could increase the efficiency of selection of 3D temporal features characterized by short temporal range.

**Chapter 6** concludes this manuscript, summarizing our contributions, as well as providing perspectives for future works on the methods we proposed, and on our propositions to the main research questions.
RELATED WORKS

<table><tr><td>2.1</td><td>Definitions and Background</td><td>17</td></tr><tr><td>2.1.1</td><td>Visualization and Visual Analytics</td><td>18</td></tr><tr><td>2.1.2</td><td>Immersive Analytics: Virtual Reality and 3D Interaction</td><td>21</td></tr><tr><td>2.1.3</td><td>3D Temporal Data: Towards the Notion of 4D Objects</td><td>27</td></tr><tr><td>2.2</td><td>Visualization Techniques for Temporal Data</td><td>32</td></tr><tr><td>2.2.1</td><td>High Dimensional Data Visualization</td><td>32</td></tr><tr><td>2.2.2</td><td>Temporal Visualization: Dynamic vs Static Approaches</td><td>33</td></tr><tr><td>2.2.3</td><td>Space-Time Cube Visualization</td><td>35</td></tr><tr><td>2.2.4</td><td>Timeline Visualization</td><td>37</td></tr><tr><td>2.3</td><td>Immersive Visualization for Temporal Data</td><td>40</td></tr><tr><td>2.3.1</td><td>Immersion for Scientific Visualization</td><td>40</td></tr><tr><td>2.3.2</td><td>Immersion for Information Visualization</td><td>41</td></tr><tr><td>2.3.3</td><td>Collaborative Mixed Reality</td><td>43</td></tr><tr><td>2.4</td><td>Conclusion</td><td>48</td></tr></table>

This chapter presents a literature review of the research carried in the main domains surrounding our research questions. This study focuses first on defining these adjacent domains and highlighting the main challenges they approach, in order to contextualize our own contributions. The issues tackled in such domains as Visualization, Visual Analytics and Immersive Analytics and the general approach proposed in each of these domains, will be a starting point to answer our research questions. We will also focus on the characterization of 3D temporal data, as it is an emerging issue, notably relying on previous works tackling 3D data and temporal data.

As we identified the issue of navigation in time as one of our major challenges, we then report visualization approaches to handle temporal data. Techniques to tackle the general issue
of high dimensional data were proposed, and more specific approaches refined these techniques
by exploiting time differently, as a dimension of unique nature. We compare those approaches
and identify methods that attracted our attention as bases to our contributions.

Finally, we explore how Immersive Analytics in the literature leveraged the benefits of immersive technologies to answer visualization challenges. We present approaches of Immersive Analytics application in different datasets and use cases, and conclude with collaborative immersive applications, a main concern in the domain of Immersive Analytics.

**2.1 Definitions and Background**

In this section, we introduce the main domains related to our research questions. First, we present the domains of Visualization and Visual Analytics, which are the main fields of research regarding the issues of exploration and analysis of data through human perception and interaction. Then, we focus on the domain of Immersive Analytics, which approaches the same issues relying on virtual environments and related techniques. Finally, we focus on 3D temporal data, describing their main characteristics and proposing an approach to represent 3D temporal features, illustrating it with biological imagery examples.

**2.1.1 Visualization and Visual Analytics**

**Definition of Visualization**

Visualization is a research domain that has been developed for the few past decades to innovate in the creation of views to analyze data in various fields such as medicine [10], biology [11], CAD [12] or physics [13]. Visualization is usually classified into two categories: *Scientific Visualization* [14], in which the data observed is usually scalar or vectorial fields explicitly referenced in time and space, and *Information Visualization* [15], which aims at representing abstract data. Munzner [16] proposed a definition for visualization, as follows:
Definition of Visualization

Computer-based **visualization** (*vis*) systems provide visual representation of datasets designed to help people carry out tasks more effectively.

Visualization is suitable when there is a need to augment human capabilities rather than replace people with computational decision-making methods. The design space of possible vis idioms is huge, and includes the considerations of both how to create and how to interact with visual representations. Vis design is full of trade-offs, and most possibilities in the design space are ineffective for a particular task, so validating the effectiveness of a design is both necessary and difficult. Vis designers must take into account three very different kinds of resource limitations: those of computers, of humans, and of displays. Vis usage can be analyzed in terms of why the user needs it, what data is shown, and how the idiom is designed.

This definition places the use of representation of the data as crucial for any analytical tasks and the human as a most important link in the decision-making process. It also introduces the main issues to compromise with when creating visualizations.

First, human limitations introduce an obstacle on representation. The first thing usually mentioned is the perceptual capacity. Ocular and cognitive characteristics require representations to emphasize patterns and features, as well as designing adapted color schemes, i.e. perceptually uniform, color maps [17–19]. The mental workload of the user is limited, thus the amount of information displayed must be limited accordingly. Physical characteristics also condition visualization. Human vision is stereoscopic, which directly induces issues on occlusion [9] and a need for depth cues. Furthermore, the precision, dexterity and sensation of the user's movement are key information to implement comfortable and efficient interaction on visualizations. Consequently, the second main limitations depend on the display, and more generally the Human-Machine Interface, characteristics. The most usual setup for visualization software is a screen, used with a mouse and a keyboard. This setup is very versatile and accessible to most users. Yet, screens confine the workspace to a 2D rectangle and the mouse is also restricted to 2D movements, which can be too restricting when visualizing large amounts of information or 3D data. Other interfaces explore different ways to display and interact with the data, notably with touchscreens or Mixed Reality (MR) setups. The latter allows the user to “see in 3D” through stereoscopic displays, and navigate and interact with the data through 3D interactions. Such technology, promising and scrutinized for visualization for decades now [20], leads to the last
main limitation: computing resources. Not only do algorithms must be adapted to the increasing
size of datasets, but also to the increase in the standard quality of the rendering. Furthermore,
Head-Mounted Displays (HMD) or CAVEs used in MR setup often require real time rendering
and interaction, constituting a potentially high computational cost as well.

**Definition of Visual Analytics**

Thomas and Cook [21] coined the term “visual analytics” and defined it as follows:

Definition of Visual Analytics

Visual analytics is the science of analytical reasoning facilitated by interactive visual interfaces. Visual analytics is a multidisciplinary field that includes the following focus areas:

- analytical reasoning techniques that let users obtain deep insights that directly support assessment, planning, and decision-making;

- visual representations and interaction techniques that exploit the human eye's broad bandwidth pathway into the mind to let users see, explore, and understand large amounts of information simultaneously;

- data representations and transformations that convert all types of conflicting and dynamic data in ways that support visualization and analysis; and

- techniques to support production, presentation, and dissemination of analytical results to communicate information in the appropriate context to a variety of audiences.

Keim et al. [22] gave additional insight on the objectives and components of visual analytics.
As depicted in Figure 2.1, Visual Analytics give an integral approach, incorporating elements
of Data Management, Data Analysis Technology, perception, cognition and human computer
interaction. Therefore, Keim et al. [23] proposed a mantra for Visual Analytics as follows:
“*Analyse First - Show the Important - Zoom, Filter and Analyse Further - Details on Demand*”,
in reference and extending the information seeking mantra presented by Shneiderman [24]
("*overview first, zoom/filter, details on demand*"). This ensemble needs to be supported by
adapted infrastructures and reliable evaluation in order to tackle issues that can be out of reach
for classic techniques of visualization.
<figure><img src="image_6.png" /><figcaption>Figure 2.1 – Visual Analytics integrates Visualization and core adjacent disciplines, and depends on appropriate infrastructure and evaluation; Keim et al. [22]</figcaption></figure>

As an illustration, we present a Visual Analytics system for analysis and visualization of biological imaging data, based around BioImageIT by Prigent et al. [25]. The system, as shown in Figure 2.2, integrates at its core tools for data management and data analysis. BioImageIT can be linked with multiple other existing systems of Data Management, such as OMERO [26], which proposes its own data formatting and hosting servers. Data Analysis plugins can be coded directly into the system, or added via Docker containers or Conda packages, allowing primarily easy communication between researchers, but also adaptability against the evolution in this field. Finally, the system uses visualization tools such as Napari [27] and ImageJ/Fiji [28, 29] to render the processed data. To summarize, the user can use BioImageIT system interface to manage their data, create a processing pipeline, and use their usual visualization tool to observe and interact with the resulting data.

Visual Analytics widen the scope of Visualization to attempt at improving the existing methods, in order to give more analytical tools to the user and ways to handle more diverse datasets. As such, Visual Analytics can also provide solutions for the main challenges and objectives of visualization. Kehrer and Hauser [30] notably reported the contribution of Visual Analytics to overcome the issue of scalability of systems with the increasing size, complexity and heterogeneity of data to handle, while maintaining possible interaction or acceptable processing time. Yet, as datasets still keep growing, the visualization of high dimensional data remains one of the main challenges of the research field.
<figure><img src="image_7.png" /><figcaption>Figure 2.2 – Schematic view of BioImageIT architecture; Prigent et al. [25]</figcaption></figure>

## 2.1.2 Immersive Analytics: Virtual Reality and 3D Interaction

### On Immersive Analytics

Mixed Reality technologies has already been used for visualization in several scientific fields for many years. It was used in archaeology [20, 31, 32], brain tumor analysis [33] or physics [34], giving mostly satisfying results. Experiments conducted by Bach et al. [35] made to evaluate the contribution of Virtual Reality in terms of interaction suggested that highly interactive tasks were achieved more intuitively in immersive environments. It was also noticed in other experiments at Carnegie Mellon University and the University of Virginia [20] that mistakes made in immersive environment, as distance evaluation and such, corresponded to real world mistakes, and that real world usual tasks such as rotating objects were achieved faster and with more precision [20]. Another study by Raja et al. [36] estimated that immersive environment and head tracking gave better results in terms of task performance for 3D scatter plots visualization.

The term of “Immersive Analytics” was defined in 2015 by Chandler et al. [37], as an emerging research topic stating as follows:
## Definition of Immersive Analytics

Immersive Analytics investigates how new interaction and display technologies can be used to support analytical reasoning and decision-making. The aim is to provide multi-sensory interfaces for analytics approaches that support collaboration and allow users to immerse themselves in their data. Immersive Analytics builds on technologies such as large touch surfaces, immersive virtual and augmented reality environments, haptic and audio displays and modern fabrication techniques.

As underlined out by the same author, Visual Analytics is agnostic of the interface used. As such, Immersive Analytics are a sub-domain of Visual Analytics, and share the same objectives and methods of enhancing visualization to support analytical tasks. The particularity of this recent domain is that it also integrates the methods developed around immersive interfaces. First, immersive interfaces usually provide with an additional visual channel through stereoscopic 3D rendering. This additional channel assists in the perception of various cues, notably for depth perception, which can help in visualizing 3D data rendering [38]. Second, immersive environments also provides with large work space, notably allowing the multiplication of juxtaposed views in an application. Finally, 3D user interaction allows efficient operations of manipulation, selection or application control [39], which could enhance an analytical workflow.

## Perceptual Benefits of Immersion

Most information received by human perception comes from the visual channel [1]. Humans are very good at identifying what they see, finding patterns or emergent properties. Stereoscopic displays, such as HMDs or CAVEs, usually provides a wider field of view than classic 2D displays; the information in the peripheral vision provides further awareness and context of the VE to the user [20]. Experiments described by van Dam et al. [20] showed that most human-scaled problems, such as complex structures exploration, are more easily apprehended in immersive environment, where the body can move in the environment, rather than on a 2D interface where the environment rotates around the user. Non-human-scale problems are often not approached naturally by the user, yet they can be rescaled to correspond to a more usual viewpoint. For example, van Dam et al. [20] describe an arterial blood flow visualization represented as a pipe exploration, a more familiar macro scale object for the user. Furthermore, Andrews et al. introduced the notion of “Space to Think” [40, 41], referring to and demonstrating that interacting with large displays could support the cognitive process and reduce the mental workload, notably
by spatially organizing the information material through adapted interaction. This approach was
later explored for 3D environments by Bandyopadhyay et al. [42], introducing the “Immersive
Space to Think”. More works on this approach by Lisle et al. [43, 44] showing the benefits and
proposing strategies to approach such environments to help understand large datasets.

As such, 3D displays have been used as general HCIs for 3D environments, for scientific visualization and for medical research. McIntire et al. [45] review 184 experiments based on 3D displays; they report that about 60% of the experiment they reviewed found positive effect - and 15% mitigated results - of the use of the display for usual analytical tasks such as manipulation, distance estimation or object identification. They observed that 3D display was beneficial in cases of either complex, difficult depth-related tasks, or when monocular depth cues were lacking. This statement is further supported by Marriott et al. [38], who discussed how various 3D displays from the literature was beneficial for depth perception of 3D environments. Figure 2.3 summarize the existing types of 3D displays against strong perceptual cues such as linear perspective, i.e. relative size of objects, occlusion or motion perspective. This was also confirmed in Laha et al. [46] study, which found a positive effect of head tracking and stereoscopic rendering for tasks involving visually and spatially complex search. This study also reports a significant effect of large fields of view when visualizing volumetric data, concurring with van Dam et al. [20] suggestion.

3D environments and stereoscopic technologies provide perceptual cues that help understand-
ing of complex 3D shapes and achieving depth-related tasks, which is particularly relevant for
the visualization of 3D data. To support analytical tasks, navigation and interaction methods
were developed to exploit 3D environments more efficiently.

3D Interaction for Analytical Visualization

3D interactions encompass the methods that allow users to interact with a 3D environment. A 3D interaction design can be approached using generic interaction techniques, easy to choose and implement thanks to widely available design spaces, evaluation results, by decomposing a task into canonical subtasks [39]. However, these approaches rarely take into account domain-specific knowledge. As such, the design of application-specific 3D interactions is a usual approach to improve the usability of an application [47]. Nonetheless, such approach implies an additional cost to design the interaction, based on a less well-defined design space and with little re-usability.

3D interaction techniques are usually separated into Selection, Manipulation, Navigation and Application Control [39]. Selection and manipulation are natural interactions in 3D environment, and the quality of these fundamental interactions have a tremendous impact on the quality of
<table><caption>Figure 2.3 – Mapping 3D display technologies and depth cues (“Y”: yes; “P”: possible; “D”: depends, to some extent; “N”: no); Marriott et al. [38].</caption><thead><tr><th></th><th>Linear Perspective</th><th>Aerial Perspective</th><th>Occlusion</th><th>Motion Perspective</th><th>Accommodation</th><th>Convergence</th><th>Binoc. Disparity and Stereopsis</th><th>User-controlled PoV</th><th>Subjective Motion</th><th>Interactive Content Manipulation</th></tr></thead><tbody><tr><td>Regular photography or print</td><td>Y</td><td>Y</td><td>Y</td><td>N</td><td>N</td><td>N</td><td>N</td><td>N</td><td>N</td><td>N</td></tr><tr><td>Desktop Computer Virtual Reality</td><td>Y</td><td>P</td><td>Y</td><td>Y</td><td>N</td><td>D</td><td>D</td><td>Y</td><td>N</td><td>D</td></tr><tr><td>Fishtank Virtual Reality</td><td>Y</td><td>P</td><td>Y</td><td>Y</td><td>N</td><td>D</td><td>D</td><td>P</td><td>Y</td><td>D</td></tr><tr><td>Non-disparity monocular/binocular viewing</td><td>Y</td><td>P</td><td>P</td><td>P</td><td>N</td><td>N</td><td>N</td><td>P</td><td>P</td><td>D</td></tr><tr><td>Head-mounted Binocular Displays</td><td>Y</td><td>P</td><td>Y</td><td>Y</td><td>N</td><td>Y</td><td>Y</td><td>P</td><td>Y</td><td>D</td></tr><tr><td>Multi-display Environments, Large Displays</td><td>Y</td><td>P</td><td>Y</td><td>Y</td><td>N</td><td>N</td><td>N</td><td>P</td><td>P</td><td>D</td></tr><tr><td>Binocular CAVEs</td><td>Y</td><td>P</td><td>Y</td><td>Y</td><td>N</td><td>Y</td><td>Y</td><td>P</td><td>Y</td><td>D</td></tr><tr><td>Gazer (Simulation of Accommodation)</td><td>Y</td><td>P</td><td>P</td><td>P</td><td>D</td><td>P</td><td>P</td><td>P</td><td>P</td><td>D</td></tr><tr><td>Accommodation Optics VR Headset</td><td>Y</td><td>P</td><td>P</td><td>P</td><td>Y</td><td>Y</td><td>Y</td><td>P</td><td>Y</td><td>D</td></tr><tr><td>Multiview Autostereoscopic</td><td>Y</td><td>P</td><td>Y</td><td>Y</td><td>D</td><td>Y</td><td>Y</td><td>P</td><td>Y</td><td>D</td></tr><tr><td>Volumetric 3D Displays</td><td>Y</td><td>N</td><td>N</td><td>Y</td><td>Y</td><td>Y</td><td>Y</td><td>P</td><td>Y</td><td>D</td></tr><tr><td>Optical Holographic 3D Displays</td><td>Y</td><td>N</td><td>N</td><td>Y</td><td>Y</td><td>Y</td><td>Y</td><td>P</td><td>Y</td><td>D</td></tr><tr><td>Augmented Reality (AR)</td><td>Y</td><td>P</td><td>D</td><td>Y</td><td>D</td><td>D</td><td>Y</td><td>N</td><td>Y</td><td>D</td></tr><tr><td>AR Hybrids</td><td>Y</td><td>P</td><td>Y</td><td>Y</td><td>D</td><td>D</td><td>Y</td><td>D</td><td>P</td><td>D</td></tr><tr><td>Physical Visualisations (reality)</td><td>Y</td><td>P</td><td>Y</td><td>Y</td><td>Y</td><td>Y</td><td>Y</td><td>N</td><td>Y</td><td>Y</td></tr></tbody></table>

the whole 3D user interface. Selection techniques use different metaphors [48], such as a virtual hand, mostly for short-range or realistic interactions, or virtual pointing techniques for mid to long range interaction. The latter are notably differentiated by the number of degrees of freedom (DoFs) involved in the interaction, as illustrated in Figure 2.4. Lower DoFs usually makes for interactions that are easier to control [39, 48]. Volumetric pointers can be used to select small objects more easily, yet in dense environment, it can create an ambiguity on the selected object, which needs to be answered with adequate and usable disambiguation methods [49]. Vanacken et al. [50] evaluated other techniques supporting object selection in dense or occluded environment, notably advising for multimodal selection feedback. Manipulation tasks relate to the application of rigid transformations to 3D objects [51]. Direct manipulation approaches
involving one or two-handed interactions simulate natural interactions, with techniques as the
Handlebar metaphor [52], or relying on widgets [53]. Indirect manipulation techniques allow the
user to manipulate object at a distance, with straightforward approaches as HOMER [54], or with
metaphors like the World-in-miniature approach [55]. For selection and manipulation techniques,
the control-display ratio is sometimes altered, resulting in an anisomorphic control mapping,
to compromise differently the strength and weaknesses of the interactions. For example, the
Go-Go selection [56] applies a gain to the distance between the user and their hand, increasing
the range but decreasing precision; the PRISM technique [57] applies a gain to the hand speed to
filter out noise and increase precision on slow movements. Finally, navigation techniques define
how the user can move in space. These techniques are usually separated into Physical travel and
Virtual travel. Physical travel involves movement directly from the users, such as simply walking,
redirected walking or walking in place techniques. These methods require more or less physical
space, and can be tiring for extensive use. However, they also provoke less motion sickness, as
real movements from the user trigger the virtual movements. Virtual travel uses controller inputs,
for example through steering techniques or teleportation. If these methods are less constraining
in terms of physical space, they do not provide vestibular cues and can be perceived as less
immersive.

<figure><img src="image_8.png" /><figcaption>Figure 2.4 – Virtual pointing techniques: (a) raycasting, involving 2+3 DoFs, (b) occlusion selection, involving 3 DoFs and (c) raycasting from the eye involving only 2 DoFs; *Argelaguet and Andujar* [48].</figcaption></figure>

Fonnet and Prié [58] report an overview of interaction tasks for Immersive Analytics. While still relying on Selection, Manipulation and Navigation general tasks, they notably separated the task of Application Control between several Visualization tasks. The main tasks explored in the literature are as follows:
**Details on Demand:** Interactions which goal is to display more complete or additional information about some data. This information is commonly displayed through a pop-up window upon selection [59, 60], yet also through labels appearing predictively depending on the application context [61], or using side screen that displays information [62, 63].

**Arrange:** This task consist in changing the spatial organization of data points, views or view components. This interaction relies either on manipulation of the mentioned elements, or through automatic methods, like force-directed placement [64].

**Change:** This regroups the interactions which goal is to modify the representation or visual encoding of data points or other view components, through ways of highlighting or changing attribute mapping.

**Filter:** This task, classic in visualization applications [16], consists in the determination of exclusion or inclusion criteria, in order to remove elements and simplify the view.

The remaining tasks concern Aggregation, Annotation, Import, Derive and Record, and are way less explored in the literature. Fonnet and Prié [58] argued that these overlooked tasks will nonetheless need to be developed in order to implement complete data-analysis systems.

### 2.1.3 3D Temporal Data: Towards the Notion of 4D Objects

Spatial 3D temporal data are more and more present in scientific visualization. These data are of various nature, notably point clouds [65, 66], more [67–70] or less [71, 72] large trajectory datasets, simulated flow data [73, 74] and surface or volumetric datasets [75]. Numerous methods have been developed to represent the time component and space-time relationships [3, 76]. Yet, the representation of spatial 3D temporal features remains challenging. In this section, we explore the main characteristics of 3D temporal data and discuss a representation of 3D temporal features, namely 4D objects, and provide with examples from the biology imagery domain.

#### 3D Temporal Features and 4D Objects

We approach the issue of representation of 3D temporal features as a way to partition logically the data. First, we consider 3D objects as represented in the 3D interaction literature. Overall, 3D objects in 3D environments are defined in a really transparent manner, i.e. with virtual objects that represent their real counterpart. The selection methods presented in the previous section highlighted different use cases related to the type of objects, yet also considered the characteristics of the 3D environment. Such factors as the density of objects in the environment,
the size of each object and their position compared to the user are determining in the choice of
technique used. Manipulation methods will be also potentially constrained by the shape of the
object.

We define 3D features in a context of data analysis by analogy of 3D objects. As such, like an actual object, a 3D feature must be a continuous volume, basically an independent semantic entity. As 3D objects, it will be notably characterized by its size, shape and position of data pieces represented in space. The encoding of the object, i.e. volumetric, vector-based, mesh-based, point clouds, or defined by functions, and the respective precision of the encoding (i.e. resolution, number of vertices, ...) is likewise a salient characteristic in the representation of the object. Moreover, additional categorical and numerical characteristics are usually attributed to 3D objects, such as a name, a color, a weight, or physical properties for example. Again, by analogy, such additional information, often present in high dimension datasets, can be attributed directly to a defined 3D feature. It can come from annotation,

Finally, we need to integrate the temporal dimension to these features. The first evident
property is the lifespan of the feature, i.e. when it appears and disappears, and the duration in
between. Corresponding to the matter of the encoding of the spatial 3D object, time can be
defined as continuous or discrete, and the resolution in time is also an important property of this
dimension. Furthermore, each of the characteristics evoked previously can evolve along time,
constituting the dynamics of the objects.

Eventually, we introduce the notion of 4D objects to compile all the characteristics identified.
To summarize, we attribute several features to qualify 4D objects:

- A 4D object is an independent semantic entity.

- A 4D object is defined over a continuous subset of the time dimension encoded in the dataset. Thus, it has a lifespan, a date of appearance, of disappearance and is defined either continuously in time or discretely with a temporal resolution.

- A 4D object contains a continuous subset of the 3D space encoded in the dataset, and thus has a shape, size and position. These characteristics can evolve over the temporal dimension.

- A 4D object can be attributed several categorical or numerical information, such as a name, color or weight. These characteristics can evolve over the temporal dimension.

This representation should support logic visualization and interaction while considering all
the information of a 3D temporal feature. In the following, we illustrate how this representation
can be integrated in concrete examples.
## Examples: 3D Temporal Bioimaging Datasets

In this section, we present several datasets from live microscopy, describing the context of their study and how our 4D object approach can be applied to the analysis of these data. These datasets will serve as examples in the contributions presented in this thesis.

<figure><img src="image_9.png" /><figcaption>Figure 2.5 – Imaging of an embryo of Phallusia Mammillata, time points 1 and 75 i.e. nearly two hours of recording; data from Guignard et al. [77].</figcaption></figure>

The first dataset, illustrated in Figure 2.5 is a live recording of the embryonic development of a Phallusia Mammillata, a marine invertebrate animal, more specifically a tunicate of the ascidian class. The ascidian embryos are characterized by their fast development and low number of cells (a few hundreds). The Phallusia Mammillata also presents particularly transparent membranes and a lack of apoptosis, i.e. programmed cell death, and cell migration in its early development. The main events happening during the development will be focused on asymmetric and/or asynchronous cell divisions. The acquisition of this dataset was made using confocal multiview light-sheet microscopy [77]. It generated 180 3D images, taken once every 89 seconds. The output data, a few terabytes and 4D with isotropic spatial resolution, was then segmented using the ASTEC pipeline (Adaptive Segmentation and Tracking of Embryonic Cells) [77] and meshed using the VTK library [78] and MeshLab [79]. It resulted in a surface-based spatio-temporal dataset of a few hundred megabytes. Additional categorical and numerical data are also available: the ANISEED database [80] provides us with information about the expression of various genes in the cells; the community of scientists added data such as cell volume; we were also able to compute locally information such as remaining lifespan before division. In addition to the 3D
segmentation and meshing, the ASTEC pipeline extracted the lineage of the cells.

In this dataset, we represent each cell as a 4D object, represented by a mesh, identified by a name and related to categorical and numerical information, with tracking over time. It is to be noted that temporal relationships exist between the 4D objects, as cell division creates two new cells.

<figure><img src="image_10.png" /><figcaption>Figure 2.6 – Imaging of a cell with tracking of mitochondria and cd63 endosomes; data from Valades et al. [81].</figcaption></figure>

The second dataset we study is a recording of a live cell with marking of mitochondria and CD63-positive late endosomes [81], illustrated in Figure 2.6. CD63 is a protein usually associated to the surface of the cell membrane or present in late endosomes. It is notably involved in cell activation, adhesion and differentiation, and its activity seems to have inverse correlation with late stages of cancer development [82]. This STED microscopy recording was
acquired using a Lattice Light Sheet Microscope [83] (LLSM), using CD63-Turquoise.2 labeling late-multivesicular endosomes and PKMOrange [84] labeling mitochondria. A hundred 3D images were acquired, one per second, with a spatial resolution of 104 x 104 x 325 nm per pixel, resulting in 495 x 500 x 60 pixels volumetric images. They were then segmented using the software Imaris [85], retaining endosomes and mitochondria information. The endosome tracking was computed with Trackmate [86]. Some additional information could be computed, such as the volume of the endosomes and the distance between the endosomes and the mitochondria.

In this dataset, we represent endosomes as 4D objects, as they are segmented, tracked in time and have additional information directly related to them individually. Mitochondria are too difficult to track because of the resolution of the data, so they are kept as background contextual information.

The third dataset we focus on, illustrated in Figure 2.7, is a simulation of astrocytic calcium dynamics. Astrocytes regulate neuronal information processing through a variety of spatio-temporal calcium signals [87]. This dataset simulates calcium signals over a real 3D temporal LLSM recording of an astrocyte. These signals are classified into blips, of short duration and low amplitude, puffs, of higher amplitude and duration, or waves, similar to puffs but moving in space. Nonetheless, calcium signals of all classes appear over only a few time points at most. The dataset is constituted of a hundred volumetric images of resolution 170 x 512 x 48. The calcium signals are generated separately, and thus are directly segmented and tracked in time. Consequently, we can represent the calcium signals as 4D objects.

These datasets all encode spatial 3D temporal information, completed with categorical and numerical dimensions. The features represented are of different nature and size, both in time and space. Yet, the 4D objects representation generalizes these different features, allowing us to consider them as equivalent for visualization or interaction purposes.

## 2.2 Visualization Techniques for Temporal Data

The representation of temporal information is an issue that goes way further than just scientific visualization. Visual storytelling, from caveman paintings to comics and other artistic domains, or visual explanations, like manuals or music sheets, required to encode time. Classic methods used juxtaposition of multiple scenes, canvases or symbols, relying on the usual reading direction of the targeted audience. Movements are particularly difficult to represent, using techniques such
<figure><img src="image_11.png" /><figcaption>Figure 2.7 – Imaging of an astrocyte, with simulated calcium signals (in color, different for each signal); data from Badoual et al. [87].</figcaption></figure>

as motion lines in comics, or overlaying multiple steps of the movement.

Nonetheless, temporal data visualization is one of the main challenges of Visualization and Visual Analytics. In this section, we present the approaches taken to visualize high dimension data, then focus on methods to visualize temporal information. Finally, we detail specific examples of temporal visualization techniques that drew our attention according to our main research issues, namely the Space-Time Cube and the Timeline visualizations.

2.2.1 High Dimensional Data Visualization

High dimensional data poses a challenging visualization problem not only due to the raw
data size but also due to the heterogeneity of the different dimensions constituting these datasets.
These dimensions are spatially and temporally referenced, represent numerical or categorical
information, and come from various sources, such as imaging, simulations or annotation pro-
cesses. Therefore, there is a large diversity in datasets characteristics and topology; ensues a
visualization challenge both to represent and display such amount of information. Andrienko
et al. [2] presented various approaches to handle this challenge: data aggregation and more
generally dimensionality reduction; semi or fully automatic feature extraction; and juxtaposition.

First, dimensionality reduction approaches focus on extracting and displaying the most
relevant information with a lower-dimensional representation in order to make it easier to
apprehend or to help in classification tasks. For example, Demšar and Virrantaus [88] visualize
vessels trajectory sets by aggregating and summarizing the data as a volume by computing the
density of movement, as shown in Figure 2.8. The result is thus less cluttered and more legible.
Woodring and Shen [89] proposed a general visualization scheme allowing the projection of
several variables, combined with different volume operations into a 3D volume. Dimensionality
reduction can also rely on linear transformations. Principal component analysis is a classic
example in the analysis of multidimensional points clouds. Derived algorithms [90, 91] further
improved the method, better conserving the variance of the original data. Other methods use
non-linear transformations to reduce dimensions, such as methods based on Fisher Linear
Discriminant Analysis [92] or the locally linear embedding [93].

The second approach consists in extracting specific features or patterns from the data. This
can result in abstract representations that may complete the analysis on a direct depiction of
the data. For example, TransGraph [94] extracts transitions in features or data items in time-
varying volumetric data to produce a graph in addition to the 3D rendering. Finally, juxtaposition
helps represent and explore high dimensional data by displaying different views and projection
of the data separately. Regarding this topic, Munzner [16] proposed four design choices for
coordinated juxtaposed views. The different models were characterized by whether the encoding
used is different or not, and whether the data displayed in each view is complete, a sub-set
or a partition. For example, Duran et al. [95] proposed a system to visualize a 3D molecule
simulation, juxtaposing the 3D rendering with linear information on several objects. Al-maneea
and Roberts [96] notably reported that more than half visualization systems, among the 491
reviewed, had 2 to 5 views Respectively in accordance with the rules of Diversity and Consistency
proposed by Baldonado et al. [97], juxtaposing and coordinating several dimensionality-reduced
views is a reliable option to visualize high dimensional data.

High dimensional data are diverse and their visualization face many issues. Nonetheless,
higher dimensional data also includes temporal data, and as such, the visualization methods
adapted to the latter can be analog.

2.2.2 Temporal Visualization: Dynamic vs Static Approaches

Time-oriented data is a subset of high dimensional data, yet the temporal dimension needs
to be modeled particularly, according to the understanding and perception of time by the user
in the context of the dataset visualized. Two main categories of approaches exist to visualize
time-varying data: dynamic methods, i.e. animation and interaction, and static visualizations, of
either the full data or extracted lower-dimension data.

**Dynamic Visualizations.** Animation is an intuitive method to explore time evolution in
<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /><figcaption>Figure 2.8 – Trajectories of tankers represented in a space-time cube; Demšar et al. [88]. Representation through (a) direct rendering; (b) density with isosurface; (c) density with volume rendering.</figcaption></figure>

3D temporal datasets. In designs using animation, time increases automatically, showing the evolution of the data through time. The work of Coffey et al. [98] showed that animated and interactive design choices for time exploration can decrease the amount of error in analysis tasks. They also suggested that hybrid methods that would implement interactive, animated and/or static time exploration designs could compensate for the flaws of other methods. For instance, Akiba et al. [99] proposed an interactive animation interface. Using their tool Aniviz, the user was able to explore time-varying volumetric data by editing a time-dependent transfer function, specifying the evolution of parameters and how the data should overall be displayed and animated. However, animation can be less adequate for comparison tasks [75] or the analysis of space-time-value relationships [89]. There is a higher cognitive load for the user who has to remember the state of the data between different moments, which can limit the observation of details, as the slogan “Eyes Beat Memory” suggests. Finally, Solteszova et al. [100] proposed to use time-warping in animations to cope with this issue. Their method allows the user to manipulate time evolution of a video or a selected point of interest in the video, to allow longer and more precise observation of an event. Yet, a more intuitive solution for the fore-mentioned issue lies in static visualizations.

**Static Visualizations.** Static visualizations aim at including the temporal information directly into the visualization. This additional dimension can make direct rendering difficult, non-intuitive or even impossible, yet it can make the information reachable by the user. The approaches to solve this issue are mostly analog to the methods described for high dimensional data. Woodring and Shen [89], mentioned previously, proposed a solution for static temporal visualization for time-varying volumetric data relying on dimension reduction. Set and numerical operations, such as the definition of a time window, unions or intersections, were used to determine which data to display in the volume. Such operations aimed to reduce occlusion and focus on the points of interest. In addition, a color mapping can be applied to overlay chronological information [89].
With this method, the temporal evolution is visualized by merging the spatio-temporal data into a volume. Although, there is a compromise to make between occlusion and the amount of displayed data, it is less suited for dense spatio-temporal data. Alternatively, in order to reduce the data to display, regions of interest can be defined either manually or automatically. For example, Lu and Shen [66] used a dissimilarity measure to extract single time steps in the volumetric temporal data. The final layout displays snapshots of the data at time steps estimated as most important for understanding the data evolution. Those time steps are then linked with color mapped timeline, keeping temporal information. MotionRugs [101] is another dimension reduction based method, producing a 2D visualization by ordering objects moving in a 2D space on a 1-dimension vector using a space filling Curve. The rendered image makes patterns appear that matches trends in the movement of the objects. TransGraph, a tool developed by Gu and Wang [94], takes the approach of feature extraction, creating a graph of points of interest extracted from blocks of volumetric data. The resulting graph is juxtaposed with the 3D rendered volumetric data. Correspondences between nodes and volume regions are handled by a brushing and linking method. This allows the user to apprehend the evolution of 3D temporal data over time through a 2D interactive graph representation.

Overall, these two categories balance differently the user's capacity in terms of memory, perception and interaction with the data they will be presented. As we expected the memory workload to be too high in the case of high dimensional and temporal data, we focused on static visualizations.

### 2.2.3 Space-Time Cube Visualization

First introduced by Hägerstrand [102] as time-space volume and paths in a context of socio-economic study, the Space-Time Cube (STC) is a representation using two axes for data and a third axis for time, as shown in Figure 2.9-a. It is used for numerous representations of time-varying data, would it be for geometrical illustration, geographic data [103, 104], trajectories[105] or videos [106]; the latter is illustrated in Figure 2.9-b. Bach et al. [107] summarized, in their review of temporal visualizations based on the STC, different types of operations that can be applied to visualize data in a STC. Cutting operations allow the extraction of an image at a particular moment in time, as short exposure photography, or a particular plane in the data space, a slice of the cube. On contrary, flattening operations collapse the STC along an axis to obtain a 2D representation, when the time is collapsed, it can be depicted as a long exposure photograph. In addition, depending on the data and the visualization design, 3D rendering, interpolation, volume extraction, non-orthogonal or non-planar operations can also be used. Later work by the
same authors [108] generalized the notion of STC visualization, modelling various visualization methods as associations of operations on a STC.

Among the topics emerging from STC visualizations, two kept our attention. First, **higher dimensions** in the data, such as 3D temporal data, can be visualized as a higher dimension Space-Time Hypercube. In the case of spatio-temporal data, operations of cutting, corresponding to a projection of the 4D data, could yield a visualizable 3D image. If the idea of a higher dimension STC was evoked in Bach et al. [107, 108] reviews, this lead has not been thoroughly explored. Among the works that are the closest to it, the Woodring and Shen's [89] method described previously already evoked a time flattening operation on a STH; in this direction, Woodring et al. [109] proposed a method to generate and render 3D slices based on an arbitrary hyperplane equation. In the latter contribution, the authors gave a few guidelines to interpret the output visualization with particular value of the hyperplane equation. However, the intuitiveness of the equation parameterization and concrete interpretation of the visualization have not been formally evaluated. Finally, another method was proposed by Krone et al. [110] in which the focus was on the evolution of the surface of molecular membranes. In this scenario, surface data for each time step was projected into a plane, enabling the visualization of the evolution of the membrane in a STC.

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /><figcaption>Figure 2.9 – Examples of STCs from the literature. Figure (a) is STC representation of social interactions by Hägerstrand [102]; Figure (b) illustrates the Khronos Projector [106], here representing a video of a scene transitioning from day to night into a STC.</figcaption></figure>

The second emerging topic from STC visualizations is **interactivity**. If Bach et al. [108] advice against operations of 3D rendering of STCs, it seems that interaction and navigation
methods, notably with immersive technologies [35] could help in solving some of the difficulties evoked [108], such as occlusion, depth ambiguity and perspective distortion. This would help take advantage of the general overview of the data that this operation can offer. STC visualization can also take advantage to direct interactions in immersive environments. For example, the GeoGate system [103] displays geographic trajectories (2D + time) in an augmented reality environment using a STC. A tabletop system displaying satellite information is coupled with the STC visualization to provide additional context information. Furthermore, the user can interact with the data through a tangible ring controller, providing natural interactions.

### 2.2.4 Timeline Visualization

Timelines are a classic visualization for temporal data, which are mainly used to represent series of events linearly or in form of a tree. Timelines have a wide range of design choices [111], and this range allows to create expressive representations. Hence, they are often used for summarizing events, storytelling or historical summaries, but also for planning, using calendars or Gantt charts for instance. Narrative visualizations are also often used to present data and information in an attractive yet understandable way, and timelines can be designed to balance perceptual and narrative effectiveness for this purpose. For example, TimelineJS [112] and Timeline Setter [113] are tools used in general media to generate slideshow timelines to describe long narratives, gathering numerous major events. In a context of data analysis, Lifeflow [114] and Timeline Trees [115] aggregate data using tree structures, and use timelines to explore the temporal aspect of these data.

Brehmer et al. [111] proposed a classification of timelines based on a review of 263 timeline designs, proposing a 3-dimension design space, illustrated in Figure 2.10. The first dimension is the *Representation* of the timeline, i.e. its “most visually salient aspect, its guiding visual metaphor”. The most common and versatile representation of a timeline is linear, yet radial or spiral representations can be more adapted to show periodic data. Other representations can rely on grids, as for calendars, or on arbitrary curves designed to support a narrative. The second dimension is the *Scale*, which determines the link between temporal and displayed distance; it is characterized notably by its reference point and the function mapping the temporal dimension (e.g. linear or logarithmic). The third dimension is the *Layout*, which describes if the display shows one timeline or multiple faceted ones, if the timelines are segmented or not. Following this work, Di Bartolomeo et al. [116] evaluated different representations on classic visualization tasks and proposed additional design recommendations. As the design space includes some non-intuitive possible designs, Brehmer et al. [111] proposed 20 most “viable” designs that were
<figure><img src="image_17.png" /><figcaption>Figure 2.10 – The three dimensions of our design space for expressive storytelling with timelines: representation, scale, and layout; Brehmer et al. [111].</figcaption></figure>

emphasized in their study. Notably, they pointed out that about half of the studied timelines follow linear chronological unified or linear chronological faceted designs. Nonetheless, 3D timeline designs are not approached in this design space, and are actually lightly explored in the literature. The few examples include Beedocs [117], a timeline authoring tool using 3D animations to explore a 2D timeline, HeloVis [118] which displays radar signal data on an helical visualization or works by Kullberg et al. [119], who use the horizontal space of a 3D environment to display time.

Nonetheless, the creation of timelines, and infographics in general, can be very time-
consuming, whether it is for the extraction of information or the visual design. Several methods
exist to generate timelines from different types of information. For example, a lot of temporally
annotated information can be found from blogs, Twitter or Facebook content; tools like Story-
Graph [120], EvenTweet [121] or similar methods [122, 123] allows the extraction of relevant summarizing information and the generation of timelines to represent series of events. Time-LineCurator [124] extracts automatically temporal information from unstructured texts, such as journal articles, and also provides controls for curating and editing the events on a timeline. Charles et al. [125] proposed a method to generate timelines from 3D interactive narratives, which could be used to support more intuitive exploration and edition. To tackle the issue of visual designs of timelines, Chen et al. [126] proposed a deep-learning based tool to extract extensible templates from existing infographics, to use with other types of information.

To help the exploration of timelines, interaction methods adapted to the visualization have been developed. The most common interaction is based on scrolling or sliders to move in time, which is notably recommended by Blascheck et al. [127], for the familiarity with which the user will approach it. For example, works by Charles et al. [125] or Card et al. [128] use timelines to give an overview of events while using sliders to explore data in time, that is displayed in a juxtaposed view. TimeZoom [129] also relies on horizontal scrolling to move in time, yet also base several other interactions on *regions of focus*. The interaction set notably include scrolling in time by dragging the region, zooming on the region or the whole timeline in order to variate levels of detail, and finally means of creation, deletion and edition of such regions. Such classic methods as sliders or pan and zoom were evaluated by Schwab et al. [130], to give creators more guidance to adapt the navigation on their visualizations. Yet, the need for enhanced interactivity led to the use of various user interfaces for timelines. As such, Morawa et al. [131] proposed Time Beads to interact with the timeline using a touch user interface to support intuitive manipulation of time points, and Drossis et al. [132] proposed to explore the temporal information through a *time-tunnel*, displayed in an immersive 3D environment.

However, S4D data is rarely considered on timeline visualizations. A few examples can be found in contexts of 3D model editors, as proposed by Denning et al. [133] or Doboš et al. [134], that use timelines to display the step-by-step construction of the model. Although, S4D visualization tools sometimes use timelines as sliders, juxtaposing 1-dimension additional information with the 3D spatial data, as in Duran et al. [95] work or RubberSlider [59], which uses a VR adjustable slider to explore a timeline and display 3D data.

The two static temporal visualization methods presented in this section provide static views of temporal information, with very general and versatile approaches. On both cases, the visualization can be dense, introduce clutter or be overall difficult to access information precisely. As
such, interactive designs are often proposed to extend the capacity of the user to exploit these visualizations. Although, visualizing spatial 3D temporal data in such context introduces further obstacles, notably in terms of displayable dimension, workspace and user perception of 3D and temporal information. Nonetheless, the field of Immersive Analytics explores the usage of immersive interfaces and displays for visualization. Such system and associated methods could provide solutions to tackle the issues aforementioned.

## 2.3 Immersive Visualization for Temporal Data

As reported in LaViola et al. [39], technologies that support 3D user interfaces, such as stereoscopic displays, 3D tracking or pointing device, are becoming mature and are proliferating. Notably, the release of affordable high-quality HMDs since 2016, notably with the Oculus Rift and HTC Vive, replaced the CAVE system [135] that got popular in the late 1990s. Applications are getting more and more common, in fields as art, education and formation, architecture or gaming. Concurrently, the number of immersive visualization applications publications increased, as reported by Fonnet and Prié [58]. These applications rely on various characteristics of 3D virtual environments (VEs) that match the related tasks: the feeling of presence provided by VEs can enhance gaming and simulation; 3D interactions simulate natural skills and thus support training applications.

In this section, we report Immersive Analytics applications for scientific visualization, information visualization, and finally we consider solutions proposed to tackle the major issue of comparison of 3D and 3D temporal data.

### 2.3.1 Immersion for Scientific Visualization

As mentioned in Section 2.1.2, scientific visualization can take advantage of the perceptual benefits of virtual environment for the representation and display of 3D data. Natural interaction completes this display, helping efficient exploration of the data. As such, many Immersive Analytics systems were developed for scientific visualization, for various use cases.

In the context of biology imaging, El Beheiry et al. [136] proposed DIVA, a Virtual Reality system supporting the exploration of high-resolution microscopy 3D images. The system renders images with up to 4 channels of information, with customizable transfer function for each channel, and includes basic manipulation and annotation features. Another VR system, Genuage [137], handles point cloud data, notably generated through biological microscopy. It
proposes methods to select a group of points in the cloud by defining a convex mesh, placing the vertices through direct 3D interaction. In the domain of archeology, several immersive applications exist to either explore archeological sites or interact with fragile artifacts, as reported by Gaugne et al. [138]. For example, ArtifactVis2 [32] renders 3D environments generated through datasets of GIS data, 3D artifact scans and digital photography, recreating and allowing users to explore complete archaeological sites excavation. Lécuyer et al. [139] proposed the software INSIDE to visualize and manipulate artifacts, relying on the stereoscopic and interactive screen zSpace. Likewise, Immersive Analytics systems approach geographical and trajectory datasets. Hurter et al. [140] propose FiberClay, depicted in Figure 2.11, a system that renders large 3D trajectory sets, which includes a bimanual intersection brushing technique to efficiently select trajectories, and refine that selection by adding or subtracting trajectories from the selection set. Moreover, Filho et al. [105] propose to display trajectory datasets using a Space-Time Cube in immersive environment, as shown in Figure 2.11-c. Providing with a basic interaction toolset, they notably report that it helps in usability and partially addresses the issue of the steep learning curve of Space-Time Cubes.

A specific issue often mentioned in visualization, and especially in scientific visualization, regards comparison. Such task is often a very important step in the analytical process, through the comparison of a simulation or a model to an observed ground truth, or else of multiple acquisitions of a same entity. Yet, it basically adds a dimension to the data, making visualization and interaction challenging, as highlighted by Kim et al. [75] survey. The latter also notably mention that juxtaposition is a natural approach for comparison tasks, and therefore the large workspace available in 3D environments can come especially useful. Hence, immersive analytics systems were designed around comparison tasks. For example, Johnson et al. [141] proposed BentoBox, a Virtual Reality data visualization interface for simulated 3D temporal data, illustrated in Figure 2.11-b. BentoBox disposes of a range of tools focused on the comparison of datasets and manipulation of parameters, using animations, color mapping and 3D bimanual interactions. Following a small multiple approach, it juxtaposes the same simulated volumetric dataset under different parameters, related either to the display options or to the simulation, in an array allowing direct comparison of up to twenty to thirty instances. The small multiple approach was further developed by Liu et al. [142], who proposed a design space to arrange small multiples in a 3D environment, notably to improve comparison.
<figure><img src="image_18.png" /><figcaption>(a) FiberClay, *Hurter et al.* [140]</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>(b) BentoBox, *Johnson et al.* [141]</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>(c) Wagner Filho et al. [105]</figcaption></figure>

Figure 2.11 – Examples of Immersive Analytics user interfaces from the literature. Figure (a) is Fiberclay [140], which features a bimanual interaction for selection tasks in a large trajectory dataset; Figure (b) is BentoBox [141], showing simulations of blood flow in a heart. Each column has different simulation parameter, each row shows a different view of the data; Figure (c) is a VR implementation of a STC representing trajectories by Wagner Filho et al. [105].

## 2.3.2 Immersion for Information Visualization

An additional challenge comes with information visualization, regarding how to use the 3D environment to display non-spatialized data. To create efficient and natural interactive exploration and visualization of such data, several toolkits were developed for immersive environment, such as DXR [143] or IATK [144]. Both use a visualization and interaction grammar inspired by Vega-Lite [145]. Vega-Lite, itself based on the Vega language [146], proposes high-level specification for interactive data visualization, combining a grammar of graphics and a grammar of interaction. The grammar of graphics defines units to describe one set of data, its transforms, filters, type of display and other visualization parameters. Units can be used to create more complex views,
called *layers*. Several operators are then available, to merge shared axis, apply concatenations, create different views. The grammar of interaction is based on *selections*. It is designed to select points corresponding to a predicate when an event occurs, and then apply transforms on the data, such as addition or removing from selection, translation, zoom.

DXR [143] is a toolkit for Unity to design immersive data visualizations. It relies on a Vega-Lite inspired graphics grammar. It defines an analog to the Vega-Lite *unit*, but simplified, named *dxrvis*, composed of the data, the graphical object, a description on how the data is mapped. The graphical object is represented by a Unity prefab, representing each data as a recognizable object. With this grammar, DXR proposes a high-level GUI and a panel of object templates, easy to learn and efficient to prototype quickly visualization, as illustrated in Figure 2.12. The interaction part is though limited to scaling, rotating and details on demand when pointing any of the data item visualized, and performances decrease after an order of magnitude of a thousand objects. IATK [144] is, similarly to DXR, a Unity package for immersive visualization. It is a high-level interface based on a composable grammar capable of rendering millions of data items and offers a large interaction panel. The graphic grammar defines several elements:

- view-frames to choose the type of representation of the data, such as scatterplots or matrices,

- geometric objects to represent the data item, as spheres, cubes or quads, named *glyphs*,

- data mapping,

- visual linking across multiple visualization.

This allows the user to create a wide range of visualization. Some commonly used, such as scatterplots, line graphs or trajectories, are preconfigured in the IATK package, yet the expressive API also allows the creation of novel immersive visualizations, as shown in Figure 2.13. Representing the data through glyphs optimizes the rendering, such that the system can display several million data points at usable frame rate. The interaction model is also designed to support large datasets. It includes functionalities of filtering, i.e. removing outliers or defining ranges on the axes, details on demand, animated transitions and a *brushing and linking* tool to outline corresponding data on several visualizations, using GPU computation to maintain performances.

These approaches compromise differently the spatialization and quality of representation of the data, the interaction and the amount of data available to the user. These different characteristics are adapted to different use cases, notably depending on the amount of data, the type of feature to observe or the profile of the user.
<figure><img src="image_21.png" /><figcaption>Figure 2.12 – Visualization prototypes using DXR, on different use cases and using different types of graphical representation of the data points; Sicat et al. [143]</figcaption></figure>

### 2.3.3 Collaborative Mixed Reality

The question of collaboration in Immersive Analytics is often raised in the community [31, 147]. Ens et al. [147] discuss related challenges, in terms of hardware and communication between users, and also suggest cross platform collaboration for specific use cases requiring different perspectives or users with different roles. We discuss in this section the main solutions existing for multi-user MR applications, and then potential of asymmetrical collaboration setups.

#### Multi-user Applications in Virtual Environments

Margery et al. [148] proposed a general framework for cooperation in VEs, based on three levels of cooperation. The first level, basic cooperation level common to all multi-user systems,
<figure><img src="image_22.png" /><figcaption>Figure 2.13 – VR visualization of multidimensional data (Dow Jones dataset) using IATK. The user brushes a 3D trendline, and the selection is colored similarly on the other visualizations; *Cordeil et al.* [144]</figcaption></figure>

is done by enabling perception of the other users through avatars, and providing communication means between them. The second level is defined by the action the user can execute on the scene. This level can be subdivided, depending whether the interactions are constrained by the scene or if the users have freedom of actions. The third level is attained if multiple users can interact in cooperation on one object. This level can be subdivided as well, first if the cooperation is done on independent properties of the object, or if the properties can be modified through a combination of the input of the users cooperating.

These levels of collaboration must be supported with adapted hardware and infrastructure to share one same virtual environment between multiple users. The most usual solutions rely on see through AR HMDs, VR HMDs or CAVEs. Although, the latter implies important cost and infrastructure, whereas VR HMDs get similar or even better results on classic analytical tasks, according to Cordeil et al. [149], so CAVEs system are now less considered for such application. Network software architecture has to be adapted as well to ensure information and application state consistency between the users. For example, Fleury et al. [150] compared multiple models of network architecture for data distribution and communication between nodes.
The authors also proposed a dynamic adaptive model implementing three data distribution modes, in order to answer to constraints of network capacity, consistency and responsiveness of various environments.

<figure><img src="image_23.png" /><figcaption>Figure 2.14 – Interactive user interface of the data visualizer iViz; Donalek et al. [151]</figcaption></figure>

Multi-user applications for Immersive Analytics are explored a little, with systems as Telearch [152], a virtual reality system for archeology or Shvil [153], an augmented reality system for land navigation, which both proposed distributed collaborative features. Donalek et al. [151] presented a progress report on VR systems directed toward information visualization, and presented iViz, illustrated in Figure 2.14. iViz is a system based on Unity for information visualization, multi-platform and notably supported on web-browser, allowing multiple users to share a view with one user navigating through the data.

Although, as evoked previously, immersive applications are beneficial for certain complex 3D and depth-related tasks, yet some tasks are more suited to 2D setups. One of the solutions proposed in the literature, notably by Isenberg [154], relies on asymmetric collaboration.

## Hybrid Systems and Asymmetrical Collaboration

The development of hybrid systems, i.e. systems that are based on several different interfaces, is closely related to asymmetrical types of collaboration. Hybrid systems allow to implement techniques using multiple displays, and especially using both immersive and 2D setups, in order to cover more efficiently visualization tasks and to display data of different nature [155]. In this dynamic, several systems were proposed, such as the CyberCommons [156] and CAVE2 [157] systems, that propose stereoscopic screens, more adapted to 3D data for example, adjacent to
2D high-definition screens, to display graphs or 2D images. The DataSpace system proposed by Cavallo et al. [158] is a room-sized environment designed for high adaptability of heterogeneous devices. It is composed of 15 high-resolution displays controlled by robotic arms to modify the configuration of the room and screens, depending on the analytical need; it also includes table projectors, AR and VR headsets, and delivers the content through web-based technology to connect easily to smartphones or laptops. With such diverse interfaces, multi-platform compatibility, 3D and 2D visualization capabilities, the authors introduce the term of “Hybrid Analytics” to design such environments.

<figure><img src="image_24.png" /><figcaption>Figure 2.15 – DataSpace environment, with 15 high-resolution displays attached to robotic arms, 2 table projectors, AR and VR headsets; Cavallo et al. [158]</figcaption></figure>

Nonetheless, some simpler systems involve two users in a AR/VR setups or even desktop/VR setups, and can focus on specific scenarios. For example, Vishnu [159] proposes a teacher-student type of collaboration: an expert is equipped with an AR headset and is working on real material, while the student is equipped with a VR setup and interacts with a virtual environment. Usually in collaborative analytical scenarios, the tasks and the information are, at least partially, separated and the nature of the visualizations on both displays are different. Consequently, the issue of communication comes up to synchronize the information and interaction to achieve the analytical task. Reski et al. [160, 161] experimented on the communication between users collaborating through different interfaces. Figure 2.16 illustrates one of the experiments [161], with the two user interfaces, one on desktop, the other in VR. The experiment consisted in finding correlation between climate information, help by the desktop user, and plant information, held by the VR user. The participants then answered a questionnaire proposed by the authors and based on Snowdon et al. [162] characterization of key features of collaborative virtual environments:

— Transition between shared and individual activities;
— Negotiation and communication, through verbal and non-verbal exchange of information;

— Sharing context, i.e. the characteristics and features of ones part of the interface and information;

— Awareness of others, of the activities of the peer.

The results of this experiment reflected good awareness of the peer's activities independently of the interface, frequent verbal communication and a shared understanding of the overall data. Not only the separation of the interfaces and information is not that much of an obstacle to analytical tasks, the users also were enthusiastic about working together in a balanced way, yet through different interfaces.

<figure><img src="image_25.png" /><figcaption>Figure 2.16 – Two user interfaces from Reski et al. [161] experiment. The left image shows the immersive interface, with a 3D Radar Chart visualization [163]; the right image shows the desktop interface, with views of a map and graphs. The desktop interface shows, through a red dot on the map, the position of the immersed user.</figcaption></figure>

## 2.4 Conclusion

In this chapter, we gave the definitions and described the main research questions and objectives of the domains, in order to contextualize our approaches. We notably introduced the notion of 4D object as a representation of 3D temporal features in order to support the interaction and the visualization of the main characteristics of these features. Then, we summarized the main techniques of visualization of temporal data and 3D temporal data, focusing on static visualizations and especially Space-Time Cubes and Timelines. Finally, we reported solutions developed in the field of Immersive Analytics to approach scientific and information visualization, and eventually for collaborative Immersive Analytics. We also presented examples of 3D temporal datasets taken from biological imagery, that we will use to illustrate our contributions presented in the following chapters.
IMMERSIVE AND INTERACTIVE
VISUALIZATION OF 3D SPATIO-TEMPORAL
DATA USING A SPACE TIME HYPERCUBE
APPLIED TO BIOLOGICAL IMAGING DATA

<table><tr><td>3.1</td><td>Space-Time Hypercube Visualization</td><td>51</td></tr><tr><td>3.1.1</td><td>Generalizing the STC in 4D</td><td>51</td></tr><tr><td>3.1.2</td><td>Extracting a 3D STC from a 4D STH</td><td>52</td></tr><tr><td>3.1.3</td><td>Visualization: 3D Rendering and Quantitative Data</td><td>53</td></tr><tr><td>3.1.4</td><td>Interacting with the STH and STC</td><td>55</td></tr><tr><td>3.1.5</td><td>Generation and Interaction Performance</td><td>59</td></tr><tr><td>3.2</td><td>Use Case: Application for Morphogenesis</td><td>59</td></tr><tr><td>3.2.1</td><td>VR Visualization Application</td><td>59</td></tr><tr><td>3.2.2</td><td>Use Cases</td><td>60</td></tr><tr><td>3.3</td><td>Evaluation</td><td>62</td></tr><tr><td>3.3.1</td><td>Tasks and Hypotheses</td><td>63</td></tr><tr><td>3.3.2</td><td>Apparatus and Participants</td><td>64</td></tr><tr><td>3.3.3</td><td>Experiment</td><td>65</td></tr><tr><td>3.3.4</td><td>Results</td><td>66</td></tr><tr><td>3.3.5</td><td>Discussion</td><td>67</td></tr><tr><td>3.4</td><td>Expert Feedback</td><td>68</td></tr><tr><td>3.5</td><td>Additional examples</td><td>70</td></tr><tr><td>3.6</td><td>Conclusion</td><td>70</td></tr></table>
Our first approach regards the visualization of 3D temporal regions of interest with large temporal components. Such region describes the continuous evolution of values or shape of objects, which is essential to analyze an object and can potentially give insight on the global evolution of the dataset.

In this chapter, we propose an extension of the Space-Time Cube (STC) visualization technique [107] adapted to 4D spatio-temporal data, named hereinafter Space-Time Hypercube (STH). Conceptually, the STH is a representation of the 3D temporal data which consists in laying the data in a 4D hypercube with three spatial dimensions and one temporal dimension. To enable the direct visualization of the 4D hypercube, we extended the classical operations of the STC to generate varied meaningful 3D visualizations, that are juxtaposed to obtain a compact overview of spatio-temporal data. Precisely, we propose a projection operation on the hypercube, independent of the type of spatial representation (e.g. mesh, volumetric), which projects the hypercube into a 3D volume that can be directly visualized. The projection operation relies on a user-defined cross-section on the spatial dimension. This cross-section is then computed along the temporal dimension and stacked together into a 3D volume, which can be enriched with numerical and categorical data (see Figure. 3.1).

In order to explore efficiently the complex 3D structure generated, we use the STH in an immersive environment and propose a set of interactive tools to explore and analyze the dataset. The enhanced depth perception of the virtual environment should improve the visual extraction of meaningful structures of the STH. and the use of 3D user interfaces would allow users to explore naturally the STC from different perspectives. The set of interaction proposed includes methods to explore naturally the complex structure, yet also selection methods that use the notion of 4D objects defined previously, supporting further analytical tasks. Furthermore, the enlarged interaction space of VR allows the juxtaposition of the STH and snapshots of the original spatio-temporal data. This juxtaposition enables the synchronized exploration and manipulation of the two visualizations. Finally, the chapter presents a use-case illustrating the usages of the STH for the visualization of spatio-temporal data in embryo developmental studies, as well as an evaluation of this visualization method both with non-expert and domain expert users. The illustrations used and use cases explored in this chapter are based on the recordings of embryonic development of a Phallusia Mammillata, the first dataset presented in Chapter 2, Section 2.1.3.

In summary, the main contributions of the chapter are:

- A novel visualization method for spatial 3D temporal data based on the Space-Time Cube method, with a generation algorithm independent of the spatial data encoding.

- Description of VR system for the visualization of the STC including specific 3D interaction
methods for the exploration and manipulation of the juxtaposed views.

— A formal evaluation of the proposed method on embryology use cases by non-expert users and domain experts.

<figure><img src="image_26.png" /><figcaption>Figure 3.1 – From a cross-section on the 3D surface-based temporal data shown on the first figure, we generate a Space-Time Cube visualization, displayed in the second image, showing the evolution over time of the spatial data of the cross-section displayed on the x and y axes. The third picture shows how the visualization can be enriched with quantitative and qualitative data using different color coding. A set of interaction tools help the user to explore the generated visualization, as seen in the last image.</figcaption></figure>

**3.1 Space-Time Hypercube Visualization**

This section presents our main visualization method, based on the Space-Time Cube. First, we present a generalization of the STC in 4 dimensions, the STH, then we detail the design choices for the generation and visualization, as well as the interaction methods available.

As a support example for this section, we will use a dataset of a time-varying 3D spatial data, a 6-hour-long live recording of an embryo development. In this work, we use 100 of the 180 time points recorded, from a 65 to 383 cells embryo. Figure 3.1 shows some of the time steps of the embryo that were used. The dataset is composed of surface meshes, for each cell at each time step. The temporal links between cells, notably after division, are also present, as well as quantitative and qualitative information. The dataset will be described more thoroughly in Section 3.2.
**3.1.1 Generalizing the STC in 4D**

In a classic 3D STC, the data visualized is composed of 2 spatial dimensions as well as a
temporal dimension (x, y, t). In contrast, a STH can contain 3 spatial dimensions and a temporal
component (x, y, z, t). With this abstraction, any elementary operations applicable on a STC,
notably described by Bach et al. [107, 108], should be extendable and applicable to the STH.
Geometry and content transformations now modify a 4D volume, filling operations now includes
4D interpolation, extraction operations can render 4D volumes and flattening operations can
compress 4D data into a 3D volume. Complex operations for a STH can be designed based
on this extended set of operations. One of the main objectives of such operations is to provide
meaningful visualizations of the data, but with varied points of view and dimensionality.

In practice, while a STC can be directly rendered, a STH cannot be directly rendered as
is. To enable a direct visualization, a projection operator on the STH has to be defined either
by using an extraction or a flattening operation that would result on a 3D volume, yielding a
3D visualization. However, using such method on denser data can result in a volume which
would potentially be prone to occlusion issues, because of the increased density due to data
aggregation. In contrast, a volume extraction operator would enable the reduction of the data
dimension without the increased data density. Thus, we opt for a volume extraction operating by
defining a hyperplane laying in a 4D space. Nevertheless, to major issues remains:

- An extraction operation implies a loss of data during the creation of the visualization. Thus, the user must have control over which data is selected.

- The concept of a 4D hypercube is not easy to comprehend; the extraction of a volume along a hyperplane is even more difficult. The complexity of the model will be a constraint during the extraction operation, as well as any other interactive operation based on the 4D data.

These two constraints will influence our design choices, especially the generation process for
our 3D STC visualization.

**3.1.2 Extracting a 3D STC from a 4D STH**

The proposed projection method relies on the use of a 4D hyperplane, manually defined by
the user, to extract the 3D visualization. However, the definition of a hyperplane of this nature
is an abstract task which cannot be directly visualized. A simple example would be to define
a hyperplane perpendicular to the time axis which would result on the extraction of the spatial
<figure><img src="image_27.png" /><figcaption>Figure 3.2 – Flow diagram of the STC generation. In step ①, the user places the interactive clipping plane to get the desired cross-section. In step ②, camera parameters are automatically set in order to render the cross section at each time point. The image presents the output of the rendering operation, using the RGB channels to save cell identifiers. Stacking the rendered images yields a 3d volume, as shown in step ③. Each voxel contains a cell identifier, and its position in terms of depth indicates a time point $t_i$.</figcaption></figure>

data at a given time point. In order to ease the process and obtain a hyperplane yielding more informative STCs, we propose a user defined approach based on the cross-section, a commonly used tool in scientific visualization to explore spatial 3D data, extracting a 2D view from 3D data.

Precisely, the user can place an interactive clipping plane, that will act as a cutting plane, as they would do to get a cut-away view of 3D spatial data. With the same analogy that we used to consider our spatio-temporal data as a 4D volume, the plane, as a time-varying object in the 3D space, can be considered as a hyperplane in the 4D space. The operation thus corresponds to a projection such as $(x, y, z, t) \rightarrow (x', y', t)$, with $x', y'$ the projected coordinates on the cutting plane. Such projection avoids any spatial distortion that could impair the interpretation of the output shape. Considering the whole dataset as a 4D Space-Time Hypercube, it sums up as a space cutting [107] by a hyperplane, extracting a 3D space-time cube.

The proposed generation method is data agnostic (either mesh-based or volumetric) and its three main steps are illustrated in Figure 3.2, and summarized in Algorithm 1. ① The user places a clipping plane on the model at any time point in order to define the “hyperplane” which will determine the projection. ② For each time step, a 2D image of the cross-section of the 3D model is computed. The rendering is achieved by setting an orthogonal camera perpendicular to the clipping plane and setting the near and far planes at a distance of $\epsilon = (z_{near} - z_{far})/2$ towards the clipping plane. The field of view of the virtual camera is minimized according to the maximum size of the cross-section over time. The color channel of the rendered image encodes objects (cells in our examples) identifiers, which allows indexing numerical or categorical data during
<figure><img src="image_28.png" /><figcaption>Figure 3.3 – Visualization process. Figure ① corresponds to the output of the STC generation. Each voxel contains the cell identifier $c_j$ and a time point $t_i$ can be found. Figure ② is a texture containing quantitative information - here the volume of each cell at each instant - normalized between 0 and 1. The value at $(c_j, t_i)$ points to a 1D color gradient, shown in figure ③. Figure ④ is the resulting visualization after shading and color mapping. Smaller cells can thus be identified by a colder color, bigger cells by a hotter color.</figcaption></figure>

visualization. ③ The output images are stacked into a 3D texture in which the RGB channel is used to keep track of the object identifier. The depth coordinate for each pixel encodes the time step. The use of object identifiers is considered for convenience when the spatio-temporal data has this information. In case that this information is not available, other information could be stored, such as density values from a CT or fMRI volumetric dataset.

**Algorithm 1** Generation of the Space-Time Cube. First, the camera parameters are initialized and the identifiers of the cells are encoded into colors. Then, for each time point, the camera renders the cross-section and the result is added into the 3D texture.

1: **procedure** GENERATESTC(*ray*, *start*, *end*) : Texture3D

2: Texture3D texture3D

3: InitCameraParameters()

4: **for each** cell ∈ Cells **do**

5: cell.color ← encodeIdAsColor()

6: **for each** t ∈ [[*t*₀, *t*₁]] **do**

7: tempTexture2D ← camera Render()

8: texture3D.AddTexture2DAt(tempTexture2D, t)

9: **return** texture3D

3.1.3 Visualization: 3D Rendering and Quantitative Data

The projection operator generates a 3D texture which can be directly rendered using Direct
Volume Rendering methods [164]. Due to the nature of our datasets, we consider an opaque
rendering avoiding semi-transparency, in order to limit color distortion when using color to encode information. Thus, once a hit is found, the final color for the ray can be computed. In order to compute the color of the ray, two indexations are required so that we can properly use the cell identifier and annotate the objects accordingly. First, using the cell identifier ($c_i$) and the time step ($t_j$), encoded in the RGB channels of the 3D texture and the depth ($z$) position of the voxel respectively, we can obtain the attribute value of the cell at the given time step ($a_{i,j}$). This information is encoded in a 2D texture (see Figure 3.3 ②) Second, the attribute value allows to index a 1D texture, which encodes the color (Figure 3.3 ③). Thanks to this method, switching either the information displayed on the visualization or the colormap used only consists in switching a small texture. Finally, we apply a Blinn-Phong illumination model on the cube to enhance depth cues and get a better understanding of the shape of the volumetric data. To reduce the rendering time, normals are pre-computed during the generation of the STC and stored in an additional 3D texture. Normals are computed using 3D Sobel-Feldman gradient operators on each direction:

$$ \vec{n} = \left( \frac{\partial S}{\partial x} \quad \frac{\partial S}{\partial y} \quad \frac{\partial S}{\partial z} \right)^T \qquad (3.1) $$

with S(x,y,z) representing the surface, $\vec{n}$ normal to the surface. The result of the 3D rendered color mapped STC is shown in Figure 3.3 ④. The full adapted raycasting algorithm is described at Algorithm 2. Depending on the numerical or categorical information that has to be rendered, a number of attributes and color textures can be pre-computed. As discussed in the previous section, when object identifiers are not available, the information stored at each voxel could be used to directly index a color palette.

However, the rendered STC could still contain a lot of information and could present a complex layout. The following section details manipulation techniques proposed in order to improve the exploration of the STC.

### 3.1.4 Interacting with the STH and STC

Interaction will be a key concept to compensate for issues implied by direct rendering, such as occlusion. Bach et al.[107, 108] described several generic operations that could be applied to a STC, and mentioned dynamic uses of those operations. In addition, by displaying the STC in a VR environment, additional depth cues will be available, such as binocular cues and motion parallax cues [165], which will increase the identification of the intricate structures in the STC.

In the first place, as previously mentioned, the STC is interactively generated, since the user
can choose the base cross-section, and thus the hyperplane, and start the generation process at run time, doing a space-cutting operation of the 4D data. Taking advantage of immersive environment in terms of interaction, we considered and designed a number of interactions adapted to our data structure and the benefits it would imply for data visualization. All the techniques described in the following sections assume that the user is wearing a head mounted display (HMD) and holding by the hands two 6DoF hand-held controllers. As such, the user will be able to explore the STC by just naturally walking in the virtual environment and use both hands to use the different tools. We refer the reader to the accompanying video which showcases the different interaction tools.

Exploration tools

The tools described in this section are meant to help the user to explore the potentially dense
and occluded volumetric representation.

**Selection operator:** This tool is a pointer that displays summarized information of the hovered object, such as its identifier. More detailed information can be accessed upon clicking on

**Algorithm 2** Simplified Ray Casting Algorithm. The method considers a *ray* generated from the virtual camera position and *start* and *end* define the first and last intersections points with the 3D volume. Four major textures are used: 3D texture containing the cell identifiers (textureCell), the 2D texture containing the quantitative data for each cell (textureData), the gradient (1D) used to encode the quantitative data (texture Palette), and the normal 3D texture (textureNormal). As we are considering and full opaque rendering pipeline, once the first intersection is found the *RayCasting* is terminated.

1: **procedure** RAYCAST(*ray*, *start*, *end*) : RGBColor
2:     current ← start
3:     **while** current < end **do**
4:         voxel ← computeVoxel(*ray*, current)
5:         cellId ← fetchCellId(*textureCell*, voxel)
6:         **if** cellId ≠ -1 **then**
7:             timeStep ← voxel.z
8:             value ← fetchData(*textureData*, cellId, timeStep)
9:             color ← fetchColor(*texture Palette*, value)
10:            normal ← fetchNormal(*textureNormal*, voxel)
11:           color ← computeLighting(*color*, normal)
12:        **return** color
13:    current ← current + samplingDistance
14:    **return** (0, 0, 0)
<figure><img src="image_29.png" /><figcaption>Figure 3.4 – STC (right) and meshed model (left) visualizations in the VR framework. On the laser selector, a list of available information allowed the user to color map the remaining lifespan of cells on the visualizations. A cell is selected on the meshed model. The name of the cell is displayed on the pointer, and feedback appears on the STC, highlighting the lineage of the cell.</figcaption></figure>

the trigger. This is overall a detail-on-demand operator, making the whole data accessible at any time. It appears as a red arrow object, so that the user can put the pointer precisely on the desired object. The interaction is supported by a highlight feedback upon hovering and selection.

**Switching the color mapped information:** When various numerical or categorical information are present in the dataset, the user can scroll through the list of information displayable (see Figure 3.4). As shown in Figure 3.3, the underlying operation corresponds to switching the texture containing the information. This is thus a light operation allowing interactive use.

**Clipping Plane:** An interactive clipping plane generates cut-away views of the STC. The user can directly grab the clipping plane and manipulate it using direct hand translations and rotations. Yet, some orientations are easier to interpret:

- Time cutting, i.e. putting the plane perpendicularly to the Z-axis, corresponds to going through the original cross-section at given instants.

- Linear space cutting, i.e. putting the plane along the Z-axis, corresponds to the evolution of a segment of the spatial data over time, giving for instance information of the evolution of the spatial structure.

- In case of moving objects, slightly oblique cutting can help track the object over time.
This type of cut was used to generate the cut in Figure 3.6.

**Light:** Taking advantage of the Blinn-Phong illumination model, an interactive light helps the user to reveal shape, reliefs and details of the STC, using this tool as a point light. This helps understand the shape of the volumetric data by adding depth cues. The user could directly grab the light source (yellow sphere) and manipulate it using direct hand translation.

## Filtering Operations

As occlusion remains an issue in dense volumetric data, reducing the amount of data displayed becomes necessary.

**Value Filter:** This tool allows the user to filter the range of continuous data displayed on the STC. Filtering on value can help identify groups of objects sharing same characteristics, complementing the color mapping by removing the occluding other objects. The parameters of the filter are adjustable through a user interface which allows to define the range of values that are displayed.

**Time Filter:** This filtering operation lets the user define a temporal window to constraint the data rendered in the STC within two time steps. Events ongoing during this specific temporal window can then be more easily pinpointed. The parameters of the filter are adjustable through a graphical user interface which allows to define the range of values that are displayed.

**Object Filter and Tracking:** Focusing on objects, two issues came up. First, we need to keep track of objects in the temporal dimension. Second, the presence of numerous objects can occlude the visualization. The user can solve these issues respectively by highlighting or hiding the objects. It can be done either by pointing the object using the same tool that gives details on demand, described in 3.1.4, or by selecting an object in a list of identifiers. The selected object thus changes state —from normal to highlighted to masked —upon clicking the trigger of the controller. Note that if the objects follow a tree hierarchy in time, such as the cells with their children cells in our example dataset, the operation is applied to the children objects. Figure 3.4 shows an example of object tracking.

## Linking Multiple Visualizations

We also took advantage of the large workspace offered by virtual environments to display the meshed model in addition to the STC. We followed advice from Munzner [16] for the
design choices implied by this juxtaposition: multiple views should be coordinated, and even
interactively coordinated. Conceptually, the operations we described above are applied to the
complete dataset. Feedback of these operations is given through the visualizations displayed -
the STC and the meshed representation of the 3D spatial data. Links can then be set up between
the visualization by applying a shared encoding of the effect of the operations.

**Linking the time exploration:** In addition to the time filtering sliders evoked previously, a middle cursor controls the current time displayed on the meshed model. Feedback of this cursor appears as a transparent plane orthogonal to the temporal axis of the STC, as shown in Figure 3.4.

**Linking the spatial exploration tools:** The same color mapping is applied to the STC and meshed model, notably after switching the color mapped information interactively, as shown in Figure 3.4. The clipping plane used on the meshed model to generate the STC can go back on click to a default position, which is the one of the displayed STC's base cross-section. A marker line on this cross-section gives context information about the position of the clipping plane applied to the STC. This line corresponds to the intersection between the STC's clipping plane and the temporal feedback plane.

**Linking the filters:** Any filtering operation done on the STC is also applied to the meshed model, and vice versa. The same objects are highlighted, hidden or filtered out by time or value.

These operations were designed to help users to explore temporal evolution with the static
visualization offered by the STC, and correlate the multidimensional data or pinpoint events
by combining the two visualizations and the interactive tools available, as illustrated with the
following use case.

**3.1.5 Generation and Interaction Performance**

The VR application runs on Unity 2018.2.21 and is supported by a PC with Windows 10, an Intel Xeon W-2104 CPU (4 cores, 3.2 Ghz base frequency) and a RTX 2080 GPU. All interaction methods described previously are designed in order to maintain a framerate above 45 fps.

In terms of performance and texture resolution, the STC used in the figures has a resolution of 256x256x100. The full generation process takes about 5 seconds. This process is GPU-friendly, which enables the generation of the STC during runtime. Normals are for instance generated and saved instantly using compute shaders. However, the bottleneck of the current implementation
of the generation algorithm is the CPU - GPU transfer, provoking a framerate drop. Hence, we
chose to forbid completely other interactions during the generation time.

**3.2 Use Case: Application for Morphogenesis**

This section presents an application of the STH to the visualization and analysis of spatio-
temporal data from the recordings of embryonic development of a Phallusia Mammillata, the
first dataset presented in Chapter 2, Section 2.1.3. First, we detail a VR application in which the
STH has been integrated. Then, we describe two use cases illustrating the potential benefits of
the STH.

**3.2.1 VR Visualization Application**

<figure><img src="image_30.png" /><figcaption>Figure 3.5 – Examples of tool uses: the first one allows the creation of a scatter view ①, with green lines representing adjacency links. The cube ② is a two-handed control allowing rotation and rescaling of the embryo. The clipping plan ③ is remotely controlled by the user’s hand.</figcaption></figure>

Taking advantage of the large workspace offered by immersive environments, we developed a
VR framework for the user to interact with the meshed model, in addition to the STC visualization,
as illustrated in Figure 3.5. We based our environment on the framework *MorphoNet* [166],
an online interactive browser for the exploration of morphological data. The development of
the application was done using Unity 3D and the HTC Vive was the main visualization and
interaction system.

Our framework displays the dataset described above using meshed model of the embryo at
each time point, as well as a STC based on a cross-section placed close to the median plan of the
embryo, as shown in Figure 3.7. The color mapped on the visualizations here corresponds to the
volume of the cells.
Regarding the user interface, it was displayed as a virtual desk containing all the available tools. In order to enable a tool, the user just had to grab it using the trigger button of the HTC Vive controller. Once the tool was grabbed, the trigger could also be used to enable and disable it. Tools that enabled the control of a continuous value (e.g. the time), or the scrolling of a list require the user to perform circular motions on the circular touch pad of the HTC Vive controller. The user could grab a tool in each hand.

The transformation applied by these tools, i.e. selection, clipping or rotation, can also be applied to the STC. In addition, the rotation can be enabled at any time, even while using a tool, by holding the trigger button on both controllers. This set of tools, as well as the ones dedicated to the STC described in section 3.1.4, provides the user primary controls to explore the dataset.

## 3.2.2 Use Cases

In the way to study functional organization and arrangement of the tissue, embryo developmental studies remain essential to access cellular functions inside a self-organized isolated system.

With a fast development, a few hundreds of cells, ascidian embryo is the perfect choice to analyze the link between cells or tissue geometries and differentiation. In the case of *Phallusia mammillata*, the cell membranes are very transparent, which makes for an easier imaging of the membranes or endosomes of the cell through light sheet microscopy. Without any apoptosis, i.e. programmed cell death, or cell migration in early ascidian development, embryo topological complexity can be summarized in the result of unequal divisions of cells, resulting in two different volumes for each daughter cell, and/or asynchronous divisions, i.e. two daughter cells having different cell cycle duration. Each of these events correlates with cell fate decision of the cells. This way, with the exploration of cell architecture and adjacency in a dynamical view in embryos, we have access on a major part of the story and decisions taken by each cell.

We will illustrate in the following examples of use of the STC to observe events and behaviors with a strong temporal component related to the exploration issues described above. For this purpose, we generated a STC from the embryo cross-section shown in Figure 3.7.

### Cellular Deformation

One of the main steps of the embryonic development for most animals is gastrulation. It is characterized by morphogenetic movements that form a cavity in the embryo. These movements set the base of the determination of the morphology of the future individual. The actual source
of these movements and deformations of the cells is still unsure, and no formal measure of
deformation exists. Visualization of this deformation over time is necessary for the analysis of
the embryonic development.

The STC shows the deformation of the embryo over time, as shown in Figure 3.6. A cross-section allows the exploration of the main cells involved in the movements and provides a view of the global and local deformation. The user can then identify the cell to visualize by selecting them on the STC and continue the exploration on the 3D representation.

## Waves of Cell Divisions

During the early stages of the embryonic development, cells tend to divide approximately simultaneously. This phenomenon results in “waves” of division, during which numerous cells divides within a short time.

By sliding through the time component, one can notice this kind of event on the surface-based representation through several aspects. First, the characteristic dynamics of the membranes of a cell can be observed on a large number of cells: on the surface of the embryo, and inside by using a clipping plane. Second, as cells divide, they reduce their volume. Thus, the color mapped on the embryo will change a lot during such event.

Keeping this last property in mind while exploring the STC, color clusters become distinguishable thanks to the volume information displayed. This observation can thus be done directly, without having to do a time-sliding operation, as shown in Figure 3.7. Up to four color clusters, hence three waves of division, can be identified.

## Asynchronous Behavior in Cell Divisions

Regarding cell determination, cells with different fates will not result in the same number of cells, and embryonic development has to be controlled. Thus, at specific moments, some cells will not divide at the same time as the other ones and skip a wave of division.

Exploring the surface-based representation in order to find cells presenting this behavior, the user has to slide back and forth in the time window corresponding to the wave of division and observe which cell will not divide.

The STC can give a more global view of this time window, with a part of the context of the embryo. In Figure 3.8, membranes of a cell are highlighted in red. Only one division can be observed for each of these cells during the time window represented. Up to three were to be expected here, according to the waves of division identified before, which confirms an
asynchronous behavior for these cells. With the feedback of the cell selection tools on the STC,
the user can find quickly the related cell on the 3D model of the embryo.

<figure><img src="image_31.png" /><figcaption>Figure 3.6 – The STC at the left presents the cavity formed by gastrulation. The cross-section at the right shows the deformation of cells involved in this particular morphogenetic movement.</figcaption></figure>

<figure><img src="image_32.png" /><figcaption>Figure 3.7 – Cross-section and related STC generated. The color displayed on the STC corre-
sponds to the volume of cells, and helps distinguish 4 parts.</figcaption></figure>

**3.3 Evaluation**

In this section, we present and discuss a user evaluation aiming to assess the benefits and
limitations of the proposed method using the previously defined use cases. The evaluation was
motivated due to the complex and intricate 3D visualizations that are generated by the proposed
projection, which could be difficult to interpret. Thus, the main goal of the evaluation was to
<figure><img src="image_33.png" /><figcaption>Figure 3.8 – Cross-section of the STC. Two divisions are indicated. The membrane of the original cells and their respective daughter cells are highlighted in red.</figcaption></figure>

provide a first assessment on whether and how users could take advantage of the generated STC
visualizations when punctual and continuous events have to be located. Although other use cases
could be envisioned, we decided to focus on the biological use cases, as they were our first
motivation for proposing the STH approach. In this context, as biologists mainly explore this
dataset by exploring one time point at a time, as a baseline condition, we considered the mesh
visualization on which users can navigate in time (e.g. using a slider); hereinafter referred to as
the meshed model alone condition. In the meshed model & STC condition, the mesh data was
also juxtaposed to the STC visualization.

**3.3.1 Tasks and Hypotheses**

We focused our evaluation on the efficiency and accuracy provided by the STC through tasks
of identification of cell divisions, waves of divisions and asynchronous behavior, as described in
3.2.2. Such events and behaviors have temporal and spatial components, thus can be considered
as 3D temporal objects. Using the same abstraction as described in section 3.1, they qualify
as 4D objects. As such, we suggest that one of the visualizations will render more information
depending on the characteristics of the event. We discern two types of events. First, punctual
events can be identified by comparing the state between 2 time points. To detect such events, the
user requires more complete spatial contextual information over a very local temporal context,
i.e. a few time points. For this reason, we hypothesize that the meshed model visualization will
be more helpful is the detection of punctual events. The second type of events evolves over time,
and is thus mostly characterized by a temporal component extended in numerous time points. As
the generated STC provides complete temporal information over a reduced spatial context, we
hypothesize that the use of the STC will be more relevant to detect such events.

For this study, we considered the complexity of the tasks and the fact that our participants
were not domain experts. Consequently, we chose to focus our evaluation on the accuracy of the
analysis and not on efficiency, which could be more biased, notably by the approach taken by the
participant. Based on our previous rationale, our main hypotheses were:

**H1:** Punctual events in time will not be pointed more accurately using the STC than on the meshed model only;

**H2:** Events or behaviors with a larger temporal component will be identified more accurately using the STC than on the meshed model only.

**3.3.2 Apparatus and Participants**

The PC configuration is the same as described in 3.1.5. The virtual environment was displayed
in an HTC Vive Pro HMD, with a resolution of 1440 x 1600 per eye. Two Vive Pro controllers
were available, including 3 buttons and a touchpad. The HMD and the controllers were tracked
in an empty space of a 2.5m x 2.5m surface.

12 participants, 10 men and 2 women, were recruited from the local laboratory for this experiment, aged from 22 to 32. They all had medium to high experience in VR environments, but had no expertise on the dataset nor in embryology.

**3.3.3 Experiment**

**Task Design**

In order to keep the morphogenetic analysis context, we designed tasks of annotation of the
dataset that were still accessible to non-expert users. From the use cases presented in section
3.2.2, tasks of pinpointing waves of cell divisions or identifying asynchronous behavior in
cell divisions, respectively section 3.2.2 and 3.2.2 corresponded to events with large temporal
component.

As a task of pinpointing punctual events, we considered cell divisions. They are common
events yet essential for the analysis of the embryonic development. The temporal resolution of
the dataset allows us to consider the divisions as punctual events.

The spatial deformation occasioned by a division can be easily noticed on the 3D representa-
tion of the embryo by sliding through time. An additional hint would be the abrupt change in
the color displayed, which can be used to pinpoint this event on the STC. As described in 3.1.4,
the hierarchy, corresponding here to the cell lineage, is highlighted. Upon selection, finding the
change of color in the highlighted selection thus corresponds to finding a cell division.

Experimental Protocol

At the beginning of the experiment, participants signed an information consent form and were briefed regarding the equipment used, the data recorded and the experimental tasks. In order for the participant to understand the context of the tasks, we explained the nature of the dataset, specifying the methods of recording and the purpose of analyzing such data. Emphasizing on the temporal component of the data, we then presented the STC visualization, its generation process and its purpose in terms of temporal visualization. Since the participants were non-experts, we explained for each task the characteristics of the events to pinpoint, the scientific purpose in embryology analysis as well a few methods to identify those events in each visualization. Finally, we presented the virtual environment and the available tools to the participant for about 8 minutes, before having themselves equip the HMD and get familiar with the framework, for about 8 minutes as well.

Participants were then asked to perform the three exploratory tasks:

- Discern the number of waves of cell divisions, providing an approximate estimate of the time for each wave of divisions; duration of the task: 2 minutes

- Identify cell divisions and give corresponding instant and cell identifier; duration of the task: 4 minutes

- Identify cell with asynchronous division behavior and give corresponding cell identifier; duration of the task: 4 minutes

For the tasks of identifying divisions and asynchronous behaviors, the events were present
abundantly enough not for anyone to finish the task early. The experiment was divided in two
blocks. Depending on the visualization method available, the participant performed the tasks
first with the meshed model alone and then with meshed model & STC, or vice-versa. The order
of the visualization method was counter-balanced to minimize ordering effects.

As the main goal of the experiment was to evaluate how users will take advantage of the
generated STC visualizations, we decided to provide an initial STC visualization. In addition,
having the participants work on the same dataset and the same STC view generates smaller variability and enables better comparison of results between participants. Second, considering the potential learning curve in STC based visualizations, we assumed that the level of comprehension of the STH would be very different from user to user. Such differences could induce an important bias in the experiment, especially since the users are not familiar with the datasets observed. We thus generated a STC arbitrarily, though following one objective criterion: the plane would be placed in order to capture the gastrulation movement, as described in part 3.2.2. We expected this to act as a point of reference to help participants locate themselves in the data. Nonetheless, we left the opportunity for the participants to generate another STC from a different cross-section, but none of them did.

For each condition, we recorded the answers of the participant for each task, the total time use of each tool, and the total time watching either visualization. At the end of the experiment, participants had to fill a short questionnaire in order to gather subjective impressions of the overall app and demographic data (age, gender, and VR experience).

### 3.3.4 Results

Eventually, we get a total of 24 paired observations, characterized by 2 two-level factors, the **order** of the observation and the **condition** - i.e. whether or not the STC is present. Since we focus the study on accuracy, we computed the precision for each task:

$$precision = \frac{true \ positives}{total \ selected \ elements} \qquad (3.2)$$

Repeated measures two-way ANOVA was used to analyze the results, considering the order as a between-subjects factor and the technique as a within-subjects factor. When the normality assumption was not met (Shapiro-Wilk normality test), the Aligned Rank Transform (ART) was used before performing the ANOVA analysis. When necessary, Tukey HSD post-hoc tests were performed. The statistical analysis was done using R language.

#### Waves of Cell Division Counting Task

The two-way ART ANOVA order and condition vs precision showed a significant effect of the condition ($F(1,10) = 18.10; p < 0.01; \eta_p^2=0.64$), and no significant order nor interaction effect. Post-hoc tests showed a significantly ($p < 0.05$) better performance in terms of accuracy using meshed model & STC ($M=0.94; SD=0.16$)) rather than the meshed model alone ($M=0.72$;
SD=0.16)). This **supports H2**.

Asynchronous Behavior Identification Task

The two-way ANOVA order and condition vs precision showed a significant effect of the condition ($F(1,10) = 5.61; p < 0.05; \eta_p^2 = 0.36$), and no significant order nor interaction effect. Post-hoc tests showed a significantly ($p < 0.05$) better performance in terms of accuracy using meshed model & STC ($M=0.87; SD=0.14$) rather than the meshed model alone ($M=0.64; SD=0.27$), **supporting H2.**

Cell Division Identification Task

The two-way ANOVA order and condition vs precision showed no significant result on any factor nor interaction, so we can give **no conclusion about H1**. The box plot of the precision of the participants on this task (see Figure 3.9) does not show important difference either.

**3.3.5 Discussion**

The results from the *waves of divisions* and *asynchronous behaviors* tasks, considered as
objects with large temporal component as explained in 3.3.3, showed that the use of the STC
provided higher accuracy in those identification tasks, **supporting H2**. However, these tasks
were designed with numerous known targets but unknown location in space and time. Placing
the user in more diverse situations would help to estimate more thoroughly the added value of
the STC in general exploration tasks. The results from the cell division task gave no significant
results, allowing **no conclusion on hypothesis H1**.

As evoked by our hypotheses, we focused this study on accuracy. For various reasons, the
protocol as it is could not allow a significant efficiency evaluation. Participants were given the
same advice on how to approach the tasks, described in 3.2.2 and 3.3.3. However, the numerous
tools available made it difficult for them to choose an optimized strategy. Users usually chose a
strategy at the beginning of a task and followed it until the end, even if it felt suboptimal. Such
phenomenon creates subgroups. To counter this effect, we would require either more participants,
or ones that are trained with the system. Furthermore, we focused our study on only one dataset,
since it was the most complete one at our disposal. Considering our task design, we could expect
an important order effect for our efficiency measurements, since users would remember what
they did on the same task in the other condition.
Despite all these constraints, we could output a result on the efficiency of the system. On the *waves of division task*, some users could finish at least 20 seconds before the 2 minutes time limit. In the meshed model only condition, only 1 out of 12 finished the task early (0 on the first part, 1 on the second part). In the meshed model & STC condition, 8 out of 12 finished the task early (4 on the first part, 4 on the second part). The two-way ANOVA order and condition vs the boolean value "finished the task early" showed a significant effect of the condition ($F(1,10) = 14.41; p < 0.01; \eta_p^2=0.59$), and no significant order nor interaction effect.

<figure><img src="image_34.png" /><figcaption>Figure 3.9 – Box plots of the precision of the participant in each of the 3 tasks task and for each condition, i.e. with or without STC. Each plot corresponds to 12 observations.</figcaption></figure>

**3.4 Expert Feedback**

During the design and development of our method, five domain experts helped us by providing recommendations and testing the application. In addition to the informal discussions during the design of the system, the domain experts tested the application in two different sessions at different stages of the development of the prototype. Each session lasted one hour for each domain expert. The domain experts had no previous experience with Virtual Reality, three of them had experience with the *MorphoNet* framework and the dataset. The two others had no experience with either.
In the first session, most filtering operations and interactions on the STC were not imple-
mented yet, so the main focus was on the overall usability of the system, their first impressions
on the STC visualization and the potential integration of a VR system in their workflow. In the
second session, the domain experts were confronted with the same tasks that were considered for
the non-expert evaluation. However, mainly expecting subjective feedback, there was no time
limit for the tasks. These tasks were proposed as exercises to introduce the STC as an interactive
visualization tool, but they did not completely corroborate with usual tasks achieved by the
analysts.

During the first session, the application was overall appreciated. Despite the low experience
with VR or virtual environments, the users quickly handled the tools at their disposal, deemed
quite user-friendly. Nonetheless, the large variety of tools in the environment inevitably creates a
consequent mental load, that requires more experience with the system to get used to. The STC
visualization felt more difficult to approach due to the higher level of abstraction of the data.
However, the handling was quite fast. At the end of the test, all users were able to interpret the
visualization and have a good intuition on how to use it, especially regarding temporal behaviors
in the data. A user notably expressed the necessity to be actually confronted to the visualization
to understand it. The format of the test answered this need, introducing the visualization with an
analysis exercise.

During the second session, opinions were mitigated on the asynchrony and division detection
tasks, depending on the strategy applied by each user, i.e. their choice of tool and ease to use
them. They were although unanimous on its utility for the wave detection task. The STC was
deemed equivalent to a cell lineage tree [167]; although, the visible neighborhood relations give a
supplementary dimension in analysis. This, aided with adequate color mapping, could be relevant
in cases of analysis of signals or communications between cells for example.

Few other improvements were suggested during these tests, most of them regarding the
addition of further information in the environment, or enhancing some controls. The need of
comparing and linking different datasets was also especially pointed out, echoing with Kim et
al. [75] observations.

**3.5 Additional examples**

In this section, we present two additional examples of datasets, also taken from the Mor-
phoNet database, for which we generated STCs with the method previously described. The
cutting plane capture resolution on these examples is 512x512. For each of the two datasets, we
generated STCs based on three orthogonal cutting planes.

The first dataset shown in Figure 3.10, left side, is a simulation in 42 time points of an abstract oryzalin-treated organ. The appearing outgrowth is obtained by locally softening the membrane of a shoot-apical meristem filled with uniform and steady turgor pressure. The stress magnitude on each cell, notably implied by this outgrowth, is colormapped on the STCs. The second dataset, shown in Figure 3.10, right side, is part of a 3D temporal imaging atlas of cell morphology for the C. elegans embryo. The recording includes 150 time points, from the 4 to 350 cells stages. The STCs displayed show the remaining lifespan of each cell as a colormap.

More details about the datasets and related publications are given on the Morphonet website (https://morphonet.org/dataset), respectively under the names “Shoot-apical meristem (Simulated, oryzalin-treated organ formation)” and “Sample 04 (Wild type C. elegans , Live Confocal Imaging)”.

## 3.6 Conclusion

In this chapter, we have proposed a novel spatio-temporal visualization based on the Space-Time Cube visualization. The proposed visualization, the Space-Time Hypercube, extends the STC visualization to consider a third spatial dimension in the data. However, the interaction and manipulation of 4D remains unpractical. Thus, to enable a direct visualization, we proposed a projection operator on based on a user-driven cross-section defined in 3D space. The projection of the hypercube, consequently a STC, contains only a partial spatial information of the dataset, but creates a view containing temporal information. Numerical and categorical information could be displayed as well on the visualization. We juxtaposed and linked the STC and original dataset visualizations in a VR application, taking advantage of immersive environments benefits in terms of visualization and interaction in 3D. Various tools for exploration, filtering or tracking objects apply transformations on both visualization and come to assist in the analysis of the dataset. Moreover, we illustrated the potential usages of the STC in several use cases in the context of morphogenesis. The evaluation of the STH in a context of embryology with non-experts and domain experts showed that the STC visualization presents a number of benefits with respect to traditional visualizations, especially for the detection of events for which the identification required information for all or most of the temporal range. The STH approach could pave the way to new types of visualization and interaction methods for 3D spatio-temporal data, and we believe that such tools will help the adoption of VR technologies for data visualization.
<figure><img src="image_35.png" /><figcaption>Figure 3.10 – Examples of STCs based on 3 different base planes for the capture, on each example dataset presented in part 3.5. The top image is extracted from the shoot-apical meristem dataset, colormapped with numerical values of stress magnitude. The bottom images are extracted from the C. elegans embryo dataset. Each color represents the successive divisions of the cells present at $t = t_1$.</figcaption></figure>
# TIMELINE DESIGN SPACE FOR IMMERSIVE
EXPLORATION OF SPATIAL 3D
TEMPORAL DATA

<table><tbody><tr><td>4.1</td><td>A Timeline in a 3D Environment</td><td>74</td></tr><tr><td>4.1.1</td><td>Extending the 2D Timeline Design Space</td><td>75</td></tr><tr><td>4.1.2</td><td>Interacting with 3D Timelines</td><td>81</td></tr><tr><td>4.1.3</td><td>Choosing a Timeline Design</td><td>83</td></tr><tr><td>4.2</td><td>Evaluation</td><td>84</td></tr><tr><td>4.2.1</td><td>Dataset and Task Design</td><td>84</td></tr><tr><td>4.2.2</td><td>Hypotheses</td><td>85</td></tr><tr><td>4.2.3</td><td>Apparatus and Participants</td><td>86</td></tr><tr><td>4.2.4</td><td>Independent and Dependent Variables</td><td>86</td></tr><tr><td>4.2.5</td><td>Experimental Protocol</td><td>87</td></tr><tr><td>4.2.6</td><td>Results</td><td>87</td></tr><tr><td>4.2.7</td><td>Discussion</td><td>91</td></tr><tr><td>4.3</td><td>3D Timelines to Explore 3D Temporal Imaging</td><td>91</td></tr><tr><td>4.3.1</td><td>Application and Timeline Design</td><td>92</td></tr><tr><td>4.3.2</td><td>Finding Regions of Interests in Large Datasets</td><td>93</td></tr><tr><td>4.3.3</td><td>Comparing Objects</td><td>95</td></tr><tr><td>4.3.4</td><td>Fusion and Fission</td><td>95</td></tr><tr><td>4.3.5</td><td>Expert Feedback</td><td>96</td></tr><tr><td>4.4</td><td>Conclusion</td><td>97</td></tr></tbody></table>
This chapter approaches the issue of visualizing 3D temporal regions of interest. The objective is to be able to define a zone of the 3D temporal data, bounded both spatially and temporally, with a range adapted to a direct observation by the analyst.

To tackle this issue, we describe in this chapter an extended timeline design space for 3D environment, adapted to the visualization of 3D temporal data. This design space is inspired from the design space for 2D timelines described by Brehmer et al. [111]. Our design space extends the original one, adapting the related design choices to the additional workspace dimension. We describe how 3D timelines can partition and structure 3D temporal data, represented as 4D objects, in the 3D environment.

Moreover, due to the specific characteristics of interaction of immersive contexts and in order to take full advantage of 3D timelines, we discuss the basic set of interaction techniques to explore and manipulate them. We also propose additional interaction processes to tackle specific tasks of exploration and analysis more efficiently.

In order to explore the potential benefits of 3D timelines, we conducted two evaluations. The first evaluation, was a formal summative evaluation, which explored how 3D timelines could improve data exploration tasks. For this evaluation, mainly VR experts were considered, and we mainly focused on performance and usability. The second evaluation, was a qualitative evaluation conducted in collaboration with experts in biology. The objective was to gather feedback on how 3D timelines could be included in their analysis workflow, precisely, on how they could take advantage of 3D timelines to ease the exploration and validation process on their 3D temporal imaging data.

In summary, the main contributions of this chapter are as follows:

- A design space for 3D timelines, displayable in 3D environments and adapted for the visualization of 3D temporal data

- 3D interaction and exploration processes supporting complex analytical tasks in such visualization

- A formal evaluation of the benefits of 3D timelines in general visualization tasks, as well as feedback from biology experts giving insights on concrete use of the method.

## 4.1 A Timeline in a 3D Environment

This section introduces our design space for 3D timelines, which extends the 2D timeline design space [111] for 3D environments, and notably immersive environments. The extension
<figure><img src="image_36.png" /><figcaption>Figure 4.1 – Illustration of the proposed 3D timeline design space for immersive exploration of time-varying spatial 3D data. The first row showcases different timeline representations, i.e. guiding curve of the timeline; while the second row showcases different support configurations. The rightmost image illustrates the use of an helicoid representation for the exploration of an time-varying embryo imaging dataset. The viridis colormap is used to encode the elongation ratio of cells involved in morphogenetic movements.</figcaption></figure>

considers the additional spatial dimension in which data can be laid enabling the direct display
of the temporal dimension of the data.

As mentioned in the introduction, we focus on S4D datasets. We assume that the temporal
component of such dataset is discrete, or at least it can be discretized. As such, we can construct
a timeline in which every time point is populated by a **3D snapshot**, i.e. the state of the 3D
temporal dataset at an instant. We consider that some datasets are composed of a set of objects,
i.e. shapes with the own semantic value, and define, assuming sufficient tracking information,
**4D objects** that span along multiple time points. These objects can be defined by segmentation
process for imaging data [77], or by construction in a surface-based 3D model [133, 134].

Finally, in order to exploit the interaction capabilities provided by IVR systems and to take
advantage of the 3D timelines, we detail the set of interaction tasks needed for their exploration.

4.1.1 Extending the 2D Timeline Design Space

As presented in Section 2.2.4, Brehmer et al. [111] proposed a design space for 2D timelines
with the three following dimensions: *scale*, *layout* and *representation*. In this part, we adapt the
design space to 3D timelines. The two first characteristics are mainly similar for 2D and 3D
timelines, we added those parts for completeness of the design space, also adding a discussion for
3D temporal data. Then, we extended the design space by adapting the *representation* dimension
choices, yet also proposing other types of guiding curves adapted to a 3D environment. Finally,
we introduce a fourth dimension to this design space, as a result of using a 3D display, which
describes the supporting plane on which timeline branches are laid: the *support* dimension.

**Scale: Displaying Time**

The *scale* dimension represents the relation between the displayed distance and the temporal
distance. Although events will be laid in 3D space, this relation will remain. Therefore, the *scale*
dimension remains characterized as defined by Brehmer et al. [111]:

- A **chronological** scale spaces time points according to the actual temporal distance, either **linearly** or **logarithmically**, depending on the distribution of the events.

- A **relative** scale arranges the time points according to a baseline event. It can notably be used to compare multiple timelines regarding this event.

- With a **sequential** scale, the distance between time points is fixed and does not correspond to the temporal distance. A similar approach consists in encoding the duration between time points on the timeline as well.

**Layout: Arranging Time Points**

The layout dimension corresponds to how the timeline is partitioned in the display. Brehmer [111]
describes several characteristics of a 2D timeline layout. A first main characteristic of a layout is
whether it is **segmented** or not. Segmentation is used to cope with spatial organization issues,
such as the limitation in size of a 2D display, but also for analytic purposes, for example to
separate the timeline to make some periodic events more salient. The timeline layout can also be
qualified as **unified** or **faceted**, i.e. showing a timeline with one or several lines; the latter can be
useful for comparing data from different sources over time. Whether derived from segmentation
or faceting, we refer to timeline parts as *branches*.

These two characteristics can be used to describe 3D timelines as well. Similar factors are
taken into account for the choice in layout. Segmentation helps in solving issues of juxtaposition
of visualization and optimization of the large workspace offered by 3D environments. Its
analytical purpose is notably relevant in case of periodic data. On the other hand, the choice
between faceted or unified layout relies on the nature of the data and use case. A unified timeline
design is more suited to focus on the dynamic features of one region of interest or group of
objects, while a faceted layout allows the comparison of several ones. Faceted layouts are also
adapted to compare of multiple datasets, as in such cases, the most common task is comparison.
In any case, faceting or segmenting the layout tends to clutter the workspace. Attention should
be given to properly indicate the origin and characteristics of each facet or segment as not to
overwhelm the user.

**Representation: Shaping the Timeline**

The representation of a timeline corresponds to the shape, the guiding curve along which are placed the time points. This curve is therefore an equivalent of a time axis. In most cases, timelines are represented linearly [111], as for history timelines. This type of representation maintains a single orientation, often horizontal and following reading direction, and make for a good timeline readability [116]. Other designs use radial representation, to emphasize cyclic repetitions, or spirals, often used to get compact visualizations. Depending on the goal of the author of the timeline, some other types of representation can be designed for aesthetic, density or pedagogic purposes. From these representations used in 2D designs, we propose representations that take advantage of the 3D environment. First, the linear representation could be extended to a **3D curve**. Three main possibilities emerge:

**Flat curve.** The most direct extension of the linear representation in 2D would be to use a simple flat curve, as seen in Figure 4.2-A. In terms of representation, there is no limit to the amount of time points displayable in the environment. The main drawback is that the further the curve goes, the harder it is to reach or interact with the time points.

**Convex curve.** This representation displays the time points around the user. Several types of curves could be considered. An arc of a circle centered on the user would place all time points at the same distance, enabling the same capacity of interaction and observation for each time point. Such curve is shown in Figure 4.2-B. This counters the primary issue of the flat curve representation, yet the radius of the circle will increase with the amount of time points, limiting the proximity required for interaction. To compromise between proximity and amount of time points, the user could be brought closer from a part of the arc of circle. Another approach would be to use a parabolic curve, as in Figure 4.2-C. In both those cases, a local temporal context is brought very close from the user for interaction and observation, yet the other time points remain at a reasonable distance.

**Concave curve.** As opposed to the previous ones, a concave curve can be used to display close from the user a few time points, and sending the other ones that are further in time, even further in the environment. This representation emphasizes a local temporal context, frees some radial
<figure><img src="image_37.png" /><figcaption>Figure 4.2 – Third person and top views of options for the representation dimension of the 3D timeline design space. These illustrations show **A)** linear, **B)** convex arc of circle, **C)** convex parabola, **D)** concave parabola, **E)** helicoid and **F)** spherical representations.</figcaption></figure>
space around the user and decreases the density of information in their field of view. In this
sense, the amount of time points displayed is potentially unlimited, yet harder to access. In a
similar way to convex curves, concave curves could rely on parabolas, as in Figure 4.2-D, or arc
of circles, depending on the amount of time points that need to be close to the user.

<figure><img src="image_38.png" /><figcaption>Figure 4.3 – Third person and top views of options for the representation dimension of the 3D timeline design space. These illustrations show **A)** helicoid and **B)** spherical representations.</figcaption></figure>

Other designs for 2D timelines rely on circular or spiral representation. These representations
are particularly adapted to provide an insightful representation of periodicity in time. For
example, a spiral layout can be used to represent several occurrences of a same periodic event, as
demonstrated in Weber et al. work [168]. However, because of the varying radius of a spiral loop,
the number of periods has to be limited. Nonetheless, spirals are also intrinsically space-filling,
thus producing dense representations, and can be engaging or even playful. Consequently, we
propose two additional representations adapted to the design of a 3D timeline:

**Helicoid.** This representation coils the time points around a cylinder base. It can be centered on the user, displaying every time point close from the user, as shown in Figure 4.3-A. In a similar way to the 2D spiral representation, this shape invites the display of several occurrences
of periodic data, without the drawback of a varying radius between each period. For example, manipulating the scale dimension of the design, the circular periods of the helicoid could be aligned according to the occurrence of a particular event, for comparison purpose.

**Spherical.** Similarly, a spherical representation, as shown in Figure 4.3-B, coils around a sphere base. The time points are thus displayed around the user at the exact same distance, reducing distortions, in size of orientation for instance, implied by distance or perspective. This representation shares very similar advantages and drawbacks with the 2D spiral. It is aesthetically interesting, but the varying radius between two circular periods restrains its use for periodic data.

These two representations share several other characteristics. The exploration of the timeline is no longer horizontal as in the curve representation, but rather radial and vertical, allowing fast jumps in time. It is compact and takes full advantage of the 3D work space. However, due to the use of the spatial upward axis, it can limit the use of faceted layouts, as it could lead to confusion between the timeline branches juxtaposed and the next circle of the helicoid. Other representations using either different curves or volume could be used, that may be tailored for specific analysis use cases, for an aesthetic purpose or to create mnemonic designs. Changing the orientation in either axis can also be a way to explore original representations.

## Support: Exploiting the 3D Environment

In usual displays, timeline branches are most usually laid out on the display plane. As mentioned in part 2.2.4, a few works explored displaying timelines in 3D environments. Bee-docs [117] includes 3D perspective to animate and display the timeline at a different orientation for aesthetic purpose, while Kullberg et al. [119] even lays the timeline on the horizontal plane of a 3D environment, using the vertical plane to display the content of the time points.

As we propose to display 3D timelines in VR, we introduce an additional dimension, the *support*, to refer to the 3D shape that supports the timeline branches in such 3D environment. We describe here the main choices that will mostly influence the timeline design. This includes shape, size, position, and also count of supports on which the timeline branches are displayed. We propose the following choices for this dimension:

**Vertical plane.** The branches are laid on a plane vertically in front of the user, like a classic 2D display.
**Horizontal plane.** The branches are laid on a plane parallel to the floor. Horizontal head movements and field of view are usually preferred over vertical ones, thus this design choice could be beneficial compared to a vertical plane support in some cases, such as for comparison at one time point of multiple branches. The main drawback is occlusion, since branches are displayed one in front of the others.

**Multiple Planes.** As opposed to the previous design choices involving a single plane, here the branches are displayed on several ones. This can be useful to optimize application design, or separate clearly facets and segments.

**Cubic.** The branches are laid along multiple vertical or horizontal planes. Such layouts can be useful to organize branches along 2 parameters, much like a 2-dimensional array. While very meaningful in this purpose, this design is quite dense, which can be overwhelming for a user and cause difficulties in the exploration.

**Concentric cylinders.** The branches are displayed on multiple isoradial planes centered on the user. In terms of exploration, it is quite similar to horizontal planes, yet fills the space around the user, which can serve aesthetic purpose.

Each of these dimension choices are illustrated in Figure 4.1, second line, in order. While this dimension is unnecessary for timelines with a unified layout, these design choices can be useful for some specific use cases, for application design, notably in terms of workspace usage, or for aesthetics. Yet, some of these designs can also involve significant occlusion issues. Adapted navigation and application controls might be required when using these design choices.

### 4.1.2 Interacting with 3D Timelines

This section discusses the interaction methods required for the exploration and manipulation of a 3D timeline. First, we consider methods enabling the free exploration of the timeline, supporting consume and search tasks. Then, we consider tasks that allow the manipulation of the timeline, in order to modify the arrangement of the timeline. The discussed interactions consider that the user is using an immersive virtual reality setup (e.g. head-mounted display), in which the user can move physically to explore the virtual environment and that additional input capabilities are provided by hand-held devices (e.g. controller). For the sake of generalization, we will not
discuss how the interactions can be implemented, but which parameters (e.g. distance among
time points) could be modified.

The timeline is arranged in the environment in such a way as to have one time point, that we will call from now on *central time point* of the timeline, that is the closest from the user, as shown in Figure 4.6. Time points are identified by an indicator beneath the objects represented.

Exploring the Timeline

First, the user can physically move (e.g. walk) in order to approach the time point of interest. However, the range of time points that can be explored will be dependent on the shape of the timeline and the workspace of the virtual reality setup. Some timeline designs covering an important part of the work space around the user, for example when using convex curves or spherical representation, will enable the direct exploration of all time points without requiring a large physical workspace. In contrast, linear or parabolic representations can spread far away for the user, and faceted or segmented layouts might display data too high, limiting the number of time points that the user can access. To overcome this limitation, one solution is to consider virtual navigation methods [39], that enable the user to virtually (without physically moving) navigate. Potential good candidates are scene-in-hand methods [169] or virtual steering [170] which will allow users to reach time points arbitrarily far away from them. Yet, the choice in the virtual navigation method will depend on the implementation of the timeline, the data represented and the end user profile.

Finally, another alternative to enable a full exploration of the timeline is to move the timeline
along its time axis, thus changing the *central time point*. This can be done continuously by
scrolling the time points (e.g. using a temporal slider), or directly by selecting a time point from
afar (e.g. using a ray-based selection method [48]). These methods will minimize the user's
physical motion.

Manipulating the Timeline

There are numerous continuous parameters involved in the display of the timeline, such as
the space between time points, the center and radius of curvature for curved representations, the
height for helicoid representations or the distance between timeline branches for faceted and
segmented layouts. These values can strongly vary, notably depending on the amount time points
displayed in the timeline. For this reason, the user could be able to choose among pre-defined
configurations of timeline designs. Nonetheless, to increase the flexibility of the visualization,
the user should be able to adjust the parameters of the timeline display in order to better fit
their use-case requirements. The modification of these values can be done using graphical user
interfaces or specific 3D widgets [171]. Moreover, in order to control the information contained
in the environment level-of-detail options are also relevant. The user should be able to choose to
display only a proportion of the time points selected regularly along the timeline, and to directly
choose time zones to collapse or extend via manual selection.

Manipulating the Content of the Time Points

The final interaction category relies on how the content of the time points could be modified.
First, the 3D objects at each time point can be rotated and scaled, in order to enable the user to
explore them from different orientations. To ensure consistency, we advocate to link the rotations
and scale for all time points, thus obtaining a consistent orientation for all time points. Such
manipulation could be achieved using bi-manual manipulation methods [51].

Second, operations to reduce the information at each time point can also be considered. (A) Cut-away views defined by a clipping plane or a 3D volume (e.g. cube) operators. The user could define the position and orientation of the cutaway operator in the central time point, a corresponding cut-away view could be applied as well on each of the other object or group of objects, relatively to their barycenter. Standard 3D manipulation techniques can be used to control the cutaway operator [51]. (B) Object and value filtering operations. These operations will be highly dependent on the available data of the S3D. For example, if the data for each time point is composed by a set of 3D objects (e.g. different 3D meshes, segmented 3D volume), individual objects could be removed from the timeline by selecting them. Moreover, if the individual objects are linked with other objects at different time points (e.g. same object appearing at different time points), the selected objects and their linked objects could also be hidden. Finally, if additional categorical, ordinal or numerical attributes are available for each object (e.g. volume of a given object), filtering operations could also be defined (e.g. hide the objects with a volume lower than a threshold). These operations to reduce the displayed information could be of great relevance to reduce the amount of displayed information, reducing clutter. These interactions could be designed considering existing 3D selection methods [48] and 3D graphical user interfaces [171].

4.1.3 Choosing a Timeline Design

In this part, we propose a set of design criteria that will drive the design choices of 3D
timelines. Focusing on the constraints that can be introduced due to the attributes of the data
(topology, resolution, rendering method), application design constraints and more importantly
the narration, storytelling and expressiveness purpose, we identified the following key criteria:

**Preferred exploration.** Designs will favor exploration of the timeline simply through head rotation, or through virtual navigation and manipulation of the timeline.

**Limit of time points.** The amount of time points displayable objects easily discernible can be limited, depending on how the design partitions the workspace.

**Workspace usage and visual clutter.** Designs occupy more or less radial and vertical space around the user, which can cause occlusion and application design issues.

**Periodicity in the data.** Periodicity and temporal patterns are often important characteristics in data analysis, and some design choices, such as segmentation or helicoids can emphasize such characteristic.

**Aesthetics, originality, playfulness.** The attractiveness of a visualization can be crucial when trying to convey information. The design can be adapted to the use case and the public for this purpose.

Innovative designs might create more adapted visualizations or more catchy aesthetics. Yet, it should be noted that exotic designs might also be more difficult to interpret, as stated by Brehmer et al. [111], it is thus recommended to limit the unfamiliar choices in the design of the timeline. On the other hand, in some cases design choices are interlocked together between several dimensions. For example, a helicoid representation strongly implies the use of a cylinder support.

**4.2 Evaluation**

The previous section discussed a wide range of potential 3D timeline representations, yet, from the design on itself, it remains unclear how users will leverage the 3D timeline to explore S4D data. Thus, we conducted a user evaluation in order to assess the potential benefits of 3D timelines for the exploration of S4D data. Our objective was to assess representative 3D timeline designs to explore how users will perform exploration tasks.
4.2.1 Dataset and Task Design

<figure><img src="image_39.png" /><figcaption>Figure 4.4 – Environment for the user experiment. The right image shows the **Locating time point** task in the *No timeline* condition; the middle one shows the **Counting occurrences** task in the *Curved* condition; the left one shows the **Finding a global maximum** task in the *Helicoid* condition, where the objects are colored using the *viridis* colormap.</figcaption></figure>

The S4D dataset used in the evaluation was generated procedurally in order to keep the experiment accessible but also to limit the potential bias between users in the approach of complex S4D and in the manipulation of 3D models (see Figure 4.4). As such, we limited the number of time points and objects ensuring that users will be able to apprehend the dataset. The objects were represented by spheres, and their position did not change over time. The dataset was composed of 6 objects evolving over time, and depending on the task (see below) having either 40 or 80 time points. The size of the objects increased monotonously, so that users could keep points of reference when exploring over time. In order to enrich the dataset, objects were annotated with 2 types of information. First, a categorical information describing the affiliation of each object to one of 5 **groups**, which could change over time (see Figure 4.4, center), was encoded with different colors. We introduced randomly several occurrences of a specific pattern in the group information, on which we can evaluate success and error rates for the related task. Second, a continuous information (**float value**) evolving over time, encoded with the *viridis* color map (see Figure 4.4, right). This value was generated by adding 4 random Gaussian functions, one on each 20-time-point segment of the dataset.

During the experiment, the objective was to let participants to explore the dataset and perform different tasks. We defined four data exploration tasks that corresponded to common tasks for the exploration of time-varying data: locating a time point, counting occurrences of a particular event, finding temporal patterns or finding a global maximum. For the **location task**, a dataset of 80 time points was used. Participants had to find random defined time points between 0 and 79. They had to reach the indicated time point by using the time exploration technique available,
described below. For the **counting task**, a dataset of 40 time points, annotated with the group information was used. Participants had to count the number of objects of a given group in all time points. To limit the length of the task, it was limited to 3 minutes. For the **pattern task**, a dataset of 40 time points, annotated with the group information was used. Participants had to count the appearances of a specific 3-time-point-long temporal pattern that objects might follow in the 40 time points displayed. The task was limited to 3 minutes of exploration. Finally, for the **maximum task**, a dataset of 80 time points, annotated with the continuous information was used. Participants must find the time point exhibiting the maximum value. For simplicity, all objects from the same time point had the same value.

### 4.2.2 Hypotheses

The two first tasks (location and counting) did not require temporal context. Indeed, the information to find was punctual in time, thus any method allowing scrolling through time could be sufficient. Thus, we did not expect timelines to be particularly efficient for this type of task. Although having too much information can be overwhelming for the user, we argue that the added temporal context could still be beneficial. On the other hand, the two other tasks (pattern and maximum) required information from previous or following time points. As such, we expected better outcomes using timelines. The result could also be strongly dependent on the timeline design, as how the information is displayed and accessed. We expect that denser timeline designs would allow to access larger temporal context and could thus perform well when it is required, while also being useful to handle higher amounts of time points. Accordingly, we chose different timeline designs, which are detailed in Section 4.2.4. In summary, our hypotheses were:

**H1a** Tasks that did not require exploring the temporal context will be achieved at comparable performance with or without timeline.

**H1b** Tasks that did not require exploring the temporal context will be achieved more comfortably with a timeline.

**H2** Tasks that require exploring the temporal context will be achieved better using a timeline.

**H3** Denser timeline designs will more adequately achieve the tasks.

### 4.2.3 Apparatus and Participants

The material used for the tests and experiences described throughout this chapter was as follows: a HTC Vive Pro HMD with 2 Vive controllers, a PC with Windows 10, an Intel Xeon W-
2104 CPU (4 cores, 3.2 GHz base frequency) and a Nvidia RTX 2080 GPU. The VR application runs on Unity 2019.4.5f1.

This experiment included 21 participants, 5 women and 16 men, aged from 22 to 42, mostly recruited from the local laboratory. Two of them had low experience with VR, the others were expert VR users.

### 4.2.4 Independent and Dependent Variables

The experiment followed a within-subject design in which participants had to perform the different tasks using 3 different visual conditions. The first one was the *No timeline* condition, in which only a snapshot of the dataset at an instant was displayed. The two others used timeline designs which took advantage of the 3D workspace, a helicoid unified and a circular convex unified timeline. They will be referred to as respectively *Helicoid* and *Curved* conditions.

For all tasks, we measured the time to complete the task, as well as the movements and the time using the different available VR interaction tools. For the locate task, we also computed the difference between the selected time point with the target time point. For the counting task, we also computed the error rate. For the pattern task, we further evaluated the precision and recall. Finally, for the maximum tasks, we evaluated the accuracy.

### 4.2.5 Experimental Protocol

At the beginning of the experiment, participants signed a consent form and filled a demographic information questionnaire. Participants received an explanation of the VR system and the available interactions. We mainly considered interactions that allowed the users to achieve the designed tasks. As such, object selection and task validation were done using a virtual laser pointer, and object manipulation was done using a bi-manual control. Time exploration in the *No timeline* condition was done by scrolling on the controller touchpad or using a direct manipulation on a slider (see Figure 4.4, left). In the other conditions, scrolling made the timeline move along its directing curve, and a direct selection based on a gaze and click interaction was available, users had to orient their head towards the time point and press a controller button. We let them get used to the tools and application layout for 10 minutes. The participant then proceeds to the tasks in order, in which the display condition was counterbalanced using a Latin Square.

After each condition, participants were asked to fill the SUS [172] questionnaire to evaluate usability, and the CSQ-VR [173] and FMSS [174] questionnaires to assess cybersickness. Finally,
to subjectively assess the timelines, for each task and condition we asked the following questions: "What condition felt the most comfortable for this task?" and "What condition seemed the most efficient for this task?".

### 4.2.6 Results

For each task, we obtained a total of 63 observations, characterized by the 3-level between-subjects factor **Order** and 3-level within-subjects factor **Condition**. We used repeated measures two-way ANOVA to analyze the results. When the normality assumption was violated (Shapiro-Wilk normality test) we transform the data using the Aligned Rank Transform (ART) [175] prior conducting the ANOVA analysis. Post-hoc pairwise tests with Bonferroni correction ($\alpha = 0.05$) were used when needed.

#### Main Effects for Condition

**Locate task.** This task was the first of each condition, thus we removed the two first trials of each participant, which might have induced too much variability, as users were still getting used to the method, and averaged the 8 other observations. The users made almost no errors (($M=0.018$; $SD=0.077$)), showing that participants were able to perform the task accurately. In addition, the ART ANOVA order and condition vs **average time** showed a significant effect of the condition (*F*(2,36) = 13.87; *p* < 0.001). Post-hoc tests validated lower times during the *No timeline* condition (($M=7.12$; $SD=1.86$)) compared to the two others (*Curved* ($M=8.75$; $SD=2.38$; *p* < 0.001), *Helicoid* ($M=9.57$; $SD=4.13$; *p* < 0.001)), yet no significant difference between the latter. This tend to **invalidate H1a**, since performance was lower using timelines.

**Counting Task.** While the task was limited in time, the limit was never reached. The ART ANOVA order and condition vs **time** showed a significant effect of the condition ($F$(2,36) = 11.99; *p* < 0.001). Post-hoc tests validated that the *No timeline* condition (($M=106.8$; $SD=40.4$)) gave significantly higher times than the two others (*Curved* ($M=76.8$; $SD=42.2$; *p* < 0.01), *Helicoid* ($M=68.4$; $SD=32.7$; *p* < 0.001)), yet no significant difference between the latter. This tends to **invalidate H1a**, since performance was higher with timelines. The ART ANOVA order and condition vs error rate gave no significant results.

**Pattern task.** Participants were able to achieve the task with almost no errors showing that all methods allowed to perform the task (($M=0.993$; $SD=0.025$)). Moreover, the ANOVA order and condition vs **recall** showed a significant condition effect (*F*(1.63,29.36) = 6.23; *p* < 0.01; $\eta_p^2$=0.26). Post-hoc tests validated a significantly lower recall between the *No timeline* condition (($M=0.652$;
$SD=.201)$) and the two others (*Curved* ($M=0.798$; $SD=0.15$; $p < 0.05$), *Helicoid* ($M=0.818$; $SD=0.17$; $p < 0.01$)), yet no significant difference between the latter was found. This tends to **support H2**.

The ART ANOVA order and condition vs **time** showed a significant main effect on condition ($F(2,36) = 12.5; p < 0.001$). Post-hoc tests showed significantly lower times for the *Helicoid* condition (($M=157$; $SD=38.8$)) against the *No timeline* condition (($M=192$; $SD=20.9$; $p < 0.001$)), and also against the *Curved* condition (($M=180$; $SD=28.5$; $p < 0.05$)), yet no significant difference between the latter. This tends to **support H3**.

**Maximum task.** We averaged the measured values of each of the 5 trials of each participant on each condition. The measured error rate was always close to 1 ($M = 0.997$, Min=0.98), participants were able to accurately perform the task with all conditions. Furthermore, the ANOVA order and condition vs **average time** showed a significant effect of the condition ($F(1.69,30.43) = 8.27; p < 0.01$; $\eta_p^2=0.31$). Post-hoc validated significantly higher times for the *No timeline* condition (($M=32.5$; $SD=10.54$)) than the *Helicoid* condition (($M=26.1$; $SD=9.12$; $p < 0.05$)) and the *Curved* condition (($M=22.8$; $SD=9.19$; $p < 0.001$)). This tends to **support H2**.

<figure><img src="image_40.png" /><figcaption>Figure 4.5 – Subjective results for each timeline design and task. (a) Ranking results for the question “What condition felt the most comfortable for this task?”. (b) Subjective results for each condition.</figcaption></figure>
<figure><img src="image_41.png" /><figcaption>Figure 4.5 – Subjective results for each timeline design and task. (b) Ranking results for the question “What condition seemed the most efficient for this task?”</figcaption></figure>

## Questionnaires.

According to the CSQ-VR and FMSS results (respectively (*M*=0.76/36; *SD*=0.85) and (*M*=1.7/20; *SD*=1.5)), participants did not suffer a significant increase on cybersickness. The ANOVA order and condition for the SUS scores showed a significant effect of condition (*p* < 0.05). Post-hoc tests only showed a significant effect for the *Curved* condition ((*M*=81.9; *SD*=11.7)) against the *No timeline* condition ((*M*=70.4; *SD*=12.2; *p* < 0.01)).

Figure 4.5 provides a summary for comfort ((a)) and efficiency ((b)) assessment for each condition. In terms of comfort, the users generally preferred the *Curved* condition for every task. The *No timeline* condition was second only in the Timepoint task, for the other tasks the *Helicoid* condition was preferred. About subjective estimation of efficiency, the *Curved* condition was still preferred in all tasks, except in the Finding Timepoint task, for which the *No timeline* condition was arguably even. However, the second place was more clearly attributed to the *Helicoid* condition for this criterion.
Order and Interaction Effects

We analyzed potential order and interaction effects for each test. For most, the order effect
was not significant, or could not be confirmed by post-hoc tests. However, we did find significant
effects for some cases.

The ART ANOVA order and condition vs **average time** on the **Finding Timepoint** task showed a significant order effect ($F(2,18) = 5.00; p < 0.05$), which was validated by a post-hoc test as a difference when the *Helicoid* was the first condition (($M=10.05; SD=4.26$)) and when it was the last (($M=7.1; SD=1.24$)), $p < 0.05$. Judging from the important standard deviation, we hypothesize that starting with the *Helicoid* might have been overwhelming to some users during their familiarization with the system, causing higher achieving times in this first task of the experiment. This interpretation is further supported as the *Helicoid* was judged the least comfortable condition for this task.

The ANOVA order and condition vs **average time** on the **maximum** task showed a significant interaction effect ($F(1.69,30.43) = 8.27; p < 0.01; \eta_p^2=0.31$). Post-hoc tests confirmed that the results from the *No timeline* condition as first condition of a participant were significantly lower than any other case when the participant executes the task in the second timeline condition. This can be explained by a learning effect on the use of timelines, which becomes significant when achieving the task with a second design. Though, the condition effect showed that both *Curved* and *Helicoid* conditions obtained significantly lower times regardless of order, so this did not alter our conclusion on H2.

4.2.7 Discussion

The results from the two first tasks, that did not essentially require temporal context, were
mitigated, so we must **reject H1a**. The results from the questionnaire however **support H1b**, as
users did report preference for timelines for these tasks. Several results from the tasks of finding
patterns and global maximum **support H2, that we can accept**. Finally, results from the task of
finding patterns **supports H3**.

Overall, these results showed significant benefits of using timeline designs over classic
visualization for temporal exploration tasks, in terms of time and quality of completion, yet also
in user experience. Nonetheless, we expected a more significant difference between the two
timeline designs, notably regarding the different number of time points between the tasks. We
explain this lack of differences due to the choice of using a Latin Square ordering method: in two
out of three orderings, the *Curved* condition was tested after the *Helicoid*. As learning effects did