# On-Body Tangible Interaction: Using the Body to Support Tangible Manipulations for Immersive Environments

Houssem Eddine Saïdi, Marcos Serrano, Pourang Irani, Christophe Hurter, Emmanuel Dubois

<figure><img src="image_1.png" /></figure>

## To cite this version:

Houssem Eddine Saïdi, Marcos Serrano, Pourang Irani, Christophe Hurter, Emmanuel Dubois. On-Body Tangible Interaction: Using the Body to Support Tangible Manipulations for Immersive Environments. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.471-492, 10.1007/978-3-030-29390-1_26. hal-02278309

HAL Id: hal-02278309

https://enac.hal.science/hal-02278309

Submitted on 24 Jun 2020

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# On-Body Tangible Interaction: Using the Body to Support Tangible Manipulations for Immersive Environments

Houssem Saidi<fnref n="1" />(✉), Marcos Serrano<fnref n="1" />, Pourang Irani<fnref n="2" />, Christophe Hurter<fnref n="3" />, and Emmanuel Dubois<fnref n="1" />

¹ IRIT, University of Toulouse, Toulouse, France
Houssem.saidi@irit.fr

² Department of Computer Science, University of Manitoba, Winnipeg, Canada

³ ENAC, French Civil Aviation University, Toulouse, France

**Abstract.** Recent technological advances in immersive devices open up many opportunities for users to visualize data in their environments. However, current interactive solutions fail at providing a convenient approach to manipulate such complex immersive visualizations. In this article, we present a new approach to interact in these environments, that we call On-Body Tangible interaction (OBT): using the body to physically support the manipulation of an input device. The use of the body to support the interaction allows the user to move in his environment and avoid the inherent fatigue of mid-air interactions. In this paper, we explore the use of a rolling device, which fits well on-body interaction thanks to its form factor and offers enough degrees of freedom (DoF) for data manipulation. We first propose a new design space for OBT interactions, and specifically on the forearm. Then we validate the feasibility of such an approach through an experiment aimed at establishing the range, stability and comfort of gestures performed with the device on the forearm. Our results reveal that on-body tangible interaction on the forearm is stable and offers multiple DoFs with little fatigue. We illustrate the benefits of our approach through sample applications where OBT interactions are used to select and execute space-time cube operations.

**Keywords:** Immersive environment · On-body interaction · Tangible interaction

## 1 Introduction

Immersive systems such as the Hololens<fnref n="1" />, MetaVision<fnref n="2" /> or Moverio<fnref n="3" /> allow the user to display data and visualizations directly on the physical world by attaching them to a fixed physical anchor; we hereafter refer to these as immersive visualizations.

**Electronic supplementary material** The online version of this chapter (https://doi.org/10.1007/978-3-030-29390-1_26) contains supplementary material, which is available to authorized users.

<footnote n="1">¹ https://www.microsoft.com/en-us/hololens.</footnote>

<footnote n="2">² http://www.metavision.com/.</footnote>

<footnote n="3">³ https://epson.com/moverio-augmented-reality.</footnote>
Immersive visualization offers a large set of advantages, such as spatialization of data, physical exploration of data, wide display area, etc. It thus constitutes a compelling alternative to visualizing multidimensional data on 2D displays. However, data exploration involves many tasks, such as filter, select, adjust or annotate the dataset, which are not entirely covered in existing solutions.

Indeed, a large interaction vocabulary is required and approaches based on mouse, touch and mid-air interaction fail to offer enough degrees of freedom (DoF) [17, 34, 39]; other solutions are often ambiguous and tiring (especially mid-air gestures [7, 34, 35]); finally many restrict the user's interaction to a defined place, usually a desktop, to use the input device (3D mouse or other [30]). The challenge then is to provide an interactive solution for immersive environments that preserves the freedom of movement of mid-air interaction and the DoFs of tangible interaction.

To this end, we propose to explore On-Body Tangible (OBT) interactions, i.e. a new approach using the body as a physical support for manipulating a tangible object to interact with immersive environments. This approach combines: (1) the use of a connected physical object, that offers multiple degrees of freedom and, (2) the use of the body to guide the physical manipulations of the tangible object and exploit the user's proprioception (i.e. sensing its own body parts). While the first aspect offers the multi-DoF of tangibles, the second aspect ensures that the solution can be used anywhere and can potentially contribute to reduce muscle fatigue, two inherent characteristics of mid-air interactions.

Our contribution is both conceptual and experimental. First, we propose a design space that encompasses the physical properties of the body (support) and the interactions that can be performed on it. Second, we validate the fundamental advantages of the OBT interactive approach (the feasibility of the gestures, their amplitudes, the comfort and fatigue related to the interaction) through a three-part experiment: the first part of the study focuses on on-body translations of the object, the second on rotations and the third evaluates rolls. Finally, we discuss the advantages and disadvantages of on-body tangible interactions, and we present two illustrative applications where on-body tangible interaction is used to select and execute space-time cube operations.

# 2 Related Work

This section synthesizes the main interaction solutions proposed for immersive environments, i.e. tactile, mid-air, tangible and on-body interactions.

## 2.1 Tactile Approaches

Whether the interaction is integrated directly on the immersive device itself [39], or deported to an external device (mobile device, interactive tabletop) [14, 31], different forms of tactile interaction have been applied to manipulate immersive visualizations [18, 20, 31, 32, 39]. However, Bergé's [7] and Besançon et al. [10] works established that tactile interaction is neither the most efficient, nor the preferred interaction
modality for 3D data manipulation. Moreover, tactile interaction requires a dedicated surface [7], which can divert the attention of the user from the task to perform, and may constrain the movements if using fixed tactile display [30, 32].

## 2.2 Mid-Air Interactions

Mid-air interaction offers obvious advantages for immersive environments, such as unconstrained mobility, and are light and easy to perform [13, 38, 43, 51]. These advantages led to multiple developments, such as the interactive immersive experience proposed by Benko et al. [5] where users could employ mid-air gestures to interact with an augmented dome. Interestingly, authors evaluated the mid-air gestures and found that the proposed interaction was simple to perform but understanding how to perform it was not easy. Other well-known problems with mid-air interactions are fatigue [12], ambiguity when no appropriate feedback is provided [7] and discoverability, i.e. finding the right mid-air interaction for a given task/action [7].

## 2.3 Tangible Objects

Tangible interaction provides multiple degrees of freedom for interaction with immersive environments [1, 8, 50]. Its physicality suggests how to perform the interaction, minimizing the learning process. Used for the design of DNA (Deoxyribonucleic acid) components, tangible 3D input devices in immersive interface appear to be more satisfying than a 2D interface [41]: the direct 3D mappings between the 3D space and the tangible device manipulation facilitates the understanding. Besançon et al. [10] also established that tangible interactions perform better than its mouse and tactile counterparts, and that tangible's affordance limits the need for a learning phase. However, they pointed out that the mouse was more precise than tangibles used in mid-air. Interestingly, TUI (Tangible user interface) for have been tested in many different contexts such as interactive visualization of thin fiber structures [26], 3D visualization [15], exploration of volumetric data [25], etc.

Along the same lines, multi-DoF devices have been explored through the augmentation of a mouse in order to overcome the mere 2 degrees of freedom offered. The Rockin' Mouse [3] and the VideoMouse [23] are based on a semi-rounded shapes that allows tilting the device but do not totally support compound gestures. The Roly-Poly Mouse (RPM) [37] uses a completely rounded bottom. It has been shown to provide larger amplitude of movement than previous tilting devices and efficient in 3D contexts and Multi-Display Environments [40].

## 2.4 On-Body Interaction Techniques

As opposed to the previous solutions, on-body interactions have been scarcely used in immersive environments. Serrano et al. [42] explored hand-to-face gestures arguing that they are well suited for HWDs. Dobbelstein et al. [17] proposed the use of the belt
as a tactile surface to interact with HWDs. Wang et al. [47] proposed PalmType, an interaction technique that enables users to type with their fingers on the palm to input text in smart-glasses. These works illustrate that on-body interaction techniques allow eyes-free interactions by exploiting the proprioception capabilities of users and do not divert their attention from the task at hand. However, the explored approaches offer a limited set of possible gestures, making it unsuitable for complex data visualization.

# 3 On-Body Tangible Interactions

In this section, we present a new interaction approach for immersive environments based on the use of the body to support tangible interactions. We detail the main requirements to interact with immersive environments, our choices of body parts and tangible objects to use, before presenting the design space.

## 3.1 Requirements for Immersive Environments

We illustrate the requirements for immersive environments through the specific case of immersive visualization. There are different types of immersive visualizations according to the data to visualize, which ranges from simple 3D objects to complex multidimensional data. All these immersive visualizations share a set of basic interaction requirements:

* Unconstrained mobility (R1): it has been demonstrated that the spatial exploration of data allows for a better spatial understanding of the visualization [27]. The user can have an overview of the visualization from afar, a more detailed view by getting closer, as well as different views from different angles [27]. It is thus important that the interaction techniques do not constrain the mobility of the user.

* Multiple degrees of freedom (R2): the multidimensional nature of data in immersive systems requires enough degrees of freedom to tackle its manipulation [4] through multiple commands (filter, sort, annotate, highlight, clusterize, aggregate, etc.).

* Visual access (R3): the interaction techniques should not occult the data visualization. They should also allow the user to interact with data without diverting his attention from the visualization [27].

* Stability (R4): the interaction techniques should offer enough stability to properly tackle data visualization tasks in mobility.

While these requirements apply for immersive visualizations, they hold true for several other interaction environments, such as VR caves, augmented maintenance, gaming, etc. Therefore, the solution presented in this paper can be applied to such other interaction environments. We leave the investigation of these other environments for future work.
## 3.2 Combining Tangible and On-Body Interaction

After clarifying the requirements for immersive visualizations, we now describe and justify the properties of On-Body Tangible interaction: the tangible object used in the interaction, the body part used to support it and their combination.

**Tangible Object.** Among the plethora of existing tangible objects introduced in the literature, we decided to use a semi-spherical object. The rationale behind this choice is manifold. (1) It has been demonstrated [37] that such a form factor offers up to six degrees of freedom (R2), in the form of three types of manipulations (translations, rotations and rolls), facilitating the manipulation of multidimensional data [37]. (2) As opposed to other forms (that have flat surfaces), the contact of a rounded object with the interaction surface is minimal (i.e. a point), which will easily adapt to the outline of any part of the body. Other small sharp objects (such as a pen) have also a minimal contact point, but their sharpness is detrimental as they may hang onto the clothes or hurt on the skin. (3) As opposed to objects that includes flat surfaces, the manipulation of a round object remains continuous, does not interrupt the interaction flow and do not artificially promote the use and need of modes in the application. (4) The choice of a semi-spherical form rather than full spherical one is motivated by the fact that it is easier to hold [37].

**Body-Parts Used to Support the Interaction.** Many research works have focused on interaction on or with the body [9, 21, 28, 46, 49]. Arms and hands were the preferred body parts in most works. These body parts offer numerous advantages: they are easily accessible for interaction; they are in the user's field of vision and generate less social discomfort than other body parts [28, 45]. Karrer et al. experimentally identified the most appropriate region of the body to perform interaction with clothes [28]. Among their observations, the non-dominant arm as well as the hip are the preferred body-parts for interaction. Other parts of the body, such as the stomach and legs, have been rejected for social or personal reasons in the same study.

We decided to focus on the forearm of the non-dominant arm to support the interaction for several reasons. It is an always-available physical support that favors physical exploration of data and it does not constrain the movement of the user (R1). Thanks to the body's natural capacity to sense its own body parts (proprioception), the user can perform tangible interactions on the body without having to switch his attention from the data visualization to the interaction tool (R3). Using the forearm offers a large surface on which the tangible object can be laid, and it is effortlessly accessible by the dominant hand as opposed to the upper arm which needs a consequent effort to be touched by the dominant hand (R1, R2). Furthermore, its combination with a tangible object (which use can be extended to mid-air manipulations, as suggested by the Lift-Up option in [40]) does not constrain the user's movement (R1). Finally, several stable poses can be adopted with the forearm (R4), which increases the range of possibilities (R2) (Fig. 1).
<figure><img src="image_3.png" /><figcaption>Fig. 1. Design Space for on-body tangible interaction on the forearm</figcaption></figure>

## 3.3 Mapping the Input Interaction Space with the Data Visualization Space

While the user's gestures are performed in a 3D physical environment, they trigger actions in a 3D virtual environment. It is therefore important to choose the right frame of reference for the interaction. The frame of reference can be allocentric (external: it can be world-centered, data centered...) or egocentric (relative to the body). In an egocentric frame of reference, the output of a given manipulation is determined by how it is performed with regards to the body, and will have the same effect regardless of the body position and orientation in the world. In our approach, we adopt an egocentric frame of reference to allow the user to interact from anywhere with geographically anchored data in the physical world [33]. Indeed, previous research found that a lack of a logical relationship between the manipulated data's frame of reference and the user's frame of reference can affect performances negatively [36, 48].

## 3.4 Design Space for On-Body Tangible Interaction

Based on the main design properties of OBT interactions introduced above, we now present a design space describing the possible use of the forearm as interaction support. It is composed of 3 dimensions: The Pose, the Place of motion and the Tangible action.

Restricting interaction to the forearm—more socially acceptable, accessible and with a large interaction area—has the disadvantage of reducing the available interaction vocabulary. To address this limitation and increase the interaction vocabulary, we
propose to consider the posture of the arm (Pose) as well as the use of sub-regions on
the forearm (places of motion) and the tangible action (TA) that can be applied.

**Pose (inPOS).** The pose describes the position of the forearm with respect to the user's body. We identified three main poses for the forearm. In the Vertical pose, the forearm is vertical, the hand points upwards. In the Forward pose, the forearm is perpendicular to the shoulders. In the Parallel pose, the forearm is parallel to the shoulders (Fig. 1). We chose these three poses because they are the most comfortable poses, the most accessible with the non-dominant hand and they are significantly different from each other, facilitating their distinction by both the system and the user.

**Place of Motion (POM).** The Place of motion represents the surface of the forearm on which the tangible object can be used. We identified two types of places: the first one extends over the length of the forearm, from the elbow to the wrist (length POM); the second one, called width POM, divide the forearm into several sub-regions. The number of sub-regions was defined according to the following criteria: (1) the sub-regions have to be large enough to accommodate interaction with the tangible object (diameter = 8 cm); (2) the sub-regions have to be easily distinguishable. Applying these criteria, we divided the forearm in three sub-regions: close to the Elbow (Elbow POM), in the middle of the forearm (Middle POM) or close to the wrist (Wrist POM) (Fig. 1). This results into 12 different interaction supports (3 poses × 4 places), which increases the possibilities of interactions exploiting the proprioception of the user and contributes to avoid the fatigue of a mid-air usage.

**Tangible Action (TA).** It stands for the physical actions that can be performed with the tangible object. The round shape of the tangible object offers three physical manipulations: translation, roll and rotate. Rotations and rolls along the length of the forearm (Length POM) were not considered. Indeed, they are performed with the tangible object motionless, in the same spot, thus, rendering them too similar to rotations and rolls on one of the other three places of motion. The following experiments will analyze the benefits and limits of each of them separately.

# 4 Study: Translations, Rotation, Rolls on the Forearm

We present a three parts experiment in which we explore the feasibility and potential of on-body tangible interaction in the different configurations described above. We explore the potential along three aspects: (1) the maximum range of motion for each of the three physical manipulations; (2) the stability of the interaction; and (3) the width of places of motions

We define the range of motion (RoM) as the maximum comfortable amplitude of each Tangible Action (translation, rotation and roll) for each pair of Pose and Place of motion. Concretely, it represents (1) the maximum distance that can be covered with the tangible object when translating it; (2) the total amplitudes of clockwise and counterclockwise rotations; (3) the total amplitude of forward and backward rolls. The greater the range of motion, the greater the range of values that can be manipulated.
Stability concerns the forearm (i.e. is it steady enough) and the tangible object manipulation: in the first case, a stable support is required; in the second case, it is important to know if unexpected tangible actions occur while performing one specific tangible actions [37, 40].

Width of Place of Motions corresponds to the space of the forearm used by the users when performing a tangible action in this PoM. It is important to determine whether the PoMs are large enough to hold the interaction without overlapping.

## 4.1 Task

Our study was divided into three parts. During the first part, we asked participants to perform translations on the four different places of motion (PoM): the full length of the forearm (length POM), as well as on the Elbow, Middle and Wrist PoMs, as illustrated in Fig. 1. A trial is defined as a back-and-forth translation on the forearm. The starting points of the translations were chosen by the participants at the beginning of each trial. It was not necessary to control the starting points as each trial was composed by 10 consecutive gestures.

In the second and third part of the study, we examined rotations and rolls respectively. A rotation (resp. roll) trial consisted in a rotation (resp. roll) towards the wrist (clockwise) followed by a rotation towards the elbow (counterclockwise) and a return to the initial state. The participants were asked to perform the maximum comfortable rotation (resp. roll) in both directions (clockwise, counterclockwise). Rolls perpendicular to the forearm would be too difficult to distinguish from translation in the width of the forearm and were not considered in this experiment.

Participants were asked to manipulate the device with the dominant hand and to perform the gestures on the forearm of the non-dominant hand. The physical manipulations were performed for each Pose (Vertical, Parallel, Forward, see Fig. 1). The poses and places of motion were explained and illustrated to the participants at the beginning of the experiment. Participants were free to grasp the object as they wished. Finally, since the purpose of the experiment was to study the use of the tangible on the forearm, no feedback was provided to the participants.

## 4.2 Apparatus

The diameter of the rounded tangible object used for the experiment was 8 cm. In order to detect rotations and rolls of the tangible object, we used an IMU of X-io Technologies (x-IMU: 48 g, 57 mm × 38 mm × 21 mm). The IMU is composed of a triple-axis gyroscope, accelerometer and magnetometer, offering an angular precision of 1°. The refresh rate of the sensors goes up to 512 Hz and we used Bluetooth to connect the IMU with the computer.

To locate the position of the tangible object and the body parts, we used an OptiTrack system composed of 12 cameras that track infrared reflective markers with a precision of 1 mm. The markers were carefully placed on the tangible object so that they do not influence the participant's grasp (Fig. 2). Additional infrared reflective markers were placed on the main joints of the arm/forearm (Fig. 2). The wrist, elbow of the non-dominant arm and the shoulders were also tracked.
<figure><img src="image_4.png" /><figcaption>Fig. 2. Placement of the infrared tracking markers</figcaption></figure>

## 4.3 Procedure

The first part of the study (on translations) follows a 3 × 4 within-subject design with the Pose (Forward, Parallel, Vertical) and the Place of Motion (Length, Elbow, Middle, Wrist) as factors. The second and third parts of the study (on rotation and roll respectively) followed a 3 × 3 within-subject design, with the same factors but without the Length Place of Motion (which is not in our design space, see Fig. 1).

The Pose factor is counterbalanced using a 3 × 3 Latin square. Each part of the study is composed of 3 blocks, each block corresponds to one Pose and consists of all the Places of motion in a random order. For each pair of Pose and Place of motion, participants had to do 3 repetitions of each trial (which is composed of 10 consecutive gestures). The participants could take a break between each repetition. Each part of the study lasted between 30 and 40 min. We collected 4320 trials in total for the translations study (360 trials per participant), 3240 trials for the rotation study (270 trials per participant) and 3240 trials for the rolls study (270 trials per participant).

## 4.4 Participants

We recruited 36 (12 participants for each of the 3 studies) (9 female), aged 26.0 on average (SD = 3.4). All the participants were right-handed. Most of the participants were students (PhD, MSc) from the computer science department of the local university; all other participants were familiar with electronic equipment.

## 4.5 Collected Data

We measured the circumference of the forearm near the elbow and the wrist for each participant as well as the inner and outer length of the forearm. Every 10 ms, we recorded the position, rotation and roll of the tangible object, as well as the position of the wrist, the elbow and the shoulders. To evaluate the physical fatigue, we asked participants to fill out a Borg scale [19] for each pair of (Pose, Place of motion).
# 5 Study: Results

We report the forearm stability; the device range of motion; the gestures overlapping, the unintentional device movements and the user's fatigue for each tangible action. Regarding the data analysis, we chose to rely on estimation techniques with 95% confidence intervals as recommended by the APA [44].

## 5.1 Forearm Stability: Elbow and Wrist Movements

To study the forearm stability, we analyzed the movements of the elbow and wrist with regards to their starting position, collected at the beginning of each trial. Results show that the elbow was relatively stable: the maximum movement observed, all directions included (along the X, Y and Z axis) and all physical manipulations included (Translations, Rotations, Rolls) did not exceed 1,7 cm on average. This result remain valid when considering each *Pose and Place of motion* independently. Taking into consideration these findings, we can say that the forearm is sufficiently stable to support tangible interactions.

## 5.2 Range of Motion

**Translations.** The ROM for translations along the forearm was computed by calculating the average distance covered by the tangible object for each group of 10 successive gestures. It was computed for each pair of *Pose and Place of motion*. As the size of the forearm differs from one participant to another, we standardized the collected data.

Participants exploited at least 93,6% of the forearm length when performing translation on the Length PoM (Vertical pose: 93,6% [87,1%; 99,8%]; Parallel pose: 101,8% [92,8%; 110,3%]; Forward pose: 98,6% [92,2%; 104,8%]). We observed that the translations performed in the Parallel pose extended to the hand, thus surpassing the wrist (explaining the values above 100% on Fig. 3A). The results do not reveal a difference of translation amplitudes between poses: overall, participants were able to exploit almost all the length of the forearm in the three conditions (see Fig. 3A).

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** (A) Translation amplitudes along the forearm in percentage of the forearm length (with 95%CIs); (B) Translation amplitudes around the forearm in percentage of the arm circumference (with 95%CIs)</figcaption></figure>
The amplitude of translations performed around the forearm (Elbow, Middle and Wrist places of motion) represents the length of the arc covered by the movement of the object. The results are presented in the form of percentages of the forearm circumference and standardized for all participants. The biggest translation amplitudes were observed for the Forward pose, as illustrated in Fig. 3B (Vertical pose: 37,6% [34,0%; 41,5%]; Parallel pose: 38,6% [34,9%; 42,4%]; Forward pose: 45,2% [40,9%; 49,9%]). There was no significant difference between the places of motion.

**Rotations.** Regarding the rotations, we collected the maximum clockwise (CW) and counterclockwise (CCW) rotations. Overall, CW rotations among the 3 poses were slightly bigger (58° vs 50°) than CCW. This difference between CW and CCW amplitude is in line with those obtained with a device of similar form [37] (CW:73°, CCW:59°). The participants in our study used the “squeeze” hand posture [37] to grasp the object. The global amplitudes of rotations (CW + CWW) were clearly larger in the *Vertical* pose (Vertical pose: 138° [111°; 176°]; Parallel pose: 90° [74°%; 109°%]; Forward pose: 95° [82°; 107°]). We noticed that in this pose participants could move the arm holding the device to a greater extent than in the *Parallel* and *Forward* pose, hence resulting in larger rotations (Fig. 4A).

<figure><img src="image_6.png" /><figcaption>**Fig. 4.** (A) Rotations amplitudes in degrees (with 95%CI); (B) Rolls amplitudes in degrees (with 95%CI)</figcaption></figure>

**Rolls.** We collected the maximum rolls towards the wrist and the elbow (Fig. 4B): (1) towards the wrist: The observed amplitudes ranged from 46° to 51° for rolls towards the wrist (Vertical pose: 48° [40°; 55°]; Parallel pose: 46° [40%; 51%]; Forward pose: 51° [43°; 61°]). (2) towards the elbow: The amplitude ranged from 53° to 61° for roll towards the elbow (Vertical pose: 61° [55°; 69°]; Parallel pose: 53° [47%; 63%]; Forward pose: 57° [48°; 72°]). Therefore, results tend to establish that rolls towards the elbow had bigger amplitudes for the 3 poses than rolls toward the wrist.

## 5.3 Gestures Overlapping

However, when refining this result through the analysis of the distribution of the device position, results show that the interaction on the *Elbow* and *Middle* overlaps over a third of the forearm (Fig. 5A); the overlap between interaction in the *Middle* and *Wrist* represents a fifth of the forearm. However, it clearly appears that the translations on the *Elbow* and *Wrist* do *never* overlap. Finally, despite the clear instruction that required
participants to perform translation from the elbow to the wrist, we can observe that a
fair number of translations were performed beyond the wrist position (i.e. on the hand).
During **rotations**, participants used 23% of the forearm to perform rotations. The
surface used was slightly smaller on the Elbow (21%) compared to the Middle and
Wrist (23% & 24% respectively). A reason for that could be that interaction near the
elbow are more restricted due to the upper arm. Similarly, to the translations results, we
observed a large distribution of the exploited surfaces (Fig. 5B). As with translations
and rotations, the exploited surfaces when performing **rolls** on the forearm (Fig. 5C)
were fairly small (Elbow POM: 19%, Middle POM: 21%, Wrist POM: 20%) but their
dispersion was large enough for them to overlap.

<figure><img src="image_7.png" /><figcaption>**Fig. 5.** Distribution of the device position on the forearm during (A) Translations, (B) Rotations, (C) Rolls</figcaption></figure>

**5.4 Unintentional Device Movements**

As underlined in [21], rolls are the most probable unintentional manipulations that can
occur during translations, which could lead to the activation of unwanted commands.
This is particularly true in our case, since users perform translations over the cir-
cumference of the forearm, unlike the flat surface used in [21]. Therefore, we analyzed
the stability of the device during interaction by looking at unintentional rolls during
translations, similar to [21].

**Rolls During Translations Along the Forearm (Length POM).** Results show that on average, involuntary rolls during translations of the device did not exceed 12,6° [8,7°; 16,2°] for the *Forward* pose, 8.2° [6,5°; 11,2°] for the *Parallel* pose and 2.7° [1,9°; 4,2°] for the *Vertical* pose. The *Vertical* pose clearly triggers less involuntary rolls than the other poses. It also seems that the *Forward* pose is the most prone to unintentional rolls. These findings are in line with the results of the previous studies conducted on devices of similar form factors (12°) [37].

**Rolls During Translations Around the Forearm.** We wanted to know whether participants roll the device when translating it around the forearm (i.e. on the Elbow, Middle and Wrist). The results show that, for all poses, the translations around the forearm are systematically performed with a large roll of the device (at least 58°). This
reveals that the tangible object was not maintained horizontally during translations around the forearm. The rolls were more important in the *Vertical* pose where the average roll was approximately 78°. This number decreases to 62° on average for the *Parallel* pose and 58° for the *Forward* pose. This observation remains valid for all *places of motion* taken separately (*Elbow, Middle and Wrist*).

## 5.5 Fatigue

Fatigue was measured using a 6–20 Borg scale [11]. The average Borg score obtained for the poses ranged from ‘very light’ for the Forward (9) and Parallel (10) pose, to ‘light’ for the Vertical pose (11). The pose does not appear to affect the fatigue scores and overall, participants did not consider tangible on-body interaction tiring. However, with an average of 13, rotations in the Vertical pose were found relatively tiring to perform (‘somewhat hard’). It should also be noted that while participants had the opportunity to take breaks during the experiment between each group of 10 gestures, only one participant asked for a break.

# 6 Discussion

The findings presented above consolidate our hypothesis that on-body tangible interaction is a promising approach for use in immersive environments. It is stable, fatigue free, and offers a very rich set of interactions. Multiples actions, poses and places can be combined to best fit the user’s set of tasks to perform, offering a wide input vocabulary to manipulate immersive environment.

The **forearm stability** did not diminish over time. The forearm position was steady for the entire study (all physical manipulations included), making this body part a reliable support for interaction. The interaction itself did not induce fatigue according to the users’ feedback, even if the sessions took more than 30 min.

The **surfaces covered** while performing physical manipulations with the tangible object on the Elbow, Middle and Wrist POM were small enough (less than a fourth of the forearm at most) to consider three distinct regions of the forearm for interaction. However, the dispersion of these surfaces showed that they overlap (all physical manipulations included), making it difficult to employ more than 2 distinct regions of the forearm in practice. We want to point out that our experimental setup did not offer any feedback regarding the available places of motion. We believe that with a visual feedback showing the position of each region on the forearm in the immersive environment, the three regions would be more easily distinguishable. This hypothesis should be studied in a complementary study.

From the previous results, we can draw some **design suggestions** on how to use on-body tangible interaction for visualization tasks:

* The translations performed on the length POM were the most robust in terms of involuntary rotations and rolls (in line with previous results on desktop [37]). In terms of exploitable interaction surfaces, most (at least 73,7%) of the forearm subregions available for interaction was exploited to perform on-body tangible
interactions. Due to their stability and large amplitudes, translations on the length POM can be used to control a large set of values, to have a substantial precision or a greater data coverage. For instance, they could be used to manipulate slider type controllers that require a certain degree of precision depending on the manipulated data. Their stability allows them to be used in compound gestures (in combination with a roll or a rotation) which can allow for the design of expert modes.

* Translations around the forearm (Elbow, Middle and Wrist POM) exploited at most half of the surface available for interaction. These gestures seem to be better adapted to controllers that do not require a large amplitude, i.e. "rate control" type input.

* The amplitudes observed for rolls (ranging from 46° to 51°) were slightly bigger than those reported for a similar device in [37] on a desktop context (40° on average), while rotations' amplitudes were smaller (66° on average). This shows that on-body tangible interaction offers similar interaction possibilities than its desktop counterpart.

* Rotations in the vertical pose were larger than the two other poses at the cost of an increased fatigue. Therefore, these gestures are prime candidates for short interactions, such as controlling menus (limited number of sub-items) or activate toggles (two modes, discrete two-state tasks).

Theoretically, our design space describes a vocabulary composed of 30 continuous interactions: 12 1D-Translation gestures (3 poses * 4 POM), 9 1D-Rotation gestures and 9 1D-Roll gestures (3 poses * 3 POM each). The three-part study conducted in this work showed that it is possible to perform all the translations and rotations identified in our design space. Thus, our approach interaction vocabulary offers a rich and large set of possibilities to carry data visualization tasks. However, it is necessary to choose the right mapping between the possible On-Body Tangible controls and the tasks, which raises several questions: What is the optimal mapping that allows the user to keep a good balance between exploiting a great number of the possible interactions and memorizing them? Is it better to allow the user to create his own mapping, or should a predefined mapping be proposed to him? Answering these questions is not possible without an evaluation of the two approaches, which is one of the perspectives of this work. In the following sections, we introduce both of these approaches through a usage scenario for controlling space-time cube operations, and by introducing a meta-UI developed to help users assign OBT interactions to tasks.

# 7 Illustrative Application

In this section, we present an illustrative use of OBT interaction in the context of space-time cube data visualization. It is firstly used to select the appropriate space-time cube operation (e.g. scale); Then it is used to execute the selected operation (e.g. adjust the scaling ratio). In the first case we illustrate how the operations can be assigned to OBT interactions in a predefined manner; In the second case we illustrate how OBT interactions can be dynamically assigned to different aspects of the operation execution.
## 7.1 Space-Time Cube Operations

Visualizing and interacting with spatio-temporal datasets is rather difficult, as illustrated by the continuous research dedicated to this topic. One of the reasons for this difficulty is the important number of operations that can be performed on these datasets as well as the complexity of these operations. Bach et al. [2] introduced a taxonomy of elementary space-time cube operations to organize them. Bach’s taxonomy encompasses interactive operations that are relevant to all visualizations representing a combination of time (1D) and space (2D). The taxonomy is composed of four main groups of operations: extraction; flattening; geometry transformation and content transformation. As per Bach’s descriptions [2]:

* Extraction: consists in selecting a subset of a space-time object (e.g., extracting a line or cut from a volume).

* Flattening: consists in aggregating a space-time object into a lower-dimensional space-time object (e.g., projecting a volume onto a surface).

* Geometry transformation: consists in transforming a space-time object spatially without change of content, i.e. without affecting the data presented.

* Content transformation: consists in changing the content of a space-time object without affecting its geometry.

In the following subsection, we demonstrate how our OBT interaction approach can be used to support the selection of one of these operations (Sect. 7.2) and we present a different use of OBT interaction to support their execution (Sect. 7.3).

## 7.2 Mapping of OBT Interactions to Elementary Space-Time Cube Operations

In this section, we illustrate how a predefined mapping can be built between a set of operations and the 30 available OBT interactions, to allow selecting one of the space-time cube operations.

**Available OBT Interactions.** We used the following subset of gestures from our interaction vocabulary:

* The three main poses: Forward, Vertical and Parallel, each corresponding to one line on Fig. 6

* Each pose has three places of motion: Elbow, Middle and Wrist, each corresponding to one column in Fig. 6

* On each Place of motion, we decided to use discrete instead of continuous tangible actions and retained the following tangible actions:

- 2 Translations: a translation can occur in two directions around the forearm (clockwise, counterclockwise) (red arrows in each cell of Fig. 6).

- 4 Rolls: a roll can be performed in two directions (towards the hand or towards the elbow). In each direction, 2 thresholds have been defined so that a roll in each direction can be assigned to 2 different commands (green dots in each cell of Fig. 6).

* As a validation mechanism, we used a physical button positioned on top of the device.
<figure><img src="image_8.png" /><figcaption>**Fig. 6.** Illustration of the use of OBT interaction for selecting space-time cube operations. (Color figure online)</figcaption></figure>
A total of 54 commands can be selected with such vocabulary (6 per place of motion × 3 Places of Motion × 3 Poses).

**Mapping Space Time Cube Operation to OBT Interactions.** Bach’s taxonomy [2] lists 4 groups of general operations. Each one contains several specific operations: *extraction* (16 operations); *flattening* (4 operations); *geometry transformation* (9 operations) and *content transformation* (13 operations). As the flattening operations are performed directly on the cube, we added them to the geometry transformation group (performed directly on the cube too). This resulted in a new group of operations that we called “Geometry transformation and flattening”. Regrouping these operations resulted in 3 balanced groups of operations. We mapped each group of operations to a pose (see Fig. 6-POSES): (1) Extraction (group with the most tasks) is mapped to the forward pose (the most accessible pose); (2) Geometry transformation and flattening are mapped to the vertical pose; (3) Content transformation is mapped to the parallel pose. Next, we mapped each sub-group to a place of motion on the forearm (Fig. 6-POM), and finally, the specific operations to one of the possible tangible actions (translate or roll). As an example, to select the task Rotation pitch (Fig. 6), the user puts his forearm in the Vertical pose, places the tangible object on the wrist POM, and performs a small roll towards the hand before validating by pressing the button on the device. In this case, the mapping is predefined and appropriate feedback will have to be displayed to the user to confirm the selected operation. Of course, one important issue in this process is to adopt a coherent, easy to learn mapping. Alternatively, the choice of the mapping may be left to the user. In the next section we illustrate the use of a MetaUI to help the user establish himself the desired mapping between operations and OBT interactions.

## 7.3 A Meta UI Supporting User's Assignments of OBT Interaction to Operation

The use case in the previous section focused on the use of OBT interaction for the selection of space time cube operations. After the command is selected, the next step consists in applying a tangible action of the device (translation, roll or rotation) to control the execution of the space-time cube operation. To facilitate this second step, we implemented a tool (Fig. 7) that allows the user to quickly define his own mapping between the available OBT interactions and the operations to control.

The metaUI includes a mapping panel (Fig. 7A). Here the user chooses a pose, place of motion and gesture, and assigns to it a function i.e. controlling one aspect of the visualization operation selected (e.g. value of the operation filtering; starting time of the operation Volume-Chopping; ...). The MetaUI also includes a simulation panel (Fig. 7B) that allows to simulate the defined mapping through a set of UI elements. The results of the simulated OBT interaction are applied directly to the visualization. This tool is demonstrated in the supplementary video.
<figure><img src="image_9.png" /><figcaption>**Fig. 7.** Customization interface: (A) Defining the mapping between OBT interactions and visualization operations; (B) Simulating the defined mapping (control: B - left, effect: B - right).</figcaption></figure>

# 8 Conclusion and Future Work

In this paper, we proposed, described and studied a new approach for interaction with immersive environments: on-body tangible interactions. This approach is based on the use of the forearm to support tangible interactions using a rounded object that offers multiple degrees of freedom. It takes advantage of the body's natural capacity to sense its own body parts (proprioception) without switching user's attention from the data and minimizing the fatigue. We proposed a design space describing the Pose and the Place of Motion of interaction. We conducted a study to confirm the stability of the forearm to support tangible interaction, to measure the range of motion of the physical manipulations and measure user fatigue. The results showed that on-body tangible interactions are a promising approach to interact with immersive environments since the interaction support (forearm) is stable, the tangible object combined with the different poses and places of motions offers a large number of degrees of freedom, and users found the approach comfortable. Finally, we illustrated the possible usage of this approach through a concrete illustrative application concerning the selection of spatio-temporal operations. This example was thought to demonstrate several ways of using OBT: with a predefined mapping or with a user defined mapping (i.e. with the Meta UI). Our hypothesis is that the user defined mapping will help to recall the commands and ultimately, improve data visualization exploration. However, further experiments need to be conducted to measure the impact and benefits of each approach.

The next step of this work consists in exploring the technological feasibility of our approach. As it is, the tangible object integrates an IMU to detect rolls and rotations autonomously, while the arm pose and the object translation are detected with a camera-based tracking system. This approach is robust and satisfied the objective of our work, i.e. evaluate the advantages of OBT interaction. To implement a viable technical solution, we plan to explore the use of a bracelet to detect the arm pose, and the use of the accelerometer of the IMU to detect the object translation.

Once we develop this integrated solution, we plan to investigate other relevant research questions, such as the performance (in terms of precision and time) or the memorization of OBT gestures. Indeed, while our current approach allows to map a very large number of commands, it increases the difficulty of learning them. To solve this limitation, one solution is to provide suitable feedforward and feedback mechanisms, which will help with both discoverability and recall of mappings. Moreover, the
memorization of OBT gestures could also benefit from semantic and spatial aids of the
body places considered in our design space, as demonstrated in previous work [19].
Finally, it will be interesting to explore the usage of this approach under different
frames of reference: while the egocentric approach seems to better fit this type of
interaction, the effects of using a world-centric reference should be evaluated in a
controlled study.

**Acknowledgments.** This work is partially funded by the AP2 project grant AP2 ANR-15-CE23-0001.

References

1. Bach, B., Sicat, R., Beyer, J., Cordeil, M., Pfister, H.: The Hologram in my hand: how effective is interactive exploration of 3D visualizations in immersive tangible augmented reality? IEEE Trans. Vis. Comput. Graph. (TVCG) 24, 457–467 (2018)

2. Bach, B., Dragicevic, P., Archambault, D., Hurter, C., Carpendale, S.: A descriptive framework for temporal data visualizations based on generalized space-time cubes. Comput. Graph. Forum 36(6), 36–61 (2017). https://doi.org/10.1111/cgf.12804

3. Balakrishnan, R., Baudel, T., Kurtenbach, G., Fitzmaurice, G.: The Rockin’Mouse: integral 3D manipulation on a plane. In: Proceedings of the ACM SIGCHI Conference on Human factors in Computing Systems, pp. 311–318, Atlanta, Georgia, USA, 22–27 March 1997. https://doi.org/10.1145/258549.258778

4. Beaudouin-Lafon, M.: Instrumental interaction: an interaction model for designing post-WIMP user interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2000), pp. 446–453. ACM, New York (2000). https://doi.org/10.1145/332040.332473

5. Benko, H., Wilson, A.D.: Multi-point interactions with immersive omnidirectional visualizations in a dome. In: ACM International Conference on Interactive Tabletops and Surfaces (ITS 2010), pp. 19–28. ACM, New York (2010). https://doi.org/10.1145/1936652.1936657

6. Benko, H., Izadi, S., Wilson, A.D., Cao, X., Rosenfeld, D., Hinckley, K.: Design and evaluation of interaction models for multi-touch mice. In: Proceedings of Graphics Interface 2010, 31 May–02 June 2010, Ottawa, Ontario, Canada (2010)

7. Bergé, L.-P., Serrano, M., Perelman, G., Dubois, E.: Exploring smartphone-based interaction with overview+detail interfaces on 3D public displays. In: Proceedings of the 16th International Conference on Human-Computer Interaction with Mobile Devices & Services (MobileHCI 2014), pp. 125–134. ACM, New York (2014). https://doi.org/10.1145/2628363.2628374

8. Bergé, L.-P., Dubois, E., Raynal, M.: Design and evaluation of an “Around the SmartPhone” technique for 3D manipulations on distant display. In: Proceedings of the 3rd ACM Symposium on Spatial User Interaction (SUI 2015), pp. 69–78. ACM, New York (2015). https://doi.org/10.1145/2788940.2788941

9. Bergstrom-Lehtovirta, J., Hornbæk, K., Boring, S.: It’s a wrap: mapping on-skin input to off-skin displays. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI 2018), 11 p. ACM, New York, Paper 564 (2018). https://doi.org/10.1145/3173574.3174138
10. Besançon, L., Issartel, P., Ammi, M., Isenberg, T.: Mouse, tactile, and tangible input for 3D manipulation. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI 2017), pp. 4727–4740. ACM, New York (2017). https://doi.org/10.1145/3025453.3025863

11. Borg, G.: Borg's Perceived Exertion and Pain Scales, p. 104. Human Kinetics, Champaign (1998). viii

12. Burgess, R., et al.: Selection of large-scale 3D point cloud data using gesture recognition. In: Camarinha-Matos, L.M., Baldissera, T.A., Di Orio, G., Marques, F. (eds.) DoCEIS 2015. IAICT, vol. 450, pp. 188–195. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16766-4_20

13. Clarke, S., Dass, N., Chau, D.H.(Polo).: Naturalmotion: exploring gesture controls for visualizing time-evolving graphs. In: Proceedings of IEEE VIS (Poster Session) (2016)

14. Coffey, D., et al.: Interactive slice WIM: navigating and interrogating volume datasets using a multi-surface, multi-touch VR interface. IEEE Trans. Vis. Comput. Graph. **18**(10), 1614–1626 (2012). https://doi.org/10.1109/TVCG.2011.283

15. Cordeil, M., Bach, B., Li, Y., Wilson, E., Dwyer, T.: A design space for spatio-data coordination: tangible interaction devices for immersive information visualisation. In: Proceedings of the 10th IEEE Pacific Visualization Symposium (PacificVis) (2017)

16. Cordeil, M., Dwyer, T., Hurter, C.: Immersive solutions for future air traffic control and management. In: Proceedings of the 2016 ACM Companion on Interactive Surfaces and Spaces (ISS Companion 2016), pp. 25–31. ACM, New York (2016). https://doi.org/10.1145/3009939.3009944

17. Dobbelstein, D., Hock, P., Rukzio, E.: Belt: an unobtrusive touch input device for head-worn displays. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI 2015), pp. 2135–2138. ACM, New York (2015). https://doi.org/10.1145/2702123.2702450

18. Giannopoulos, I., Komninos, A., Garofalakis, J.: Natural interaction with large map interfaces in VR. In: Proceedings of the 21st Pan-Hellenic Conference on Informatics (PCI 2017), Article no. 56, 6 p. ACM, New York (2017). https://doi.org/10.1145/3139367.3139424

19. Fruchard, B., Lecolinet, E., Chapuis, O.: Impact of semantic aids on command memorization for On-Body interaction and directional gestures. In: Proceedings of the International Working Conference on Advanced Visual Interfaces, AVI 2018 - International Conference on Advanced Visual Interfaces, Castiglione della Pescaia, Grosseto, Italy, May 2018. ACM, AVI 2018 (2018)

20. Hancock, M., Carpendale, S., Cockburn, A.: Shallow-depth 3D interaction: design and evaluation of one-, two- and three-touch techniques. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems—CHI 2007, pp. 1147–1156. ACM Press (2007)

21. Harrison, C., Tan, D., Morris, D.: Skinput: appropriating the body as an input surface. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2010), pp. 453–462. ACM, New York (2010). https://doi.org/10.1145/1753326.1753394

22. Hinckley, K., Wigdor, D., Input technologies and techniques. In: The HCI Handbook, 3rd edn. Taylor & Francis. Chap. 9

23. Hinckley, K., Sinclair, M., Hanson, E., Szeliski, R., Conway, M.: The VideoMouse: a camera-based multi-degree-of-freedom input device. In: Proceedings of the 12th annual ACM Symposium on User Interface Software and Technology, pp. 103–112, Asheville, North Carolina, USA, 07–10 November 1999. https://doi.org/10.1145/320719.322591]
24. Hurter, C., Tissoires, B., Conversy, S.: FromDaDy: spreading aircraft trajectories across views to support iterative queries. IEEE Trans. Vis. Comput. Graph. **15**(6), 1017–1024 (2009). https://doi.org/10.1109/tvcg.2009.145

25. Issartel, P., Guéniat, F., Ammi, M.: A portable interface for tangible exploration of volumetric data. In: Proceedings VRST, pp. 209–210. ACM, Edinburgh, Scotland (2014). https://doi.org/10.1145/2671015.2671130. ISBN 978-1-4503-3253-8

26. Jackson, B., et al.: A lightweight tangible 3D interface for interactive visualization of thin fiber structures. IEEE Trans. Vis. Comput. Graph. **19**(12), 2802–2809 (2013). http://doi.org/10.1109/TVCG.2013.121. http://ieeexplore.ieee.org/xplabs/abs_all.jsp?arnumber=6651934

27. Jansen, Y., Dragicevic, P., Fekete, J.-D.: Tangible remote controllers for wall-size displays. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2012), pp. 2865–2874. ACM, New York (2012). http://dx.doi.org/10.1145/2207676.2208691

28. Karrer, T., Wittenhagen, M., Lichtschlag, L., Heller, F., Borchers, J.: Pinstripe: eyes-free continuous input on interactive clothing. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2011), pp. 1313–1322. ACM, New York (2011). https://doi.org/10.1145/1978942.1979137

29. Kim, J.-S., Gračanin, D., Matković, K., Quek, F.: Finger walking in place (FWIP): a traveling technique in virtual environments. In: Butz, A., Fisher, B., Krüger, A., Olivier, P., Christie, M. (eds.) SG 2008. LNCS, vol. 5166, pp. 58–69. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-85412-8_6

30. Klein, T., Guéniat, F., Pastur, L., Vernier, F., Isenberg, T.: A design study of direct-touch interaction for exploratory 3D scientific visualization. Comput. Graph. Forum **31**, 1225–1234 (2012)

31. Lopez, D., Oehlberg, L., Doger, C., Isenberg, T.: Towards an understanding of mobile touch navigation in a stereoscopic viewing environment for 3D data exploration. IEEE Trans. Vis. Comput. Graph. **22**(5), 1616–1629 (2016)

32. Lundström, C., et al.: Multi-touch table system for medical visualization: application to orthopedic surgery planning. IEEE Trans. Vis. Comput. Graph. **17**(12), (2011). https://doi.org/10.1109/TVCG.2011.224

33. Milgram, P., Colquhoun, H.: A taxonomy of real and virtual world display integration. Mixed Reality Merg. Real Virtual Worlds **1**, 1–26 (1999)

34. Miranda, B.P., Carneiro, N.J.S., dos Santos, C.G.R., de Freitas, A.A., Magalhães, J., Meiguins, B.S., et al.: Categorizing issues in mid-air InfoVis interaction. In: 2016 20th International Conference Information Visualisation (IV), pp. 242–246. IEEE (2016)

35. Ortega, M., Nigay, L.: AirMouse: finger gesture for 2D and 3D interaction. In: Gross, T., et al. (eds.) INTERACT 2009. LNCS, vol. 5727, pp. 214–227. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-03658-3_28

36. Parsons, L.M.: Inability to reason about an object’s orientation using an axis and angle of rotation. J. Exp. Psychol. Hum. Percept. Perform. **21**(6), 1259–1277 (1995)

37. Perelman, G., Serrano, M., Raynal, M., Picard, C., Derras, M., Dubois, E.: The roly-poly mouse: designing a rolling input device unifying 2D and 3D interaction. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Republic of Korea, 18–23 April 2015. https://doi.org/10.1145/2702123.2702244

38. Rautaray, S.S.: Real time hand gesture recognition system for dynamic applications. IJU Int. J. UbiComp **3**(1), 21–31 (2012)

39. Rudi, D., Giannopoulos, I., Kiefer, P., Peier, C., Raubal, M.: Interacting with maps on optical head-mounted displays. In: Proceedings of the 2016 Symposium on Spatial User Interaction (SUI 2016), pp. 3–12. ACM, New York (2016). https://doi.org/10.1145/2983310.2985747
40. Saidi, H., Serrano, M., Irani, P., Dubois, E.: TDome: a touch-enabled 6DOF interactive device for multi-display environments. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI 2017), pp. 5892–5904. ACM, New York (2017). https://doi.org/10.1145/3025453.3025661

41. Steven, S., Ishii, H., Schroder, P.: Immersive design of DNA molecules with a tangible interface. In: Proceedings of the Visualization, pp. 227–234. IEEE Computer Society, Los Alamitos (2004). https://doi.org/10.1109/VISUAL.2004.47

42. Serrano, M., Ens, B.M., Irani, P.P.: Exploring the use of hand-to-face input for interacting with head-worn displays. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2014), pp. 3181–3190. ACM, New York (2014). https://doi.org/10.1145/2556288.2556984

43. Song, P., Goh, W.B., Fu, C.-W., Meng, Q., Heng, P.-A.: WYSIWYF: exploring and annotating volume data with a tangible handheld device. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2011), pp. 1333–1342. ACM, New York (2011). https://doi.org/10.1145/1978942.1979140

44. VandenBos, G.R.(ed): Publication Manual of the American Psychological Association, 6th edn. American Psychological Association, Washington, D.C. (2009)

45. Vo, D.-B., Lecolinet, E., Guiard, Y.: Belly gestures: body centric gestures on the abdomen. In: Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational (NordiCHI 2014), pp. 687–696. ACM, New York (2014). https://doi.org/10.1145/2639189.2639210

46. Wagner, J., Nancel, M., Gustafson, S.G., Huot, S., Mackay, W.E.: Body-centric design space for multi-surface interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13), pp. 1299–1308. ACM, New York (2013). https://doi.org/10.1145/2470654.2466170

47. Wang, C.-Y., Chu, W.-C., Chiu, P.-T., Hsiu, M.-C., Chiang, Y.-H., Chen, M.Y.: PalmType: using palms as keyboards for smart glasses. In: Proceedings of the SIGCHI Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI 2015), pp. 153–160 (2015). http://dx.doi.org/10.1145/2785830.2785886

48. Ware, C., Arsenault, R.: Frames of reference in virtual object rotation. In: Proceedings of APGV 2004, pp. 135–141 (2004)

49. Wong, P.C., Zhu, K., Fu, H.: FingerT9: leveraging thumb-to-finger interaction for same-side-hand text entry on smartwatches. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, p. 178. ACM, April 2018

50. Zhai, S., Milgram, P.: Quantifying coordination in multiple DOF movement and its application to evaluating 6 DOF input devices. In: Karat, C.-M., Lund, A., Coutaz, J., Karat, J. (eds.) Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 1998), pp. 320–327. ACM Press/Addison-Wesley Publishing Co., New York, NY (1998). https://doi.org/10.1145/274644.274689

51. Zhu, K., Ma, X., Chen, H., Liang, M.: Tripartite effects: exploring users’ mental model of mobile gestures under the influence of operation, handheld posture, and interaction space. Int. J. Hum.-Comput. Interact. 33(6), 443–459 (2017)