<figure><img src="image_1.png" /></figure>

HAL
open science

# Visuo-haptic Illusions for Motor Skill Acquisition in Virtual Reality

Benoît Geslain, Gilles Bailly, Sinan Haliyo, Corentin Duboc

► To cite this version:

Benoît Geslain, Gilles Bailly, Sinan Haliyo, Corentin Duboc. Visuo-haptic Illusions for Motor Skill Acquisition in Virtual Reality. Spatial User Interaction, Nov 2021, Virtual Venue, United States. 10.1145/3485279.3485291. hal-03354079v2

HAL Id: hal-03354079

https://hal.science/hal-03354079v2

Submitted on 8 Nov 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
# Visuo-haptic Illusions for Motor Skill Acquisition in Virtual Reality

Benoît Geslain

Sorbonne Université, CNRS, ISIR, SEGULA Technologies

France

Sinan Haliyo

Sorbonne Université, CNRS, ISIR

France

Gilles Bailly

Sorbonne Université, CNRS, ISIR

France

Corentin Duboc

SEGULA Technologies

France

## ABSTRACT

In this article we investigate the potential of using visuo-haptic illusions in Virtual Reality environment to learn motor skills in a real environment. We report on an empirical study where 20 participants perform a multi-object pick-and-place task. The results show that although users do not perform the same motion trajectories in the virtual and real environments, skills acquired in VR augmented with visuo-haptic illusions can be successfully reused in a real environment: There is a high amount of skill transfer (78.5%), similar to the one obtained in an optimal real training environment (82.4%). Finally, participants did not notice the illusion and were enthusiastic about the VR environment. Our findings invite designers and researchers to consider visuo-haptic illusions to help operators to learn motor skills in a cost-effective environment.

## CCS CONCEPTS

* Software and its engineering → Virtual worlds training simulations; Hardware → Tactile and hand-based interfaces;

* Human-centered computing → Empirical studies in HCI.

## KEYWORDS

Virtual Reality, Haptic Feedback, Visuo-haptic illusions, Transfer of learning, User Study.

## 1 INTRODUCTION

Virtual Reality (VR) has been proven useful to train users on procedures that are difficult, expensive or dangerous to practice in situ [28], e.g. medical procedures [4, 8], assembly tasks [33]. To increase the realism of the scene, the virtual environment can be augmented with full-sized physical models, e.g. a airplane cockpits or physical tools [5, 20] to provide haptic feedback. It can also be augmented with haptic devices with an extended workspace like Haption Scale1 [1]. However, full-sized physical models or haptic devices often remain complex and/or expensive.

We argue that visuo-haptic illusions might be a cost-effective approach to train users in a VR environment. Visuo-haptic illusions consists of introducing subtle discrepancies between the vision and the proprioception to trick the user's perception. For instance, users can believe interacting with a large collection of physical

objects while, in fact, they only interact with a unique physical prop [3]. This emerging approach has been shown to enrich user experience: It provides realistic haptics [3], expands the interaction space [37] or augment interactivity [30, 52]. However, visuo-haptic illusions has not been investigated in the context of skill transfer. One reason might be that the discrepancy between vision and proprioception alters user's movements and gesture trajectories [3]. It may generate interference when used in a training scheme, thus degrade the transfer of learning in a motor control task.

In this paper, we study the effects of visuo-haptic illusions on proactive transfer of motor skills. Proactive transfer occurs when learning a skill in a specific context (VR) influences the performance of such skill in a different context (real world). We report on a laboratory experiment where participants perform a multi-object pick-and-place task common in training assistance systems [25]. We study whether skills acquired in a VR environment with visuo-haptic illusions can be transferred in a real-world environment.

Our results not only suggest that visuo-haptic illusions can be used in VR training, but also that the amount of transfer is comparable to training in real conditions. These results encourage the use of visuo-haptic illusions in a virtual learning environment for any task requiring the users to interact with multiple similar parts (pieces of an assembly, buttons) or tools (screwdrivers). They can help to design VR training environments with less complex full sized physical models or haptic devices, i.e. VR environments providing haptic feedback to the user with lower cost and less maintenance than traditional approaches.

## 2 BACKGROUND AND RELATED WORK

### 2.1 Proactive transfer and Virtual Reality

Proactive transfer of learning describes how a skill influences the learning of a new skill [44]. However, in the context of VR, transfer of learning is generally defined as how a skill learned in a virtual environment influences the learning of the same skill in the real world. The main measure of transfer learning is the percentage of transfer [44]. It refers to the ratio $T_\%$ between the performance before and after training the previous skill:

$$T_\% = \frac{A - B}{C - B} \qquad (1)$$

where A is the performance during the beginning of the test phase after training. B is the performance during the beginning of the test phase without training and C is the ultimate performance, i.e. the performance at the end of the test phase. A positive value indicates training in the virtual environment increases the performance on

<footnote>SUI '21, November 9–10, 2021, Virtual Event, USA</footnote>

<footnote>© 2021 Association for Computing Machinery.</footnote>

<footnote>This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in Symposium on Spatial User Interaction (SUI '21), November 9–10, 2021, Virtual Event, USA, https://doi.org/10.1145/3485279.3485291.</footnote>
the task in the real world [15, 25] whereas a negative value indicates
interference of the previous training [47].

*Virtual Reality* has proven useful to train users on medical or industrial procedures [2, 21, 26] and sports [13, 31]. The use of several modalities such as audio or haptic feedback often promotes a positive transfer of learning [46] because it can improve the reproduction of real-life perceptions [32, 42] or assist and guide users by providing additional information [13, 34, 36]. However, these approaches do not always favor learning: the modalities are not always appropriate for a given skill or because users tend to rely too much on the guidance received during the training, which ultimately hinders skill acquisition [48, 50].

## 2.2 Visuo-haptic Illusions

Visuo-haptic illusions manipulate user perception through discrepancies between the visual and proprioceptive senses [23, 29]. These improve realism and multimodal feedback in a VR environment. These are possible because humans tend to rely more on visual feedback when multiple senses enter in conflict [14, 39]. We distinguish two main types of visuo-haptic illusions:

**World manipulation** consists of altering users' point of view, by subtly modifying mapping ratios between the real and virtual head movements. It can be used for instance to repurpose a real prop for several virtual objects [3], to increase the perceived interactive space [37, 45] or hide unrealistic virtual elements [11].

**Hand Redirection** adds a subtle visual shift between users' virtual and real hands. Users believe they interact with multiple virtual objects while manipulating the same real object [3, 10]. It's also possible to map an object's shape onto another [22, 53]. Similarly, designers can change the apparent physical properties of objects, for example their stiffness, by visually modifying their deformation [27] or their weight through their vertical motion [38, 43]. Hand redirection is a more adapted technique in manual handling cases.

Visuo-haptic illusions have been shown to improve presence and interactivity in VR [3, 17, 30], and appears as a viable way to reduce cost and complexity to set-up a virtual training environment. However, when motor skills are involved, they may induce a negative effect as suggested by studies in *visuo-motor adaptation*.

Visuo-haptic illusions rely on a shift small enough so that it goes undetected. On the other end of the spectrum, **visuo-motor adaptations** explore the effect of a *large* distortion which is immediately perceived by the users. Researchers then study how users adapt to this change in perception, for instance, how users adapt to a constant visual shift of their hand on a motor control task [18]. These distortions degrade motor control performance because users have to adapt to this new perception-action scheme. A loss of performance is observed immediately when the distortion is introduced, and participants generally regain similar performance after practice. Furthermore, a loss of performance is similarly observed when the distortion is removed. This is called an after-effect [9, 14, 35]. These studies suggest that using visuo-haptic illusions in VR training might hinder transfer of learning in the real world. While some works discussed the potential benefits of visuo-haptic

illusions for skill transfer [7, 12, 17, 24, 49], the amount of transfer in term of performance has never been properly quantified.

## 3 RESEARCH QUESTIONS

Based on previous work on skill transfer, and visuo-haptic illusions, our research questions are: Do visuo-haptic illusions like Hand Redirection in VR influence the learning of motor skills? and Do they promote/hinder performance in the real world? Can we quantify the transfer of learning from the virtual to the real world? To answer these questions, we conduct a user study

## 4 USER STUDY

This study investigates the transfer of learning between a virtual environment with a visuo-haptic illusion: hand redirection, and the real environment. Envisioned real-world tasks are product assembly or engine repair, currently targeted by training assistance systems [19, 33]. These tasks encapsulate several actions such as placing objects precisely, avoiding obstacles and following trajectories with time constraints. These tasks are classified as procedural tasks involving both motor and cognitive skills [40]. To operationalize these complex tasks in a laboratory setting, it is common to use a **multi-object pick-and-place** task [16, 41] where users pick a labeled cube and place it on a designated target. Our experiment thus relies on this modus operandi. Moreover, this task is particularly appropriate for visuo-haptic illusions as it is possible to implement n-to-1 virtual to physical mappings [3], i.e. one physical prop for several virtual objects.

To study proactive transfer, we use a well-established methodology where the experiment is segmented in a training phase and a testing phase with a 24 hours rest time between [44]. We compare the performance of three groups (Figure 3): the first group (*baseline*) only performs the test in the real environment with no training. The second group also performs the test in the real environment, but only after practicing in the real environment. The third group also performs the test in the real environment, but only after practicing in the virtual environment exploiting hand redirection. The ratio (equation 1) between with and without training informs on the nature of the transfer (positive or negative) and its magnitude.

This experiment has been approved by the IRB 2020-020.

### 4.1 Participants

20 participants (11 male, 9 female, aged 28.4, 2 left handed) took part in the present study. 15 played video games regularly and all had previous experience with VR. Participants were compensated with candies after each session.

### 4.2 Apparatus

Participants are seated on a chair in front of a table. The experiment consists in picking in sequence 6 cubes laying on predefined positions on a complex rectangular shape, and placing them on designated square targets. All cubes and their targets are marked with the same number, which also gives the order of execution (Fig. 1-a). The cubes are 8 cm in size and a cube is considered correctly placed when its center is within 3 cm of the target center, with an alignment error of less than 10°. This precision is similar to
<figure><img src="image_2.png" /><figcaption>**Figure 1:** The task set-up for the experiment. (a) The setting with 6 cubes used in *Test Only* condition, (b) the real set-up with a single cube for the *Virtual Training + Test* condition and (c) the virtual replica of the setting used in the *Test Only* condition. (d) Participant's right hand wearing reflective markers.</figcaption></figure>

[3]. The location of the real cubes were chosen in order to maximize the movement distances given the available space during the training phase. The virtual cubes are placed so that the amount of maximum redirection (warping distance) is large and spans both horizontal (X and Z) directions (cf table 1). A screen displaying the instructions stands in front of the user, behind the table. A motion capture system (*Optitrack*) tracks the hand trajectory and the 6 cubes. Participants wear 3 reflective markers on their dominant hand. An exact replica of this scene is built in VR using *Unity* (Fig. 1-c). Participants training in VR experience the scene through an Oculus Rift S Head-Mounted Display (HMD). Those are not made aware that they will be subject to a visuo-haptic illusion. They put on the HMD only after taking the seat, hence observe the real set-up beforehand. At this point, the experimenter, unseen to the participant, removes discreetly 5 out of 6 cubes from the set-up (Fig. 1-b). The visuo-illusion is detailed below.

## 4.3 Experimental Design

4.3.1 *Task and stimulus.* The task is a multi-object pick-and-place task [16, 41], involving obstacle avoidance. The participants are informed that the experiment will evaluate their performance and they are asked to be as accurate and fast as possible. They are instructed to pick the cube whose number is shown on the display and place it on the designated target, with the correct orientation. They press a green button located on the table to initiate the trial. The screen displays instructions and the participants pick the corresponding cube with their dominant hand and place it on the target. They then re-press the button to mark completion. If the cube is not placed accurately enough at the correct position or orientation (Fig. 2), the feedback on the screen requests them to rectify it. At the end of the sequence of 6 cubes, the execution time is displayed.

4.3.2 *Groups and Conditions.* We have three conditions:

* **Test only.** This condition measures the performance of participants performing the test phase in a real environment, without any training, on the full set-up (Fig. 1-a).

* **Real Training + Test** Participants first practice in the real environment in the same setup as the test phase (Fig. 1-a). They then perform the test in the real environment 24 hours later, identically to the previous condition.

<figure><img src="image_3.png" /><figcaption>**Figure 2:** Target placement of a cube. In (a), the cube is in the right position but wrong orientation. In (b), the cube is in the wrong position but in the right orientation. In (c), the cube is both in the right position and orientation. The sequence moves ahead only for (c).</figcaption></figure>

* **Virtual Training + Test.** Participants first practice in a virtual environment exploiting a visuo-haptic illusion. They then perform the test in the real environment 24 hours later. The virtual scene used in *Virtual Training + Test* is a replica of the setting used in test as shown in Fig. 1-c. Virtual obstacles, cubes and screen have *exactly* the same geometry, and are reasonably well co-located with the real set-up. The real setting is however modified without their knowledge, after they put on the HMD (Fig. 1-b). All cubes except the number 1 are removed, as well as most real obstacles. This reflects the possibility that the training set-up may be significantly simpler than the real task. A mismatch between the virtual scene and the real setup hence occurs. Indeed, participants are made to believe they manipulate 6 real cubes while there is only one (1:6 mapping). This illusion is implemented through a hand redirection technique: participants virtual hand's position is shifted from their real hand's. In consequence, participants do not perform the same trajectory in the training phase and the test phase.

The training consists in 15 blocks of 6 trials. The test consists in 5 blocks of 6 trials, 24 hours after the training when applicable (see figure 3). Twenty participants are split into two groups of ten assigned to the **Real Training + Test** condition (5 male, 5 female, aged 29) and **Virtual Training + Test** condition (6 male, 4 female, aged 27.7). The beginning of the training phase of the **Real Training + Test** condition is identical to the **Test only** condition. Hence, we did not assign participants to this group and used the data from the first five blocks of the **Real Training + Test** condition for the **Test only** condition. This allowed us to have less participants overall while using the same experiment design.

We expect participant in the **Real Training + Test** condition to perform best. The condition **Real Training + Test** provides an upper bound in terms of performance as users perform the training and the test in the same setting. So, our objective is to study how far is the condition **Virtual Training + Test** from this upper bound.

4.3.3 *Hand Redirection Implementation.* Our implementation of Hand Redirection relies on "Interpolated Reach" detailed in [17]. The shift between the real hand and the virtual hand is applied inside a sphere of radius *R* centered on the real target. It is a continuous function, maximum at the target location and null at the surface of the sphere and beyond. It is tuned so that users place down the real cube not on the real target but where the participant will pick up the next cube meanwhile the virtual hand places the cube on the virtual target. This way, in the next step, participants are
<figure><img src="image_4.png" /><figcaption>Figure 3: The procedure of the user study in two days: one day of test for the Test only and one day of training and one day of test for the Real Training + Test conditions and Virtual Training + Test.</figcaption></figure>

tricked into picking up the same cube they've just placed down. Hence, the illusion initiates with a null shift once the cube is picked up, increases on the way to the target, and decreases when the hand goes back to the button. The same real cube is thus used in subsequent manipulations. The position of the shifted objects (i.e. the virtual hand and the virtual cube) depends on the distance *d* to the target and is computed as follows:

$$ \vec{P}_v = \begin{cases} \vec{P}_r + \frac{(R-d)}{R} \times (\vec{T}_v - \vec{T}_r) & \text{if } R - d \ge 0, \\ \vec{P}_r & \text{otherwise.} \end{cases} \quad (2) $$

$$ \text{with } d = \|\vec{P}_r - \vec{T}_r\| \\ \text{and } R = \min(\|\vec{T}_0 - \vec{T}_r\|, \|\vec{P}_r - \vec{T}_r\|) + 0.05m. $$

$\vec{P}_v$ and $\vec{P}_r$ are the virtual and real positions of the shifted objects so that *d* represents the distance between the real and virtual hand, $\vec{T}_v$ and $\vec{T}_r$ are the virtual and real positions of the target, $T_0$ is the position of the button and *R* (*mean* = 57.5cm, *std* = 13.7cm) defines the illusion sphere centered on the target square where the cube should be placed. The maximum shift (*mean* = 10.7cm, *std* = 5.0cm), i.e. the distance between the virtual cube and the real cube at the end of the manipulation varies between trials. These values, summarized on Table 1, are generally higher than the literature [51], as participants were **not informed** about the illusion. A pilot study had confirmed that the illusion remains undetected when participants are not notified. Our shifts are thus large enough to affect users' trajectory but small enough to not be detected and maintaining a high level of immersion [3].

### 4.3.4 Manipulations and blocks.
Each block contains 6 manipulations/trials of different amplitudes and directions to vary the difficulty and to introduce variability. The sequence of manipulations (i.e. the order of the trials) remains the same from one block to another so that each block is identical to each other. This reflects a sequence of operations in the real world to assemble or repair a product. Because all the blocks contain the same sequence of manipulations, we can study learning phenomena by comparing performance between blocks.

### 4.3.5 Procedure.
Participants first received written instructions explaining the experiment procedure and the task. They then performed a pre-test consisting of picking a cube and placing it on a target in the real environment to ensure that the different groups of participants achieve a similar level of initial performance and understood the task properly. We measured the TIME as described in section 4.3.7 to realize the task and found no significant difference with ANVOA: $F_{1,19} = 0.25, p = 0.59$. This manipulation was not present in the main experiment. Participants then performed the main experiment. They have a 20 seconds break between each block. When realizing the task in the real or virtual environment, the breaks give time to the experimenter to reset cubes' position. After they complete the entire training phase, i.e. at the end of the first day, participants filled a questionnaire and performed a semi-structured interview. In the questionnaire, participants are asked to rate the perceived difficulty of each of the six manipulations (Likert scale from 1 to 7). They also answer two additional questions related to the illusion:

* How many real cubes did you think you handled?

* My hand moved as I intended. (1-7: "Fully agree/disagree")

The semi-structured interview comprised of the questions:

* How difficult did you find the Test compared to the Training?

* Would you like to train in a similar VR simulator?

The other questions such as "Did a particular manipulation seem more difficult during the test?" depended on the answers to the two first questions. Figure 3 summarizes the procedure.

### 4.3.6 Design.
This study follows a between-subject design. All groups perform a pre-test and a test. Both groups assigned to the conditions Virtual Training + Test and Real Training + Test performed 15 blocks of 6 trials respectively in VR and in the real environment during the training phase. During the test phase, the second day, all three groups performed 5 blocks of 6 trials in the real environment. The order of the trials within blocks is fixed.

In summary, the design is 2 groups × 10 participants × 15 blocks × 6 trials for training + 2 groups × 10 participants × 5 blocks × 6 trials + 1 group × 10 participants × 5 blocks × 6 trials for test = 2700 manipulations.

<table><caption>Table 1: The direction and distance for hand trajectories and warping for all trials. R is the same value as in eq 2. X represents the sideways horizontal direction (positive to the left), Y represents the vertical direction (positive upwards) and Z is the depth horizontal direction (positive forward).</caption><thead><tr><th rowspan="2">Trial</th><th colspan="2">Trajectory</th><th colspan="2">Warping</th><th rowspan="2">R</th></tr><tr><th>Main direction</th><th>Distance</th><th>Main direction</th><th>Distance</th></tr></thead><tbody><tr><td>1</td><td>X</td><td>66.8 cm</td><td>X</td><td>9.8 cm</td><td>52.2 cm</td></tr><tr><td>2</td><td>X</td><td>74.6 cm</td><td>Z</td><td>15.2 cm</td><td>42.3 cm</td></tr><tr><td>3</td><td>X&Y</td><td>44.6 cm</td><td>Z</td><td>9.1 cm</td><td>66.6 cm</td></tr><tr><td>4</td><td>X</td><td>41 cm</td><td>X&Z</td><td>13.5 cm</td><td>71 cm</td></tr><tr><td>5</td><td>X&Z</td><td>45.2 cm</td><td>X&Z</td><td>14.6 cm</td><td>73.6 cm</td></tr><tr><td>6</td><td>Y&Z</td><td>58.3 cm</td><td>Z</td><td>4 cm</td><td>39.2 cm</td></tr></tbody></table>
### 4.3.7 Dependent variables.
For each trial, we log TIME, the time to perform a trial (time measured between 2 button presses). We can then derive our main dependent variable, the amount of proactive transfer ($T_{z}$) based on the equation 1 [44].

We also log the position of the cubes at 60Hz to estimate the evolution of the trajectory VARIABILITY i.e. how much different are the trajectories between each other. To achieve this, we used the Dynamic Time Warping (DTW) distance [6]. DTW produces a discrete matching between existing elements of two temporal series before estimating the distance. By applying DTW on each pair of trials (i.e. pair-wise) and averaging the result for each conditions, we then compute the variability. The computed DTW distance is independent of time and the more similar the series, the lower the distance will be. DTW computations are made using the python `dtaidistance` library.

# 5 RESULTS

## 5.1 Proactive Transfer and Test Performance

Figure 4-left shows the performance for TIME for all three conditions during the test phase. We used the equation 1 to calculate the percentage of proactive transfer, which is 78.5% (std=18.5) for **Virtual Training + Test** and 82.4% (std=14.2) for **Real Training + Test**. ANOVA does not reveal an effect of Training (Virtual vs. Real) on the amount of transfer ($F_{1,19} = 0.25, p = 0.62$). As expected, the amount of transfer is higher for the **Real Training + Test** but it is nonetheless very high for the **Virtual Training + Test** condition.

Figure 4-right shows the performance for VARIABILITY for all three conditions during the test phase. The variability during the test phase for **Virtual Training + Test** is 1.38 and for **Real Training + Test** is 1.49. ANOVA reveals a significant difference between **Virtual Training + Test** and **Test only** ($F_{1,19} = 4.92, p = 0.027$) and no significant difference between the **Real Training + Test** and the other conditions (Virtual Training + Test: $F_{1,19} = 0.11, p = 0.74$, Test Only: $F_{1,19} = 2.33, p = 0.13$). These results suggest the hand redirection illusion had little to no negative impact on the performance during the test.

## 5.2 Training Performance

While our main objective is the initial performance during the test (proactive transfer), we also analyzed data collected during the training. Figure 5 & 6 shows the intramodal learning improvement, time and variability, during the training phase. We use the Power Law of Practice (PLP) model [44] to analyze it:

$$T = ae^{-b(P-1)} + c \quad (3)$$

where T is the performance (e.g. time in seconds), P is the trial number starting from 1. a, b and c are three parameters representing respectively the amount to be learned, the rate of learning, the asymptotic performance (i.e. ultimate performance).

We first fitted the learning curve on the evolution of TIME and obtained $a = 6.64, b = 0.51, c = 4.17$ for **Virtual Training + Test** and $a = 4.77, b = 0.49, c = 3.84$ for **Real Training + Test**. The Mean Square Error (virtual: MSE = 0.22, real: MSE = 0.01) and R-Squared (virtual: R² = 0.94, real: R² = 0.92) confirm that the PLP reflects well users' behavior during the training phase.

<figure><img src="image_5.png" /><figcaption>Figure 4: Performance during the test phase. Left: TIME performance; Right: VARIABILITY. Areas / bars shows the 95% confidence intervals.</figcaption></figure>

Similarly, we fitted the learning curve on VARIABILITY and obtained $a = 0.98, b = 1.12, c = 0.26, MSE = 0.01, R² = 0.92$ for **Virtual Training + Test** and $a = 0.90, b = 0.94, c = 1.14$ ($MSE = 0.005, R² = 0.95$) for **Real Training + Test**. The reduction of variability of the trajectory followed by the users according to the PLP is important as it confirms motor learning, i.e. a gradual reduction of variability necessary to optimize the accuracy and efficiency [44].

Our results thus show that the proactive transfer for the **Virtual Training + Test** condition is both positive and large (78.5%) and similar to the **Real Training + Test**. Our setup thus favors learning and immersion while using a high amount of redirection (12°).

## 5.3 Questionnaire and interviews

The median result to the affirmation “My hand moved as I intended” for the **Virtual Training + Test** condition is 2 (“Agree”) for both conditions. This result was confirmed in the interview as well, suggesting participants (8/10) did not notice the illusion and did not notice that there was only a single cube during training. When it was revealed to participants that they trained with only one cube, they reacted very enthusiastically: “It’s amazing, I would never have noticed there was only one [real] cube!” (Participant 2) or “This works very well, you fooled me properly” (Participant 14). Two participants noticed there was only one cube because they went to grab cubes during the 30 seconds break despite the experimenter telling them otherwise or because they had to correct the position of a cube and their hand passed through one of the virtual cubes. The findings do not change when excluding these two participants.

During the semi-structured interview when asked about how participants of the **Virtual Training + Test** condition felt when placing a cube, participants sometimes mentioned that something was a bit off. They all agreed to say this sensation didn't bother them and they would gladly train again in a similar VR simulator.

Finally, A Kruskal-Wallis Test on difficulty rating indicates that the manipulations 2 and 4 are significantly more difficult to perform than the other manipulations (time comprised between 3.2 seconds
<figure><img src="image_6.png" /><figcaption>Figure 5: Intramodal learning improvement during the virtual training phase on TIME performance. The areas show the 95% confidence intervals.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 6: Intramodal learning improvement during the virtual training phase on VARIABILITY REDUCTION. The areas show 95% confidence intervals.</figcaption></figure>

and 5.5 seconds, except for cube 2 and 6: statistic= 3.03, *p* = 0.08). The feedback from participants during the interview are in line with the results of the questionnaire.

# 6 QUALITATIVE ANALYSIS OF THE TRAJECTORY

Figure 7 illustrates the trajectories in the horizontal plane of all participants for the first/last block of the training/test phase for the two training conditions for the fifth manipulation. The variability reduces with practice during the training, but not during the test, which might be due to participants moving the cube faster at the

costs of less efficient trajectories. Moreover, several trajectories in *Virtual Training* have a kink toward the end of the movement (red cross). This result is similar to what has been observed in previous retargeting studies [3] and indicates that participants had to adjust their trajectory towards the end to compensate the illusion.

This kink is less noticeable at the end of the training phase (block 15), indicating there might be an adaptation to the illusion. On the contrary, during the first block of the test (after virtual training), we do not observe an after-effect in the sense of visuo-motor adaptation (i.e. a kink in the inverse direction). Additionally, it seems that the adaptation had little to no effect on the transfer of learning.

# 7 DISCUSSION

*Findings.* The main objective of this study was to investigate the transfer of learning when training with the hand redirection illusion. Our results show a large positive transfer of learning on TIME ($T_{\%Virtual}$ = 78.5%, *std* = 18.5%). Moreover, the transfer is close to the upper bound, i.e. the optimal setup (*Real Training* + *Test condition*): $T_{\%Real}$ = 82.4% (*std* = 14.2%). VR training with hand redirection also reduces movement variability by 45% similarly to what was observed after training in the real environment (35%). We were not able to measure an effect between the *Virtual Training* + *Test* and *Real Training* + *Test* conditions so it is likely to be small even with more participants as the 20 participants had similar performance between conditions during the pre-test and there is a large overlap of the confidence intervals. These results thus motivate the use of visuo-haptic illusions during virtual training. Finally, participants expressed enthusiasm during the virtual training. Indeed, only 2 out of 10 participants noticed the illusion despite using a large amount of warping above the detection thresholds recommended in the literature.

*Future work.* We aim to further explain these positive results. One possible reason is that the participants did not experience a single illusion but multiple ones: the virtual hand is shifted in different directions and amplitude during the training phase. This can explain why we do not observe an after-effect or a retro-active interference during the test phase. One study would consist of comparing the effect of the number of illusions users experience on performance.

We also plan to investigate how these results can be generalized to different visuo-haptic illusions such as the World Warping or the Hybrid technique of Haptic Retargeting [3], Redirected Walking [37] or Pseudo-haptic [29]. These illusions could have different effects on performance and learning than hand redirection. Additionally, the combination of illusions such as the Hybrid technique or Redirected Walking and hand Redirection could yield an interaction effect on the transfer of learning. Finally, we plan to investigate possible interaction effects between the nature of illusions and the absence/presence of haptic feedback.

*Implications for design.* Our results demonstrate the potential of using visuo-haptic illusions for motor-skill acquisition. Indeed, designers can use an excessive warping, higher than what's recommended in previous studies [51] on visuo-haptic illusions, and still favor skill transfer on task requiring the users to interact with multiple similar parts (pieces of an assembly, buttons) or tools
<figure><img src="image_8.png" /><figcaption>Figure 7: Cubes trajectories for the first and last blocks of training and test for both training conditions. The black dot indicates the start position, the red/blue cross indicates the target position in respectively the Real / VR Training. The call-out, bottom left highlights the end of the movement and the kink observed in VR Training.</figcaption></figure>

(screwdrivers). These results are rather unexpected. Indeed, users do not perform the same trajectories in the virtual and real environment and visuo-motor adaptation literature suggests a loss of performance when such a distortion is introduced. A main difference is that in our setup most users do not notice the illusion despite the intensive warping, and a high level of immersion is maintained. It is especially interesting given the practicality of visuo-haptic illusions such as providing rich haptic feedback while reducing the total number of props in comparison with the number of virtual objects (1:6 in our study). Our findings thus invite designers and researchers to consider visuo-haptic illusions to help operators to learn motor skills in an immersive, cost-effective, and less-maintenance environment.

## ACKNOWLEDGMENTS

We would like to thank Flavien Lebrun and Catherine Geslain-Lanéelle for their valuable feedback, as well as the participants of the experiment. The code is available at https://github.com/BenoitGeslain/Transfer-Illusions.

## REFERENCES

[1] 2021. *Scale1TM*. https://www.haption.com/en/products-en/scale-one-en.html

[2] Gunnar Ahlberg, Lars Enochsson, Anthony G. Gallagher, Leif Hedman, Christian Hogman, David A. McClusky, Stig Ramel, C. Daniel Smith, and Dag Arvidsson. 2007. Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. *American Journal of Surgery* 193, 6 (June 2007), 797–804. https://doi.org/10.1016/j.amjsurg.2006.06.050

[3] Mahdi Azmandian, Mark Hancock, Hrvoje Benko, Eyal Ofek, and Andrew D. Wilson. 2016. Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences. In *Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI '16*. ACM Press, Santa Clara, California, USA, 1968–1979. https://doi.org/10.1145/2858036.2858226

[4] C. Basdogan, C.-H. Ho, and M.A. Srinivasan. 2001. Virtual environments for medical training: graphical and haptic simulation of laparoscopic common bile duct exploration. *IEEE/ASME Transactions on Mechatronics* 6, 3 (Sept. 2001), 269–285. https://doi.org/10.1109/3516.951365

[5] C. Basdogan, M. Sedef, M. Harders, and S. Wesarg. 2007. VR-Based Simulators for Training in Minimally Invasive Surgery. *IEEE Computer Graphics and Applications* 27, 2 (March 2007), 54–66. https://doi.org/10.1109/MCG.2007.51

[6] Donald J Berndt and James Clifford. 1994. Using dynamic time warping to find patterns in time series. In *KDD workshop, Vol. 10. Seattle, WA, USA*,: 359–370.

[7] Lazar Bibin, Anatole Lécuyer, Jean-Marie Burkhardt, Alain Delbos, and Madeleine Bonnet. 2008. SAILOR: A 3-D Medical Simulator of Loco-regional Anaesthesia Based on Desktop Virtual Reality and Pseudo-haptic Feedback. In *Proceedings of the 2008 ACM Symposium on Virtual Reality Software and Technology (VRST '08)*. ACM, New York, NY, USA, 97–100. https://doi.org/10.1145/1450579.1450600
event-place: Bordeaux, France.

[8] C. Bryanton, J. Bossé, M. Brien, J. McLean, A. McCormick, and H. Sveistrup. 2006. Feasibility, motivation, and selective motor control: virtual reality compared to conventional home exercise in children with cerebral palsy. *Cyberpsychology & Behavior: The Impact of the Internet, Multimedia and Virtual Reality on Behavior and Society* 9, 2 (April 2006), 123–128. https://doi.org/10.1089/cpb.2006.9.123

[9] M. E. Bunch. 1946. Retroactive inhibition or facilitation from interpolated learning as a function of time. *Journal of Comparative Psychology* 39, 5 (1946), 287–291. https://doi.org/10.1037/h0055990 Place: US Publisher: Williams & Wilkins Company.

[10] Lung-Pan Cheng, Eyal Ofek, Christian Holz, Hrvoje Benko, and Andrew D. Wilson. 2017. Sparse Haptic Proxy: Touch Feedback in Virtual Environments Using a General Passive Prop. In *Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17)*. ACM, New York, NY, USA, 3718–3728. https://doi.org/10.1145/3025453.3025753 event-place: Denver, Colorado, USA.

[11] Claudiu Bogdan Ciumedean, Cristian Patras, Mantas Cibulskis, Norbert Váradi, and Niels C. Nilsson. 2020. Mission Impossible Spaces: Using Challenge-Based Distractors to Reduce Noticeability of Self-Overlapping Virtual Architecture. In *Symposium on Spatial User Interaction (Virtual Event, Canada) (SUI '20)*. Association for Computing Machinery, New York, NY, USA, Article 6, 4 pages. https://doi.org/10.1145/3385959.3418453

[12] Franck Crison, Anatole Lécuyer, Av A. Savary, Daniel Mellet-d'Huart, Jean-Marie Burkhardt, and Jean-Louis Dautin. 2004. The Use of Haptic and Pseudo-Haptic Feedback for the Technical Training of Milling. In *EuroHaptics 2004 Conference poster, Munich, Germany*.

[13] Iwan de Kok, Julian Hough, Felix Hülsmann, Mario Botsch, David Schlangen, and Stefan Kopp. 2015. A Multimodal System for Real-Time Action Instruction in Motor Skill Learning. In *Proceedings of the 2015 ACM on International Conference on Multimodal Interaction* (Seattle, Washington, USA) (ICMI '15). Association for Computing Machinery, New York, NY, USA, 355–362. https://doi.org/10.1145/2818346.2820746

[14] James J. Gibson. 1933. Adaptation, after-effect and contrast in the perception of curved lines. *Journal of Experimental Psychology* 16, 1 (1933), 1–31. https://doi.org/10.1037/h0074626

[15] James G. Greeno, Joyce L. Moore, and David R. Smith. 1993. Transfer of situated learning. In *Transfer on trial: Intelligence, cognition, and instruction*. Ablex Publishing, Westport, CT, US, 99–167.

[16] T. Gutiérrez, J. Rodriguez, Y. Vélaz, S. Casado, A. Suescun, and E. J. Sánchez. 2010. IMA-VR: a multimodal virtual training system for skills transfer in Industrial Maintenance and Assembly tasks. In *19th International Symposium in Robot and Human Interactive Communication*. 428–433. https://doi.org/10.1109/ROMAN.2010.5598643

[17] Dustin T. Han, Mohamed Suhail, and Eric D. Ragan. 2018. Evaluating Remapped Physical Reach for Hand Interactions with Passive Haptics in Virtual Reality. *IEEE Transactions on Visualization and Computer Graphics* 24, 4 (April 2018), 1467–1476. https://doi.org/10.1109/TVCG.2018.2794659

[18] Hermann Helmholtz and James Powell Cocke. 1926. Helmholtz's Treatise on Physiological Optics. Nature 118, 2959 (July 1926), 74. https://doi.org/10.1038/1180740

[19] S. J. Henderson and S. K. Feiner. 2011. Augmented reality in the psychomotor phase of a procedural task. In *2011 10th IEEE International Symposium on Mixed and Augmented Reality*. 191–200. https://doi.org/10.1109/ISMAR.2011.6092386

[20] Nicholas Ho, Pooi-Mun Wong, Matthew Chuai, and Chee-Kong Chui. 2018. Virtual reality training for assembly of hybrid medical devices. *Multimedia Tools and Applications* 77, 23 (Dec. 2018), 30651–30682. https://doi.org/10.1007/s11042-018-6216-x

[21] Yu-Hsuan Huang, Hao-Yu Chang, Wan-ling Yang, Yu-Kai Chiu, Tzu-Chieh Yu, Pei-Hsuan Tsai, and Ming Ouyoung. 2018. CatAR: A Novel Stereoscopic Augmented Reality Cataract Surgery Training System with Dexterous Instruments Tracking Technology. In *Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18)*. Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3173049

[22] Luv Kohli. 2010. Redirected touching: Warping space to remap passive haptics. In *2010 IEEE Symposium on 3D User Interfaces (3DUI)*. 129–130. https://doi.org/10.1109/3DUI.2010.5444703

[23] Luv Kohli, Eric Burns, Dorian Miller, and Henry Fuchs. 2005. Combining passive haptics with redirected walking. In *Proceedings of the 2005 international conference on Augmented tele-existence - ICAT '05*. ACM Press, Christchurch, New Zealand, 253. https://doi.org/10.1145/1152399.1152451

[24] Luv Kohli, M. C. Whitton, and F. P. Brooks. 2013. Redirected Touching: Training and adaptation in warped virtual spaces. In *2013 IEEE Symposium on 3D User Interfaces (3DUI)*. 79–86. https://doi.org/10.1109/3DUI.2013.6550201

[25] J. J. Kozak, Peter A. Hancock, E. J. Arthur, and Susan T. Chrysler. 1993. Transfer of training from virtual reality. *Ergonomics* 36, 7 (July 1993), 777–784. https://doi.org/10.1080/0014013908967941

[26] Belinda Lange, Sebastian Koenig, Chien-Yen Chang, Eric McConnell, Evan Suma, Mark Bolas, and Albert Rizzo. 2012. Designing informed game-based rehabilitation tasks leveraging advances in virtual reality. *Disability and Rehabilitation* 34, 22 (2012), 1863–1870. https://doi.org/10.3109/09638288.2012.670029

[27] Anatole Lecuyer, Sabine Coquillart, Abderrahman Kheddar, Paul Richard, and Philippe Coiffet. 2000. Pseudo-haptic feedback: can isometric input devices simulate force feedback?. In *Proceedings IEEE Virtual Reality 2000 (Cat. No.00CB37048)*. 83–90. https://doi.org/10.1109/VR.2000.840369

[28] R. B. Loftin and P. Kenney. 1995. Training the Hubble space telescope flight team. *IEEE Computer Graphics and Applications* 15, 5 (1995), 31–37. https://doi.org/10.1109/38.403825

[29] Anatole Lécuyer. 2009. Simulating Haptic Feedback Using Vision: A Survey of Research and Applications of Pseudo-haptic Feedback. *Presence: Teleoper. Virtual Environ.* 18, 1 (2009), 39–53. https://doi.org/10.1162/pres.18.1.39

[30] Brandon J. Matthews, Bruce H. Thomas, Stewart Von Itzstein, and Ross T. Smith. 2019. Remapped Physical-Virtual Interfaces with Bimanual Haptic Retargeting. In *2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)*. IEEE, Osaka, Japan, 19–27. https://doi.org/10.1109/VR.2019.8797974

[31] Helen C. Miles, Serban R. Pop, Simon J. Watt, Gavin P. Lawrence, and Nigel W. John. 2012. A review of virtual environments for training in ball sports. *Computers & Graphics* 36, 6 (Oct. 2012), 714–726. https://doi.org/10.1016/j.cag.2012.04.007

[32] Toemme Noesselt, Daniel Bergmann, Maria Hake, Hans-Jochen Heinze, and Robert Fendrich. 2008. Sound increases the saliency of visual events. *Brain research* 1220 (02 008), 157–63. https://doi.org/10.1016/j.brainres.2007.12.060

[33] N. Pathomaree and S. Charoenseang. 2005. Augmented reality for skill transfer in assembly task. In *ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication*, 2005. 500–504. https://doi.org/10.1109/ROMAN.2005.1513829

[34] Bernd Petzold, Michael F. Zaeh, Berthold Faerber, Barbara Deml, Hans Egermeier, Johannes Schilp, and Stella Clarke. 2004. A Study on Visual, Auditory, and Haptic Feedback for Assembly Tasks. *Presence* 13, 1 (Feb. 2004), 16–21. https://doi.org/10.1162/105474604774048207

[35] Claude Prablanc, Francesco Panico, L Fleury, Laure Sella, T Nijboer, Shigeru Kitazawa, and Yves Rossetti. 2019. Adapting terminology Clarifying prism adaptation vocabulary, concepts, and methods. *Neuroscience Research* (March 2019). https://doi.org/10.1016/j.neures.2019.03.003

[36] Georg Rauter, Roland Sigrist, Robert Riener, and Peter Wolf. 2015. Learning of Temporal and Spatial Movement Aspects: A Comparison of Four Types of Haptic Control and Concurrent Visual Feedback. *IEEE Transactions on Haptics* 8, 4 (Oct. 2015), 421–433. https://doi.org/10.1109/TOH.2015.2431686

[37] Sharif Razaque, Zachariah Kohn, and Mary C. Whitton. 2001. Redirected Walking. (2001). https://doi.org/10.2312/egs.20011036

[38] Michael Rietzler, Florian Geiselhart, Jan Gugenheimer, and Enrico Rukzio. 2018. Breaking the Tracking: Enabling Weight Perception Using Perceivable Tracking Offsets. In *Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18)*. ACM, New York, NY, USA, 128:1–128:12. https://doi.org/10.1145/3173574.3173702 event-place: Montreal QC, Canada.

[39] Irvin Rock and Jack Victor. 1964. Vision and Touch: An Experimentally Created Conflict between the Two Senses. *Science* 143, 3606 (Feb. 1964), 594–596. https://doi.org/10.1126/science.143.3606.594

[40] Jorge Rodriguez, Teresa Gutiérrez, Emilio J Sánchez, Sara Casado, and Iker Aguinaga. 2012. Training of procedural tasks through the use of virtual reality and direct aids. *Virtual Reality and Environments* (2012), 43–68.

[41] Juan Jesús Roldán, Elena Crespo, Andrés Martin-Barrio, Elena Peña-Tapia, and Antonio Barrientos. 2019. A training system for Industry 4.0 operators in complex assemblies based on virtual reality and process mining. *Robotics and Computer-Integrated Manufacturing* 59 (2019), 305–316. https://doi.org/10.1016/j.crim.2019.05.004

[42] Eva-Lotta Sallnäs, Kirsten Rassmus-Gröhn, and Calle Sjöström. 2000. Supporting Presence in Collaborative Environments with Haptic Force Feedback. *ACM Trans. Comput.-Hum. Interact.* 7, 4 (Dec. 2000), 461–476. https://doi.org/10.1145/365058.365086

[43] Majed Samad, Elia Gatti, Anne Hermes, Hrvoje Benko, and Cesare Parise. 2019. Pseudo-Haptic Weight: Changing the Perceived Weight of Virtual Objects By Manipulating Control-Display Ratio. In *Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19)*. ACM, New York, NY, USA, 320:1–320:13. https://doi.org/10.1145/3290605.3300550 event-place: Glasgow, Scotland Uk.

[44] Richard A. Schmidt and Timothy D. Lee. 2011. *Motor control and learning: A behavioral emphasis, 5th ed.* Human Kinetics, Champaign, IL, US.

[45] Lichao Shen, MHD Yamen Saraiji, Kai Kunze, Kouta Minamizawa, and Roshan Lalinha Peiris. 2020. Visuomotor Influence of Attached Robotic Neck Augmentation. In *Symposium on Spatial User Interaction* (Virtual Event, Canada) (SUI '20). Association for Computing Machinery, New York, NY, USA, Article 14, 10 pages. https://doi.org/10.1145/3385959.3418460

[46] Roland Sigrist, Georg Rauter, Robert Riener, and Peter Wolf. 2013. Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review. *Psychonomic Bulletin & Review* 20, 1 (Feb. 2013), 21–53. https://doi.org/10.3758/s13423-012-0333-8

[47] Mark K. Singley and John R. Anderson. 1987. A Keystroke Analysis of Learning and Transfer in Text Editing. *Human-Computer Interaction* 3, 3 (1987), 223–274.
https://doi.org/10.1207/s15327051hci0303_2

[48] J. Solis, C.A. Avizzano, and M. Bergamasco. 2003. Validating a skill transfer system based on reactive robots technology. In *The 12th IEEE International Workshop on Robot and Human Interactive Communication, 2003. Proceedings. ROMAN 2003.* IEEE, Millbrae, CA, USA, 175–180. https://doi.org/10.1109/ROMAN.2003.1251841

[49] Jonas Spillmann, Stefan Tuchschmid, and Matthias Harders. 2013. Adaptive Space Warping to Enhance Passive Haptics in an Arthroscopy Surgical Simulator. *IEEE Transactions on Visualization and Computer Graphics* 19, 4 (April 2013), 626–633. https://doi.org/10.1109/TVCG.2013.23

[50] Xing-Dong Yang, Walter F. Bischof, and Pierre Boulanger. 2008. Validating the Performance of Haptic Motor Skill Training. In *2008 Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems*. IEEE, Reno, NV, USA, 129–135. https://doi.org/10.1109/HAPTICS.2008.4479929

[51] Andre Zenner and Antonio Kruger. 2019. Estimating Detection Thresholds for Desktop-Scale Hand Redirection in Virtual Reality. In *2019 IEEE Conference on*

*Virtual Reality and 3D User Interfaces (VR)*. IEEE, Osaka, Japan, 47–55. https://doi.org/10.1109/VR.2019.8798143

[52] Yuhang Zhao, Cynthia Bennett, Hrvoje Benko, Ed Cutrell, Christian Holz, Meredith Ringel Morris, and Mike Sinclair. 2018. Enabling People with Visual Impairments to Navigate Virtual Reality with a Haptic and Auditory Cane Simulation. (Jan. 2018). https://www.microsoft.com/en-us/research/publication/enabling-people-visual-impairments-navigate-virtual-reality-haptic-auditory-cane-simulation-2/

[53] Yiwei Zhao and Sean Follmer. 2018. A Functional Optimization Based Approach for Continuous 3D Retargeted Touch of Arbitrary, Complex Boundaries in Haptic Virtual Reality. In *Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18)*. ACM, New York, NY, USA, 544:1–544:12. https://doi.org/10.1145/3173574.3174118