# Biomechanical fidelity of simulated pick-and-place tasks: impact of visual and haptic renderings

Simon Hilt, Thomas Meunier, Charles Pontonnier, Georges Dumont

► To cite this version:

Simon Hilt, Thomas Meunier, Charles Pontonnier, Georges Dumont. Biomechanical fidelity of simulated pick-and-place tasks: impact of visual and haptic renderings. IEEE Transactions on Haptics (ToH), 2021, 14 (3), pp.692-698. 10.1109/TOH.2021.3052658. hal-03093454

HAL Id: hal-03093454

https://inria.hal.science/hal-03093454

Submitted on 3 Jan 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
# Biomechanical fidelity of simulated pick-and-place tasks: impact of visual and haptic renderings

Simon Hilt, Thomas Meunier, Charles Pontonnier, and Georges Dumont,

*The authors are with Univ Rennes, CNRS, Inria, IRISA - UMR 6074, F-35000 Rennes, France.*

***Abstract***—Virtual environments (VE) and haptic interfaces (HI) tend to be introduced as virtual prototyping tools to assess ergonomic features of workstations. These approaches are cost-effective and convenient since working directly on the Digital Mock-Up in a VE is preferable to constructing a physical mock-up in a Real Environment (RE). However it can be usable only if the ergonomic conclusions made from the VE are similar to the ones you would make in the real world. This study aims at evaluating the impact of visual and haptic renderings in terms of biomechanical fidelity for pick-and-place tasks. Fourteen subjects performed time-constrained pick-and-place tasks in RE and VE with a real and a virtual, haptic driven object at three different speeds. Motion of the hand and muscles activation of the upper limb were recorded. A questionnaire assessed subjectively discomfort and immersion. The results revealed significant differences between measured indicators in RE and VE and with real and virtual object. Objective and subjective measures indicated higher muscle activity and higher length of the hand trajectories in VE and with HI. Another important element is that no cross effect between haptic and visual rendering was reported. These results confirmed that such systems should be used with caution for ergonomics evaluation, especially when investigating postural and muscle quantities as discomfort indicators. The last contribution of the paper lies in an experimental setup easily replicable to assess more systematically the biomechanical fidelity of virtual environments for ergonomics purposes.

***Index Terms***—Biomechanics-Virtual Reality-Haptics-Human Factors and Ergonomics

## I. INTRODUCTION

THE development of virtual reality headsets and glasses has been accelerated in the last few years, becoming popular in many applications. One of those applications is the virtual prototyping of workstations and products, that is a real asset in the industry [1], [2]. The interest of the use of virtual environment for applied ergonomics has been shown for more than twenty years [3]. More recently, studies have examined the use of virtual prototyping for the evaluation of ergonomics and human performance during the design phase [4]. Indeed, virtual prototyping is more cost-effective and convenient since creating and modifying a Digital Mock-Up (DMU) in a virtual environment (VE) is easier and less time-consuming than creating and modifying a physical one [5], [6]. The integration of physical risk factors assessment - ergonomics evaluation - in this prototyping has been recently introduced, showing interesting results for manual handling operations [3], [7], [8]. In those applications, the end-user - ideally the operator for which the design is realized, is immersed in a VR-based simulator and interacts with it as he would interact in the real world. One of the major features to be fulfilled to make such an

evaluation valuable is to guarantee that results and conclusions obtained in VE are similar to the ones you would obtain in the real world. Indeed, the usefulness of the results is deeply related to the level of fidelity of the simulator – the extent to which interactions in a VE are indistinguishable from interactions in a real environment (RE). Fidelity of VEs has been described in literature as a composite feature of three main dimensions: simulation, display and interaction fidelity [9], [10], to which we can add a transverse dimension called biomechanical fidelity, or biofidelity - the degree of similarity between motions, forces, and tasks realized in RE and VE at a biomechanical level (the level used to assess physical risk factors in ergonomics) [11]. A high biomechanical fidelity rate would ensure that the conclusions of studies using VE may be valid for the real world. Indeed it explains the idea that biomechanical quantities are consistent between a real and an equivalent virtual activity.

Biomechanical fidelity assessments were mostly conducted on tasks involving low levels of interaction or force, enabling the use of tracked devices with no haptic feedback to interact with the virtual scene [12], [13], [14], [15]. However, for more complex interactions and tasks involving moderate to high level of forces, the interaction must be rendered through haptic interfaces (HI). The current study aimed at assessing the biomechanical fidelity for pick-and-place tasks realized in RE and VE with or without haptic interaction.

A few studies investigated the fidelity of a virtual scene involving haptic rendering for ergonomics purposes. First, in [16], sorting tasks with low masses (wooden pieces) were assessed in real, virtual, and virtual plus force feedback conditions - realized with a 6-DOF haptic device based on impedance control (Virtuose 6D, Haption SA). Postures and muscle activities were assessed and compared between environments and between subjects, showing that the haptic device limited the interaction fidelity, with a poor level of similarity between joint angle trajectories and muscle activities recorded in RE and VE. The main reason behind this result was the relative ineffectiveness of the mechanical structure of the haptic device to enable complex manipulations involving large range of motions, more than the control scheme applied to it. Second, in [17] was proposed an approach in which was gradually evaluated the impact of a) the virtual head-set b) the presence of an haptic device c) the haptic rendering for assembly tasks. They used the HUG [18], consisting of two torque-controlled 6-dofs haptic robots. Three tasks were investigated: frontal and side insertion, and railing. Results showed that all synthetic interactions increased the time to
complete the tasks. The force feedback generated by the HI was always bigger than the real interaction. The muscle activity was higher with the synthetic interactions than with the real ones, especially when interacting with the haptic device. The ratings of subjective overall, contact and manipulation realism and the perception of mental and physical workloads were negatively affected by each synthetic interaction. These results clearly mitigate the usefulness of such approaches and make the VE not realistic enough to be used as it is.

The approach of the current study is similar to those presented in the previous paragraph. We developed an original setup, enabling to separate the effect of visual and haptic renderings in the scene in order to investigate individual and combined effects of those modes of immersion and interaction on the biomechanical behavior of the subject during the task. The study focused particularly on the mixed effects of the renderings on the biomechanical fidelity of the system to simulate pick-and-place tasks. The paper is first presenting the experimental protocol and the statistical processing, and then presents and discusses the results with regard to the aforementioned objectives.

## II. MATERIALS AND METHODS

In the present study, we designed and conducted experiments on simulated pick-and-place tasks with real or virtual visual feedback (VF) and with two levels of haptic interaction (HIn), a real interaction with a real object and a synthetic interaction with a commercial HI, comparing several indicators based on biomechanical quantities.

<figure><img src="image_1.png" /><figcaption>Fig. 1. Top left: Virtuose6D 3D mock-up. Top right: Kinematic scheme of the HI. Bottom: Detailed haptic interaction. End-user interacted with the haptic display by moving the end-effector $X_r$. The position of each joint $X_v^*$ was captured, digitalized and sent to the VE. It was coupled to the object scene through a virtual spring-damper because of the admittance control of Rigidbody, the physics engine used. The resulting force $F_v^*$ was low-pass filtered and sent to the HI and used to apply to the end-effector as the force feedback.</figcaption></figure>

### A. Subjects

Fourteen subjects, twelve males and two females, participated in this study after giving their informed consent. They were relatively experienced in VE (average experience of 3.6 ± 2.0 on a seven-point scale) and nearly novice with HI (average experience of 2.0 ± 1.1 on a seven-point scale). Mean subject age was 25.6 ± 9.3 y.o. They were all right-handed and none of them declared musculoskeletal disorder within the last six months. The study was approved by the Inria national ethics committee (COERLE 2017-13).

### B. Task

The chosen task was a simplified pick-and-place task (Fig. 2) that can be found in industrial situations. It consisted in moving (back and forth) a 470g object between two target zones placed on a table and spaced 36.4cm apart. Objects handle was a 3D printed reproduction of the HI handle in order to avoid any change in the grip. Each target zone included a rectangle and a circle of the size of the object base and handle. The subjects were sitting on a chair in front of the table and placed in order to have their arm along their body and the elbow flexed at 90 degrees when the object was on the nearest target. A cylindrical obstacle was placed between these two zones.

During a trial, the subject performed five back and forth motions with the object at each of the three following speed rates: slow, medium, and fast (1.5, 1 and 0.7 second to achieve a motion from one target to another respectively). It corresponded to 10 pick-and-place repetitions per speed. A pause of 1 second was respected between each repetition and a pause of 5 seconds was respected between each change of pace leading to a total duration of 69 seconds. A metronome indicated the pace to be followed by the subject. Before the start, the subject had two training trials with real visual feedback/object and virtual visual feedback/object to get used to the different rhythms to be followed and to the use of the head-mounted display (HMD) and the haptic interface. The trials were performed twice per visual feedback (VF)/haptic interaction (HIn) modes combination: interaction with real object without HMD, interaction with real object with wearing HMD, interaction with HI without wearing HMD and interaction with HI with wearing HMD. Figure 2 shows the four different combinations of VF and HIn. In summary, the subject had to perform eight sequences of fifteen back and forth motions (30 pick-and-place repetitions) for a total of 240 pick-and-place repetitions.

During the experiment, the order of the different combinations of VF and HIn was randomized and the modes of VF and HIn were announced before each trial. However speed conditions were systematically realized from the slowest to the fastest for all trials.

### C. Setup

1) Virtual environment: The VE was designed and implemented in Unity3D 5.6.1¹ (Unity technologies™) in order to

<footnote n="1">¹https://unity.com/</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 2. Top left: Experimental setup with two targets for pick-and-place tasks. A cylindrical obstacle 8cm in diameter and 10cm high was placed between the targets. The HI (Virtuose6D) was placed close by so that the movement was within its range of accessibility. The subject was equipped with a Vive tracker to record the position and orientation of his hand and with a myo bracelet to record the electromyographic activity of his arm. Finally, the subject was equipped with an HTC Vive Pro helmet during the virtual immersion where the virtual scene was projected. Bottom left: This virtual scene included the same elements as the real scene except for the HI with the addition of a virtual hand collocated thanks to the Vive Tracker. Right: Illustration of the four combinations of immersion/interaction modes.</figcaption></figure>

mimic the RE - except the representation of the HI. VE was used for both the virtual visual feedback and to control HI. The 3D representation of the manipulated object was derived from the digital mock-up used to fabricate the real one. The physics of the scene was simulated using the Unity module Rigidbody. Notably, the friction between the object and the table has been implemented as a Coulomb friction with usual coefficients of wood/steel friction: 0.5 static friction and 0.5 dynamic friction. An head mounted display (HMD) HTC Vive pro<fnref n="2" /> (HTC™) was used to display the VE to the subject. The SteamVR plugin has been used to ensure compatibility between Unity3D and the HTC Vive pro. The standard calibration protocol proposed by the software SteamVR was set before each experiment. It consisted in the set-up of the equipment (HMD, controllers and Vive Tracker), the set up of the desk as the floor to ensure that the virtual desk and the real one were parallel and at the same height and the set up of the reachable area of the subject. Finally, to ensure the collocation of the two targets, We performed a rigid solid calibration with three degrees of freedom (two horizontal translations and the vertical rotation) by putting consequently the tracker at the bottom right of the right target and at the top left of the left target.

2) *Haptic interaction:* Synthetic haptic interaction was realized with a Virtuose 6D High Force<fnref n="3" /> (Haption™), a six degree of freedom impedance HI providing a continuous force of 30N and a continuous torque of 1.4N.m. As the Rigidbody physics engine is in admittance mode, the coupling (see Fig. 1) between the HI and the physics engine was provided by a

virtual spring-damper of 250N.m⁻¹ stiffness and 15N.s.m⁻¹ damping. This setting was obtained from a previous study [19] and a pilot experiment. This coupling and the physics engine were running at 250Hz due to computing power limitations.

## D. Recordings

Muscle activity of the upper arm was recorded with a Myo armband (Thalmic Labs™). Elastic armband Myo is a low cost electromyographic (EMG) sensor system consisting in 8 EMG sensors sampled at 200Hz. It was placed one-third of the distance between the lateral epicondyle and the acromion from the coronoid fossa (see Fig. 3).

<figure><img src="image_3.png" /><figcaption>Fig. 3. Myo Armband position on subject upper arm (left) and sectional view (right). The Myo Armband, composed of 8 EMG electrode pairs, was placed on the arm of the subject. Electrode pair 4 served as reference and was always placed on the biceps brachii. The averaged muscle activation of biceps brachii (AMAb) was obtained by averaging the measure from electrodes pairs 3-4-5 (red electrode pairs), as well as the averaged muscle activation of triceps brachii (AMAt) with the electrode pairs 7-8-1 (blue electrode pairs).</figcaption></figure>

<footnote n="2">²https://vive.com/</footnote>

<footnote n="3">³https://haption.com/</footnote>
Motion of the hand was captured with a HTC Vive tracker set on the back of the hand in the middle of the second and third metacarpals. The position of the hand recorded by the tracker was sampled at 60Hz. The accuracy of such trackers has a standard deviation of 5mm [20].

At the end of the experiment, subjects answered a questionnaire including items about haptic interaction and items from the Witmer-Singer [21] presence questionnaire, on a 7 points Likert-scale [22].

## E. Indicators

1) **Averaged Muscle Activations (AMA):** Averaged Muscle Activations of biceps brachii (AMAb) and of triceps brachii (AMAt) were the averaged measures of the 3 electrode pairs of the Myo armband placed on the biceps and the 3 placed on the triceps (see Fig. 3). AMAb and AMAt gave a fairly good overview of the elbow flexor and extensor muscles activity during the task to compare the different conditions. To compute these indicators, the electrodes signals were detrended and filtered with a root mean square filter with a window set at 200ms according to [23]. These filtered signals were averaged for each pick-and-place repetition to obtain AMAb and AMAt per repetition. The AMAb and AMAt where then averaged per condition.

2) **Hand motion:** To estimate the spatial variability of the pick-and-place tasks among repetitions and conditions, the average distance of all trajectories to the median trajectory plane (APD) was studied. The median trajectory plane was defined in a least-squares sense among all repetitions of a given condition. Spatial variability is the component of the motor control variability observable in the task space, and it is relevant to compare this variability among conditions to understand how controlled is the motion performed. Indeed, this variability will be impacted by key control processes of the task realization - planning, correction for example, as well as with the familiarity of the subject with the task and the environment [24], [16], [25], [15]. In this study, APD was supposed to be a relevant metrics of this variability.

The total path length (TL) of the hand trajectory during each trial was calculated from a linear interpolation between the positions recovered by the HTC vive tracker projected in the average plane during a repetition. TL was supposed to be representative of the energy expense associated to the task (a shorter path meaning a lower energy expense). TL was averaged per condition.

The average distance (AD) between the starting and ending positions of the object on the work plane and the average avoidance height (AAH) were computed per condition, by averaging these values among repetitions. This indicator was particularly interesting to assess the perceptive differences between RE and VE.

The trajectory and three indicators - APD, TL and AD - are represented in Fig. 4.

Finally, the average maximum speed (AMS) was calculated. The position of the hand was filtered using a low pass filter with a cutoff frequency of 10Hz to suppress noise due to the accuracy of the tracker. This filtered position was numerically

differentiated to compute the speed of the hand during the task. AMS was averaged per condition. It was supposed to be a relevant measure of the effectiveness with which the task was performed (the higher the speed was, the higher the task was effective).

<figure><img src="image_4.png" /><figcaption>Fig. 4. Trajectory of one repetition (gray line) projected on the median plane $P$ obtained from the 10 repetitions -5 back and forth - of a given trial (black line). The average distance to the median plane (APD) was calculated from the average of each distance (d). The average distance (AD) and the average avoidance height (AAH) are also represented.</figcaption></figure>

## F. Statistics

Visual feedback type (set to real and virtual), haptic interaction type (set to real and synthetic) and speed (set to slow, medium or fast) were introduced as factors in a full factorial repeated measure of variance (ANOVA) of the dependent variables, which were the objective indicators (AMAb, AMAt, APD, TL, AD, AAH, and AMS). The probability value was adjusted through a Greenhouse-Geisser sphericity correction. The level of confidence was set to $p < 0.05$. A post hoc Tukey honestly significant difference (HSD) test was performed on the significant factors identified with the ANOVA.

# III. RESULTS

## A. Objective indicators

1) **AMAs:** According to Table I, AMAb - corresponding to the main muscles involved in the task realization - increased with the speed (+17% from low to medium; +14% from medium to high). In addition, AMAb was higher during virtual visual feedback (+6%) and synthetic haptic interaction (+5%), with regard to real conditions.

AMAt increased with virtual visual feedback (+36%) and speed (+11% from low to medium; +24% from medium to high), corresponding to higher muscle co-contraction during the task.

2) **Hand motion:** APD did not depend on the different factors and mean APD was $6 \pm 6$mm.

TL depended significantly on the level of immersion, interaction and speed. TL decreased with speed (-5% from low to medium; -4% from medium to high) and synthetic haptic interaction (-11%), but it increased with virtual visual feedback (+5%).

AD depended only on the speed, decreasing as the speed increased (-3% from low to medium; -11% from medium to high). At slow speed, AD, whether in immersion and
<table><caption>TABLE I
FULL FACTORIAL REPEATED MEASURES ANALYSIS OF VARIANCE RESULTS AND MARGINAL MEANS. SIGNIFICANT DIFFERENCES (p < 0.05, ADJUSTED THROUGH A GREENHOUSE-GEISSER SPHERICITY CORRECTION) ARE EMPHASIZED IN BOLD AND MARKED WITH AN ASTERISK (*)</caption><thead><tr><th rowspan="2">Factors</th><th rowspan="2"></th><th colspan="7">Indicators</th></tr><tr><th>AMAb<br>8-bit data</th><th>AMAt<br>8-bit data</th><th>APD<br>mm</th><th>TL<br>mm</th><th>AD<br>mm</th><th>AAH<br>mm</th><th>AMS<br>m.s<sup>-1</sup></th></tr></thead><tbody><tr><td rowspan="3">HIn</td><td>pValue</td><td><strong>1.2e-2*</strong></td><td>5.6e-1</td><td>8.9e-1</td><td><strong>6.8e-3*</strong></td><td>6.6e-1</td><td><strong>3.5e-3*</strong></td><td><strong>3.8e-5*</strong></td></tr><tr><td>Real</td><td>20.8 ± 1.7</td><td>5.5 ± 0.5</td><td>6.3 ± 1.0</td><td>6650 ± 280</td><td>338 ± 4</td><td>107 ± 9</td><td>0.39 ± 0.01</td></tr><tr><td>Virtual</td><td>22.1 ± 1.7</td><td>5.6 ± 0.5</td><td>5.4 ± 0.2</td><td>5950 ± 130</td><td>339 ± 5</td><td>82 ± 6</td><td>0.33 ± 0.01</td></tr><tr><td rowspan="3">VF</td><td>pValue</td><td><strong>8.5e-4*</strong></td><td><strong>8.0e-3*</strong></td><td>3.9e-1</td><td><strong>5.4e-4*</strong></td><td>4.8e-1</td><td><strong>3.9e-5*</strong></td><td>1.5e-1</td></tr><tr><td>Real</td><td>20.9 ± 1.7</td><td>5.4 ± 0.5</td><td>6.3 ± 0.3</td><td>6150 ± 190</td><td>337 ± 5</td><td>88 ± 7</td><td>0.35 ± 0.01</td></tr><tr><td>Virtual</td><td>22.0 ± 1.7</td><td>5.8 ± 0.5</td><td>6.1 ± 0.9</td><td>6450 ± 200</td><td>339 ± 5</td><td>101 ± 7</td><td>0.36 ± 0.01</td></tr><tr><td rowspan="4">Speed</td><td>pValue</td><td><strong>1.2e-9*</strong></td><td><strong>2.9e-4*</strong></td><td>3.7e-1</td><td><strong>7.2e-5*</strong></td><td><strong>3.2e-4*</strong></td><td><strong>1.8e-6*</strong></td><td><strong>1.2e-3*</strong></td></tr><tr><td>Low</td><td>18.4 ± 1.6</td><td>4.9 ± 0.5</td><td>7.0 ± 0.9</td><td>6610 ± 240</td><td>358 ± 2</td><td>106 ± 8</td><td>0.34 ± 0.02</td></tr><tr><td>Medium</td><td>21.5 ± 1.7</td><td>5.3 ± 0.5</td><td>6.1 ± 0.3</td><td>6280 ± 200</td><td>347 ± 4</td><td>98 ± 7</td><td>0.35 ± 0.01</td></tr><tr><td>Fast</td><td>24.4 ± 1.8</td><td>6.5 ± 0.6</td><td>6.5 ± 0.3</td><td>6010 ± 150</td><td>310 ± 10</td><td>80 ± 6</td><td>0.38 ± 0.01</td></tr><tr><td rowspan="3">HInxVF</td><td>pValue</td><td>1.7e-1</td><td>2.2e-1</td><td>1.5e-1</td><td>5.8e-1</td><td>9.1e-1</td><td>1.0e-1</td><td>5.2e-1</td></tr><tr><td>pValue</td><td>1.1e-1</td><td><strong>4.1e-2*</strong></td><td>2.3e-1</td><td>8.4e-1</td><td>3.4e-1</td><td>1.5e-1</td><td>2.3e-1</td></tr><tr><td>pValue</td><td><strong>4.5e-3*</strong></td><td>3.1e-1</td><td>1.7e-1</td><td><strong>7.1e-3*</strong></td><td>3.1e-1</td><td><strong>3.4e-2*</strong></td><td>5.5e-1</td></tr><tr><td>HInxVFxS</td><td>pValue</td><td>1.8e-1</td><td>2.1e-1</td><td>3.9e-1</td><td>1.4e-1</td><td>9.9e-1</td><td>9.1e-1</td><td>5.0e-1</td></tr></tbody></table>

interaction types, was close to the physical distance between targets (36.1cm for AD vs. 36.4cm for the physical distance).

AAH depended significantly on the type of interaction and especially on the type of immersion and speed. AAH was lower during synthetic haptic interaction (-23%) and higher speed (-8% from low to medium; -22% from medium to high), however it was higher in virtual visual feedback (+15%).

AMS depended significantly on the type of interaction, immersion and speed. AMS increased with speed (+3% from low to medium; +9% from medium to high) and virtual visual feedback (+3%), but decreased in synthetic haptic interaction (-9%).

## B. Subjective results

Only relevant subjective results were reported here, among the complete questionnaire proposed to the subjects. Results showed that the task specifications were adapted to the HI used: its mechanical structure did not disturb the subjects in their motion (2.79 ± 1.72). Moreover, the force feedback seemed realistic (5.07 ± 1.00) and the feeling of contact between the manipulated object and its environment did not perturb the subjects (2.71 ± 1.59). However the manipulation of the object through the HI seemed to be more difficult than the real manipulation (5.00 ± 0.68). The collocation of the elements on the scene seemed to be quite satisfying (hand: 5.00 ± 1.47, real object: 4.64 ± 1.65 and virtual object (haptic driven): 5.93 ± 1.07) and subjects found that the virtual scene rendering was very good (5.79 ± 1.12) with no perceptible delay (1.43 ± 0.65). Overall, subjects evaluated the virtual immersion to a satisfying level of presence. Finally, they preferred to have virtual visual feedback when interacting with the haptic feedback (2.93 ± 1.73; 1-Virtual 7-Real visual feedback) and preferred to have real visual feedback when interacting with the real object (5.79 ± 1.63).

## IV. DISCUSSION

Overall the results show significant differences between the conditions of the current study in terms of muscle activation,

trajectory and kinematics. As expected synthetic haptic interaction and virtual visual feedback altered significantly the pick-and-place task. The increase of the speed condition led to an increase in inertial effects, therefore the force required to move the object was greater. Furthermore, according to [26], [27], to perform a precise task at a higher rate it is necessary to increase the stiffness of the joints by increasing the muscle co-contraction. These observations explain the increase in activation for both muscle groups with regard to speed. On the other hand, the increase in speed decreased the accuracy and shortened the trajectory, reducing the average distance between starting and ending positions and the avoidance height [28]. Subjects may had issues to perform the task at higher speeds, resulting in narrower trajectories as shown by the decrease of TL accompanying the increase of AMs.

The use of a HI resulted in an increase in muscle activity of the biceps, which is the one of the main muscles involved in the task, assuming that the end-user must make an effort to move both the virtual object and the HI handle [17]. On can suppose that the effects of inertia and friction of the HI were added to the effects of inertia of the object. The subjective questionnaire evidenced that the feeling of effort during the task was more important when using the HI, as already explained in [19] for low masses manipulations. On the other hand, the synthetic haptic interaction led to a decrease in the AAH and thus in the total length of the trajectory TL. This could be explained on the one hand by the fact that the object has an increased visual burden compared to the HI handle. However, there is no significant difference between synthetic haptic interaction with real and virtual visual feedback, depreciating this assumption. In addition, the subject inexperience with HI can have led in hampering in using its entire accessible space. On the other hand, the increase of the effort inherent to the use of the HI may have caused the changes observed in the trajectory. Furthermore, this can explain the lower maximum speed with HI. The virtual object to handle was designed from the real object and mass properties were applied from those of the - estimated - real object. Even if we reported the same order of amplitude for
real and virtual forces arising from this object manipulation in [19], differences in mass properties between real and virtual object may have an impact on the behavior of the subject.

Finally, virtual visual feedback led to an increase in trajec-
tory length, with a significant increase in obstacle avoidance
height. This may be due to the error of perception of space
in virtual reality. According to [29], [30], [31], VE distance
perception is compressed so subjects tend to increase the safety
margin with the object. Additionally, no hand-eye calibration
was followed in the setup [32], that may have diminished the
distortion by applying corrective rules to the display. There
is a greater increase in triceps activity compared to biceps
activity than in the effect of speed. There is therefore an
increase mainly in the co-contraction of these muscles during
virtual visual feedback, that can be probably explained by a
unfamiliarity with the environment and therefore an increase
of trajectory control by stiffening the arm [14].

Surprisingly, no significant statistical interaction was found between immersion and interaction renderings for all the indicators. This is interesting and opens the door to study the effect of visual and haptic interaction separately for such tasks. However, subjects reported a preference to have all in virtual or all in real to perform the task, meaning that hybrid setups may be too unfamiliar to be properly handled by the user to perform this kind of task.

Another interesting result is the APD one, showing no
significant change in spatial variability from one setup to one
other. Indeed, authors assumed that this metric may have been
influenced by conditions, especially with the haptic interaction
with regard to the real object. The unfamiliarity of the subject
with haptics should have led to an increase in spatial variability
when performing the task. The subjects seemingly preferred
to adopt conservative strategies in order to perform the task
instead of keeping similar speed and trajectory between the
two conditions. In other words, they changed the way they
performed the task to adapt to the HI, that is also a strong
bias in assessing biomechanical quantities in such conditions.
The adaptation of the subject to the device change drastically
the macro (task) and micro (muscle) motor control for similar
conditions. Therefore, the similarity of the biomechanical
quantities between RE and VE can not be ensured. In addition,
only the trajectory of the hand was recorded. It would be
necessary to capture the whole body motion of the subject
in order to study in more detail the biomechanical fidelity of
this task and thus its similarity with the task in real world.
Indeed we cannot see if there is any compensation strategy
explaining changes in the trajectory of the hand.

As shown in Fig. 5, and as described above, the muscle ac-
tivation strategy was different between the real and simulated
task. Indeed the activation of the biceps was more important at
the beginning and end of the movement during the real task,
whereas during the simulated task it was more important either
at the end of the movement during the outward phase or at the
beginning of the movement during the return phase. Moreover,
there was a disparity between the rest phases after the outward
and return phases concerning the activation of the biceps and
triceps. During the rest phase on the farthest target, only the
activation of the triceps increased in the case of the real task

whereas the activation of the triceps and biceps increased
in the case of the simulated task, that is also explained by
a higher limb stiffness with the synthetic haptic rendering.
Finally the speed curves seemed to be typical bell-shaped
curves that are classically found for pointing tasks, in both
conditions [33]. This is the main limit of the current study:
these conclusions can only apply on pick-and-place setups and
tasks, that are really similar to pointing tasks. The task was
voluntary simplified and the timing was clearly controlled, that
may certainly have led to an adaptation of the subject to the
task.

This study exhibits other limitations. Firstly, the number of
subjects is quite limited and the population is not represen-
tative of people concerned by such an application, since the
majority of subjects were recruited in our laboratory. Secondly,
the choice to constrain time by sound signals probably led to
a learning effect to follow the rhythm during the experiment,
even if each subject had two training trials beforehand. The
number of repetitions being quite large (240 pick-and-place
movements), the effect of learning was supposedly limited.
The, this study observed only the muscle activity of the flexors
and extensors of the elbow as well as the kinematics of
the hand without taking into account the posture and global
activity of the subject, that may have revealed compensatory
strategies. Last, as mentioned before, hand-eye coordination
and a better estimation of mass properties of the virtual object
may also have diminish the differences between real and
synthetic modalities.

To sum up, for applications requiring a high level of biome-
chanical fidelity, such as industrial workstation ergonomics
assessment, the study confirms that virtual reality and HI are
subject to caution without improving the rendering. Indeed,
in addition to decreasing the end user's experience, these
haptic interaction and immersion modes bring a bias of the
order of 10% to 15% concerning muscle activity, trajectory
and kinematics. Few studies considered the improvement of
haptic rendering from a biomechanical point of view for
object manipulation [19], [34], [35], in contrast to methods for
improving contact and shock rendering. The main drawback
of HI in the case of object manipulation is the increased
force required to move the object by introducing friction and
additional inertia. However, rendering algorithms for haptic
interaction have been studied from a system point of view.
Most of these methods are based on the addition of one
or more feedback loops in the control structure of the HI.
These feedback loops are mostly sensor-based [36] or model-
based [37]. Biomechanical fidelity may be improved by using
one of these methods instead of the classical control scheme
proposed in the current study. Finally, many studies have been
done on the effects of virtual visual feedback for object ma-
nipulation and methods of improvement have been proposed
with encouraging results [38], [39].

V. CONCLUSION

Virtual immersion with the use of haptic interface has shown
many applications such as professional training [40] and er-
gonomics [5]. These technologies have moderate performance.
<figure><img src="image_5.png" /><figcaption>Fig. 5. Averaged EMG data (top) and hand speed (bottom) for real task (real visual feedback and haptic interaction) (left) and for virtual task (virtual visual feedback and synthetic haptic interaction) (right) at low speed. Yellow zone: outward phase, green zone: return phase, white zone: rest phase.</figcaption></figure>

The aim of this study was to evaluate individual effects and cross-effects of synthetic haptic interaction and virtual visual feedback on the biomechanical fidelity of simulated pick-and-place tasks performed at different speeds. EMG signals and hand motion were used as objective indicators, and a questionnaire was used to gather subjective results. The different conditions caused significant differences in the variables. In particular, the synthetic haptic interaction exhibited a higher motor muscle activation without increasing co-contraction, the decrease in the length of the trajectory and the decrease in the maximum velocity of the hand. The significant differences were due notably to the non-transparency of HI, which exhibits a higher force feedback than the one calculated by the physics engine because of its mechanical behavior (inertia of its parts, frictions of its joints). Moreover, these effects were particularly important for low masses manipulations. On the other hand, virtual visual feedback resulted in an increase in co-contraction and a modification of the trajectory which would be mainly due to the compression of distance perception.

It is well known that such VR-based systems should be used with caution for ergonomics purposes. Regarding this observation, we developed an original setup, enabling to separate the effect of visual and haptic renderings in the scene. So, an important result of our study is that there was no cross-effect between haptic interaction and immersion enabling to study separately the mode of immersion and interaction for this kind of task, which are abundant in the literature. It opens the door to many dedicated setups to evaluate specific rendering modalities of fidelity for ergonomics purposes. Moreover, the study enabled to develop specific indicators and methodologies able to provide a fair idea of the biomechanical fidelity of a

simulated task from a set of easily accessible quantities. Such an approach should be systematized as a standardization of biomechanical fidelity assessment for ergonomics purposes.

The focus on simple tasks as pick-and-place ones makes the results of the study easily replicable and useful for many industrial tasks. However, limitations of this study leads to the necessity of future works focused on more challenging tasks, involving higher force exertion levels and more complex manipulations with more representative population and numerous cohort. This may require additional developments in haptics rendering control schemes, in order to reach acceptable levels of biomechanical fidelity for such applications.

## REFERENCES

[1] A. G. De Sa and G. Zachmann, "Virtual reality as a tool for verification of assembly and maintenance processes," *Computers & Graphics*, vol. 23, no. 3, pp. 389-403, 1999.

[2] F. Noël and L. Roucoules, "The ppo design model with respect to digital enterprise technologies among product life cycle," *International Journal of Computer Integrated Manufacturing*, vol. 21, no. 2, pp. 139-145, 2008.

[3] J. R. Wilson, "Virtual environments applications and applied ergonomics," *Applied Ergonomics*, vol. 30, no. 1, pp. 3-9, 1999.

[4] S. Aromaa and K. Väätänen, "Suitability of virtual prototypes to support human factors/ergonomics evaluation during the design," *Applied ergonomics*, vol. 56, pp. 11-18, 2016.

[5] C. Pontonnier, G. Dumont, A. Samani, P. Madeleine, and M. Badawi, "Designing and evaluating a workstation in real and virtual environment: toward virtual reality based ergonomic design sessions," *Journal on Multimodal User Interfaces*, vol. 8, no. 2, pp. 199-208, 2014.

[6] H. Nguyen, C. Pontonnier, S. Hilt, T. Duval, and G. Dumont, "Vr-based operating modes and metaphors for collaborative ergonomic design of industrial workstations," *Journal on Multimodal User Interfaces*, vol. 11, no. 1, pp. 97-111, 2017.
[7] U. Jayaram, S. Jayaram, I. Shaikh, Y. Kim, and C. Palmer, "Introducing quantitative analysis methods into virtual environments for real-time and continuous ergonomic evaluations," *Computers in industry*, vol. 57, no. 3, pp. 283-296, 2006.

[8] Y. Wang, K.-b. Liao, Y. Guo, Y.-y. Zhang, W. Zhang, and S. Wu, "Development and application of integrated human machine interaction simulation system," *Journal of system simulation*, vol. 19, no. 11, pp. 2492-2495, 2007.

[9] A. L. Alexander, T. Brunyé, J. Sidman, and S. A. Weil, "From gaming to training: A review of studies on fidelity, immersion, presence, and buy-in and their effects on transfer in pc-based simulations and games," *DARWARS Training Impact Group*, vol. 5, pp. 1-14, 2005.

[10] R. P. McMahan, D. A. Bowman, D. J. Zielinski, and R. B. Brady, "Evaluating display fidelity and interaction fidelity in a virtual reality game," *IEEE transactions on visualization and computer graphics*, vol. 18, no. 4, pp. 626-633, 2012.

[11] C. Pontonnier, "Efficient motion analysis and virtual reality methods for preventive and corrective ergonomics," Ph.D. dissertation, Ecole Normale Supérieure de Rennes, 2019, habilitation à Diriger des Recherches.

[12] B. Hu, L. Ma, W. Zhang, G. Salvendy, D. Chablat, and F. Bennis, "Predicting real-world ergonomic measurements by simulation in a virtual environment," *International Journal of Industrial Ergonomics*, vol. 41, no. 1, pp. 64-71, 2011.

[13] B. Hu, W. Zhang, and G. Salvendy, "Impact of multimodal feedback on simulated ergonomic measurements in a virtual environment: A case study with manufacturing workers," *Human Factors and Ergonomics in Manufacturing & Service Industries*, vol. 22, no. 2, pp. 145-155, 2012.

[14] C. Pontonnier, A. Samani, M. Badawi, P. Madeleine, and G. Dumont, "Assessing the ability of a vr-based assembly task simulation to evaluate physical risk factors," *IEEE transactions on visualization and computer graphics*, vol. 20, no. 5, pp. 664-674, 2013.

[15] M. A. Rizzuto, M. W. Sonne, N. Vignais, and P. J. Keir, "Evaluation of a virtual reality head mounted display as a tool for posture assessment in digital human modelling software," *Applied ergonomics*, vol. 79, pp. 1-8, 2019.

[16] A. Samani, C. Pontonnier, G. Dumont, and P. Madeleine, "Shoulder kinematics and spatial pattern of trapezius electromyographic activity in real and virtual environments," *PloS one*, vol. 10, no. 3, 2015.

[17] M. Sagardia and T. Hulin, "Multimodal evaluation of the differences between real and virtual assemblies," *IEEE transactions on haptics*, vol. 11, no. 1, pp. 107-118, 2017.

[18] T. Hulin, K. Hertkorn, P. Kremer, S. Schätzle, J. Artigas, M. Sagardia, F. Zacharias, and C. Preusche, "The dlr bimanual haptic device with optimized workspace," in *2011 IEEE International Conference on Robotics and Automation*. IEEE, 2011, pp. 3441-3442.

[19] S. Hilt, C. Pontonnier, and G. Dumont, "Model based compensation for low mass objects haptic manipulation in virtual environments," in *International Conference on Virtual Reality and Augmented Reality*. Springer, 2017, pp. 87-101.

[20] D. C. Niehorster, L. Li, and M. Lappe, "The accuracy and precision of position and orientation tracking in the htc vive virtual reality system for scientific research," *i-Perception*, vol. 8, no. 3, p. 2041669517708205, 2017.

[21] B. G. Witmer and M. J. Singer, "Measuring presence in virtual environments: A presence questionnaire," *Presence*, vol. 7, no. 3, pp. 225-240, 1998.

[22] R. Likert, "A technique for the measurement of attitudes." *Archives of psychology*, 1932.

[23] P. Konrad, "The ABC of EMG," 2005.

[24] M. L. Latash, J. P. Scholz, and G. Schöner, "Motor control strategies revealed in the structure of motor variability," *Exercise and sport sciences reviews*, vol. 30, no. 1, pp. 26-31, 2002.

[25] A. Samani, C. Pontonnier, G. Dumont, and P. Madeleine, "Kinematic synergy in a real and a virtual simulated assembly task," in *Proceedings 19th Triennial Congress of the IEA*, vol. 9, 2015, p. 14.

[26] M. Suzuki, D. M. Shiller, P. L. Gribble, and D. J. Ostry, "Relationship between cocontraction, movement kinematics and phasic muscle activity in single-joint arm movement," *Experimental brain research*, vol. 140, no. 2, pp. 171-181, 2001.

[27] B. Laursen, B. R. Jensen, and G. Sjøgaard, "Effect of speed and precision demands on human shoulder muscle electromyography during a repetitive task," *European Journal of Applied Physiology and Occupational Physiology*, vol. 78, no. 6, pp. 544-548, 1998.

[28] C. E. Wright and D. E. Meyer, "Conditions for a linear speed-accuracy trade-off in aimed movements," *The Quarterly Journal of Experimental Psychology*, vol. 35, no. 2, pp. 279-296, 1983.

[29] J. M. Loomis, J. A. Da Silva, N. Fujita, and S. S. Fukusima, "Visual space perception and visually directed action." *Journal of Experimental Psychology: Human Perception and Performance*, vol. 18, no. 4, p. 906, 1992.

[30] T. Y. Grechkin, T. D. Nguyen, J. M. Plumert, J. F. Cremer, and J. K. Kearney, "How does presentation method and measurement protocol affect distance estimation in real and virtual environments?" *ACM Transactions on Applied Perception (TAP)*, vol. 7, no. 4, pp. 1-18, 2010.

[31] V. Interrante, B. Ries, and L. Anderson, "Distance perception in immersive virtual environments, revisited," in *IEEE virtual reality conference (VR 2006)*. IEEE, 2006, pp. 3-10.

[32] J. P. Rolland, F. A. Biocca, T. Barlow, and A. Kancherla, "Quantification of adaptation to virtual-eye location in see-thru head-mounted displays," in *Proceedings Virtual Reality Annual International Symposium'95*. IEEE, 1995, pp. 56-66.

[33] T. Flash and N. Hogan, "The coordination of arm movements: an experimentally confirmed mathematical model," *Journal of neuroscience*, vol. 5, no. 7, pp. 1688-1703, 1985.

[34] M. A. Otaduy and M. C. Lin, "A modular haptic rendering algorithm for stable and transparent 6-dof manipulation," *IEEE Transactions on Robotics*, vol. 22, no. 4, pp. 751-762, 2006.

[35] R. Fesharakifard, L. Leroy, and P. Fuchs, "Development of a string-based haptic interface by using a hybrid control approach," in *Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC'07)*, 03 2007, pp. 488-493.

[36] O. Baser, E. I. Konukseven, and H. Gurocak, "Transparency improvement in haptic devices with a torque compensator using motor current," in *International Conference on Human Haptic Sensing and Touch Enabled Computer Applications*. Springer, 2012, pp. 37-46.

[37] N. L. Bernstein, D. A. Lawrence, and L. Y. Pao, "Friction modeling and compensation for haptic interfaces," in *First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference*. IEEE, 2005, pp. 290-295.

[38] D. J. Finnegan, E. O'Neill, and M. J. Proulx, "Compensating for distance compression in audiovisual virtual environments using incongruence," in *Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems*, 2016, pp. 200-212.

[39] ——, "An approach to reducing distance compression in audiovisual virtual environments," in *2017 IEEE 3rd VR Workshop on Sonic Interactions for Virtual Environments (SIVE)*. IEEE, 2017, pp. 1-6.

[40] T. R. Coles, D. Meglan, and N. W. John, "The role of haptics in medical training simulators: a survey of the state of the art," *IEEE Transactions on haptics*, vol. 4, no. 1, pp. 51-66, 2010.