# Adding Vibrotactile Feedback to Large Interactive Surfaces

Julian Seifert, Markus Packeiser, Enrico Rukzio

<figure><img src="image_1.png" /></figure>

## To cite this version:

Julian Seifert, Markus Packeiser, Enrico Rukzio. Adding Vibrotactile Feedback to Large Interactive Surfaces. 14th International Conference on Human-Computer Interaction (INTERACT), Sep 2013, Cape Town, South Africa. pp.507-514, 10.1007/978-3-642-40498-6_39. hal-01510549

HAL Id: hal-01510549

https://inria.hal.science/hal-01510549

Submitted on 19 Apr 2017

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Adding Vibrotactile Feedback to Large Interactive Surfaces

Julian Seifert¹, Markus Packeiser², Enrico Rukzio¹

¹Ulm University, Ulm, Germany

{julian.seifert, enrico.rukzio}@uni-ulm.de

²University of Duisburg-Essen, Essen, Germany

markus.packeiser@uni-due.de

**Abstract.** Interactive surfaces and multi-touch tables are increasingly available outside academic contexts, and are entering, for instance, work or educational contexts. A large variety of applications exists for a multitude of tasks. For interacting with these applications, existing interaction concepts are often directly mapped to the multi-touch surface, which is often limited by physical constraints. For instance, to enter text on an interactive surface, most often a virtual keyboard is used. However, users cannot feel when, for instance, they have accidentally pressed two keys at the same time. Research on mobile devices has identified vibrotactile feedback as an effective means to support users when interacting with touch screens. In this work, we present results of an experiment in which we investigated whether typical tasks (e.g., typing text, drag-and-drop of items) on interactive multi-touch surfaces can be supported by providing vibrotactile feedback directly on the surface*. We compared direct feedback with distal feedback provided on the user's body, as well as their combination. Surprisingly, our results show that all compared variants of vibrotactile feedback had no significant positive effect on the task performance. Yet participants rated tactile feedback significantly higher regarding interaction support and subjective speed compared to no provided feedback.

**Keywords.** Interactive surfaces, vibrotactile feedback, touch-based interaction, multi touch.

## 1 Introduction

Recently, large interactive surfaces have begun to come out of the labs and are entering diverse usage contexts such as domestic, or workplace environments. Reasons are maturing technologies and progress in interaction research, which enable an increasing number of use cases wherein direct multi-touch on interactive surfaces is the main interaction style. Touch-based interaction is commonly considered to be natural [6] as users can interact with virtual objects as they expect it based on their experiences with real physical objects [7]. However, depending on the task, the absence of correspond-

<footnote>* A demo video is available at http://youtu.be/0DzTtBTeglQ</footnote>
ing tactile feedback of virtual objects decreases user experience and performance (which is in particular a challenge in the field of virtual reality [2]).

This challenge of missing corresponding tactile feedback applies also to touch-based interaction with hand-held mobile devices such as smartphones. One solution that has been identified to significantly increase the typing performance on virtual keyboards is vibrotactile feedback [3]. That is, users benefit from short vibrotactile sensations when touching a button, indicating that an effect has resulted from the input.

Evidently, vibrotactile feedback is a promising candidate for increasing performance and accuracy of touch-based interactions on large interactive multi-touch surfaces. However, it is not obvious how differences to hand-held devices (e.g., multi-touch interactions, screen size, task-type) influence the positive effect of vibrotactile feedback.

In this work, we contribute findings of a user study investigating the effect of vibrotactile feedback during interaction with interactive surfaces enabling multi-touch. The study compared direct vibrotactile feedback on the surface with distal feedback on the user's wrist [11], as well as the combination of both. As a base-line, no vibrotactile feedback was given. Surprisingly, our findings indicate that vibrotactile feedback provided directly on a surface does not increase the users' performance in a typing task as observed on hand-held devices. In fact, none of the tested feedback conditions results in a significant improvement compared to conditions wherein no feedback was provided. However, participants expressed that vibrotactile feedback (in any form) is helpful and preferred over no feedback.

In the remainder of this paper, we discuss related works, introduce our experimental setup, and discuss the results and observations gained during the experiment.

## 2 Related Work

Early work on vibrotactile feedback was done in the field of virtual reality (VR). In this context, the absence of tactile feedback while interacting with virtual objects was identified as a challenge and different approaches for providing force feedback (e.g., [4]) and tactile feedback [5] have been introduced.

For interaction with interactive surfaces such as touch screens, the challenge is similar: users can touch objects displayed on the screen. However, it is always the same sensory experience.

Similar as in VR, additional input devices can be used which provide vibrotactile feedback when necessary. For instance, the *HapticPen* enables one or multiple users to interact with a touch screen and experience individual feedback [8]. Marquardt et al. introduced the *Haptic Tabletop Puck*, a mouse-like device providing feedback which the user moves on the tabletop to experience different surface qualities [10]. McAdam and Brewster investigated the effect of distal feedback on typing tasks using a small touch screen [11]. Conceptually, distal feedback is provided by the users' personal mobile phone and its built-in vibration alarm. In their study, they used external vibration actuators placed on the user's body. Their study results indicate that
distal feedback improves users' typing performance compared to when no feedback was provided.

Among the approaches that include an external device which provides the vibrotactile feedback, all have in common that the interaction with objects is less or not at all natural because direct touching is not possible. Hence, approaches that provide tactile feedback directly on the surface are capable of maintaining the natural interaction. Brewster et al. investigated the effect of vibrotactile feedback provided directly on the touch screen of mobile phones [3]. They found that such feedback has a significant positive effect on users' performance.

Another approach for enriching the tactile experience on interactive surfaces was presented by Bau et al. with *TeslaTouch* [1]. TeslaTouch is based on electro-vibration and hence, no physical movement of the surface is required to create the feedback. However, the electrovibration changes the force of attraction between the touching finger and the surface, which results in a specific perceived friction when moving the finger. Therefore, the feedback can only be provided for dragging interactions.

In this work, we examine the approach of providing tactile feedback through attaching vibration actuators at the bottom side of a large interactive surface which allows us to consider not only the dragging of objects but also tasks such as typing, in which the fingers of the user only touch the surface to trigger a button without dragging. Further, the size of the interactive surface is of relevance when considering multi-touch interactions.

## 3 User Study

We designed and conducted an experiment1 in order to investigate how vibrotactile feedback effects interaction with an interactive surface in comparison to settings with distal feedback and without any feedback. In the following, we introduce the design, tasks, and apparatus of the user study.

*Design.* We applied a within-subject design. The independent variable was the feedback: direct feedback on the interactive surface, distal feedback provided on the user's body, and combined direct and distal feedback. Finally, as baseline, no feedback was provided.

In counterbalanced order, participants used each of the four feedback variants to complete a series of tasks, which were also presented in counterbalanced order. While interacting with the interactive surface, participants were standing and allowed to move around the table. After each round with a feedback variant, participants filled in a questionnaire rating the previously tested feedback variant.

*Tasks.* Similar to previous work, we argue that typing is not the primary and only activity that is likely to be performed with an interactive surface. Hence, we included two additional tasks to test the effect of each feedback variant. These required drag-and-drop actions, the use of multiple fingers at the same time, and a steering task.
<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Graphical user interfaces of the task applications: (a) a typing task, (b) a puzzle, and (c) a steering task.</figcaption></figure>

1. *Typing task.* In the typing task, participants were asked to type six English sentences during each feedback condition. The sentences were taken from a standard set designed for typing performance testing [9]. The sentences are of equal length and complexity and do not contain special characters or punctuation marks. The user interface of the typing task application shows three items: the given sentence to type, a text field, and a virtual QWERTY keyboard (see Fig. 1a). Users can move the virtual keyboard to a preferred position in which they find it comfortable to type. Vibrotactile feedback (if available) was provided when hitting a character key on the virtual keyboard.

The collected data consisted of the task completion time, the number of corrections, and the number of errors.

2. *Puzzle.* In the puzzle task, participants were given a number of shapes on the interactive surface which had to be matched with predefined positions which were indicated by gray placeholders (see Fig. 1b). Vibrotactile feedback (if available) was provided when the user managed to place an item on its target (considering location and rotation within a specific tolerance). The puzzle was solved when all items were positioned correctly.

The collected data were the task completion time and the number of times an item was released outside the accepted tolerance.

3. *Steering task.* In the steering task, participants were asked to steer a rectangle through a given path without touching the borders of the path (see Fig. 1c). When they reached the end of the path, participants had to follow the path back to the starting point. Vibrotactile feedback (if available) was provided when the distance between the path border and any corner point of the rectangle was less than a specific threshold.

The collected data encompassed the task completion time and the number of times the rectangle shape accidentally hit the border of the path.

*Apparatus.* The apparatus system for the experiment consists of an interactive surface which is augmented with four vibration actuators (see Fig. 2a). The interactive surface is based on frustrated total internal reflection (FTIR) for sensing multiple touch events simultaneously. The surface display is based on a rear projection (1280×1024 px), controlled by a computer which runs surface applications.
To generate the vibrotactile feedback that can be perceived when touching the sur-
face of the multi-touch table, four 50 Watt low-frequency speakers (20-80 Hz) are
mounted inside the interactive surface. They are powered by two 100 Watt amplifiers
that are connected to the surface computer. In order to provide vibrotactile feedback,
the running surface application plays an audio file which specifies the vibration pat-
tern (a square wave was empirically found to produce a strong tactile sensation on the
surface). Similar to the direct feedback, the distal feedback is generated using a vibra-
tion actuator which was attached to the participant's wrist (see Fig. 2c).

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** The apparatus systems: (a) An interactive surface using rear projection. (b) Four ‘bass pumps’ mounted inside the interactive surface in order to generate low-frequency vibrations. (c) C2 vibration actuator to provide distal feedback.</figcaption></figure>

*Participants.* Through an email list, we recruited 18 participants (10 males and 8 fe-
males) aged between 21 and 31 years. Participants were randomly selected. All but
one participant (a teacher) were students with diverse backgrounds (technical and
humanities). Participants received 10 Euro after the study as a reward.

*Hypotheses.* Overall, we hypothesized that vibrotactile feedback has a positive effect on the interaction. In particular:

H0: Direct vibrotactile feedback provided by the surface leads to better (faster task completion and less errors) performance during interaction (typing, puzzle, and steering) compared to no feedback and to distal feedback.

H1: Combined distal and direct feedback outperforms their separate effect (faster interaction and less errors).

H2: Direct feedback is subjectively preferred over distal and combined feedback by users.

4 Results

We hypothesized that vibrotactile feedback has a positive effect on the task comple-
tion time (H0). To our surprise, the mean task completion times for the three tasks are
very close to each other. As figure 3 shows, in the typing task, the average completion
time for the condition ‘no feedback’ is higher (M=72.7s) com- pared to distal
(M=68.4s), direct (M=66.6s), and combined feedback (M=64.8s). Yet testing with a
one-way repeated-measures ANOVA shows that differences are not significantly
different (F(3; 51)=1.76; p=.17). Moreover, for the puzzle and the steering task, the
direct feedback resulted in the longest completion times. This raises the question why
direct vibrotactile feedback has a significant positive effect on typing performance when using a mobile device (see [3]), however, not when applied on a large surface.

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Task completion times for the tasks ‘typing’, ‘puzzle’, and ‘steering’ (Error bars show the standard deviation).</figcaption></figure>

Further, we compared the number of fixes (how many times the backspace key was pressed). Overall, there is a significant effect ($F(1.77; 29.99)=6.77; p=.005$): Direct feedback (M=4.22, SD=4.70), distal feedback (M=4.28, SD=3.14), and combined feedback (M=4.72, SD=4.01) equally often required the user to correct typing errors, followed by no feedback (M=10.05, SD=8.92). A pairwise comparison shows that only direct feedback requires significantly less fixes compared to no feedback ($p=.04$).

The minimum string distance (i.e., Levenshtein distance) which gives an estimation of the number of errors made, did not show significant differences between the four feedback types ($F(3; 51)=.38; p>.05$).

During the task of solving the puzzle, participants performed numerous drag-and-drop actions. This type of interaction differs from typing as the user's fingers are in contact with the surface for longer times. Hence, vibrotactile feedback should help to place the puzzle pieces in place correctly. However, looking at the average number of falsely placed pieces, it shows that combined feedback resulted in the highest number of errors (M=5.9; SD=3.3), followed by distal (M=4.8; SD=3.7), no feedback (M=4.3; SD=1.7), and direct feedback (M=4.1; SD=1.7), yet the differences are not significant ($F(3; 51)=.81; p>.05$).

Comparing the amount of errors made during the steering task shows no significant differences ($F(3, 51)=2.46; p>.05$). However, the numbers show a tendency indicating that feedback is helping: no feedback (M=4.89, SD=4.32) compared to direct (M=2.72, SD=2.73), distal (M=2.61, SD=2.35), and combined feedback (M=2.83, SD=3.22).

These results contradict to our hypothesis H0, which consequently has to be rejected. Further, the combination of direct and distal feedback does not result in better results. Consequently, H1 has to be rejected as well.
<figure><img src="image_6.png" /><figcaption>**Fig. 4.** Subjective ratings of the four feedback conditions (Bars show medians; error bars show standard deviation).</figcaption></figure>

After each task, participants rated the feedback type via different statements (see Fig. 4) on a 5-point scale (1=fully disagree). Regarding the statement S1 (‘The feedback irritated me.’), participants did not rate the feedback types significantly different ($\chi^2(18)=5.8; p>$.05). For all other statements, significant differences were reported, as illustrated in Fig. 4. However, interestingly the only significant differences are those between conditions in which feedback was given and the baseline condition in which no feedback was provided, as shown by pairwise comparison. For instance, in statement S3 (‘The feedback allowed me to quickly finish the task.’), all three conditions in which vibrotactile feedback was provided were rated the same on average (Mdn=4.0). In statements S4 and S5, distal feedback was rated lower compared to direct and combined feedback. As these differences are not significant, it might at least indicate that the presence of direct feedback is appreciated by users. These subjective ratings indicate that from the users’ point of view, it only matters that feedback is provided at all – not how this is implemented. Accordingly, hypothesis H2 cannot be confirmed.

# 5 Discussion & Conclusion

In this work, we investigated the effect of vibrotactile feedback on interaction with large interactive surfaces. Feedback provided directly on the surface, distal, or combined, did not have an effect on the users' performance. In fact, this result is surprising as previous work showed that vibrotactile feedback significantly improves speed and accuracy.

One obvious explanation for the small objectively measured effect is that performing touch and drag-and-drop interactions on an interactive surface already causes strong tactile sensations. Hence, a vibrotactile feedback has to be even stronger compared to this resulting sensation at the users' fingertips. In the case of mobile handheld devices, we hypothesize that the hand holding the device is mainly responsible for feeling the vibrotactile feedback, rather than the finger used for input.

In addition, the results of this experiment depend strongly on the used FTIR technology, which requires users to perform stronger touches compared to a capacitive
touch screen. Hence, future work should investigate if and how vibrotactile feedback is beneficial on capacitive surfaces such as the Samsung SUR40. The data measured during the experiment, such as task completion time different error values, did not support our initial hypothesis that direct feedback has a positive effect on the performance. However, users reported that any given type of vibrotactile feedback is valuable to them as they felt that they performed tasks faster.

As the tested feedback types (direct, distal, and combined) do not cause considerable differences in terms of speed (task completion) and accuracy (errors), the decision regarding which type of feedback to offer depends mainly on the given application design. For instance, in multi-user games it might be interesting to provide a *common* feedback for all players directly on the surface. At the same time, private feedback could be provided via distal means.

## Acknowledgement

This work has been conducted in the Emmy Noether research group Mobile Interaction with Pervasive User Interfaces funded by the German Research Foundation.

## 6 References

1. O. Bau, I. Poupyrev, A. Israr, and C. Harrison. Teslatouch: electrovibration for touch surfaces. In UIST '10, pages 283–292. ACM, 2010.

2. M. Benali-khoudja, M. Hafez, J. marc Alex, and A. Kheddar. Tactile interfaces: a state-of-the-art survey. In Int. Symposium on Robotics, pages 721–726, 2004.

3. S. Brewster, F. Chohan, and L. Brown. Tactile feedback for mobile interactions. In CHI '07, pages 159–162. ACM, 2007.

4. F. P. Brooks, Jr., M. Ouh-Young, J. J. Batter, and P. Jerome Kilpatrick. Project GROPEHaptic displays for scientific visualization. In SIGGRAPH '90, pages 177–185. ACM, 1990.

5. CyberGlove Systems LLC. Overview. http://www.cyberglovesystems.com/ products/cybertouch/overview, 2013. (Last access: 2013/03/15).

6. P. Dietz and D. Leigh. Diamondtouch: a multi-user touch technology. In UIST '01, pages 219–226. ACM, 2001.

7. H. Ishii and B. Ullmer. Tangible bits: towards seamless interfaces between people, bits and atoms. In CHI '97, pages 234–241. ACM, 1997.

8. J. C. Lee, P. H. Dietz, D. Leigh, W. S. Yerazunis, and S. E. Hudson. Haptic pen: a tactile feedback stylus for touch screens. In UIST '04, pages 291–294. ACM, 2004.

9. I. S. MacKenzie and R. W. Soukoreff. Phrase sets for evaluating text entry techniques. In CHI EA '03, pages 754–755. ACM, 2003.

10. N. Marquardt, M. A. Nacenta, J. E. Young, S. Carpendale, S. Greenberg, and E. Sharlin. The haptic tabletop puck: tactile feedback for interactive tabletops. In ITS '09, pages 85–92. ACM, 2009.

11. C. McAdam and S. Brewster. Distal tactile feedback for text entry on tabletop computers. In BCS-HCI '09, pages 504–511, Swinton, UK, UK, 2009. British Computer Society.