# Designing of 2D Illusory Tactile Feedback for Hand-Held Tablets

Youngsun Kim, Jaedong Lee, Gerard J. Kim

<figure><img src="image_1.png" /></figure>

► To cite this version:

Youngsun Kim, Jaedong Lee, Gerard J. Kim. Designing of 2D Illusory Tactile Feedback for Hand-Held Tablets. 15th Human-Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.10-17, 10.1007/978-3-319-22723-8_2 . hal-01610806

HAL Id: hal-01610806

https://inria.hal.science/hal-01610806

Submitted on 5 Oct 2017

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
# Designing of 2D Illusory Tactile Feedback for Hand-held Tablets

Youngsun Kim, Jaedong Lee, Gerard J. Kim<fnref n="1" />

Digital Experience Laboratory
Korea University, Seoul, Korea

{zyoko85, jdlee, gjkim}@korea.ac.kr

**Abstract.** In this paper, we investigate whether the “out of body” tactile illusion can be extended or applied to a relatively large hand-held device such as a tablet for which the hands/fingers would not be in direct contact with the vibration motors. We derived guidelines for applying tactile illusion techniques in 2D space with regards to operational conditions such as the size of the device, holding position, minimally required vibration amplitudes, and the effects of matching visual feedback. For this purpose, a series of exploratory pilot experiments were first conducted in 1D space. Based on the results, a 2D illusory tactile rendering method was devised and tested for its effectiveness. We have found that for a tablet sized device (e.g. iPad mini and iPad), the illusory perception was possible with a rectilinear grid resolution of 5×7 (with a grid size of 2.5 cm) with matching visual feedback.

**Keywords:** Funneling, Illusory feedback, Vibro-tactile feedback, User experience, Mobile/Hand-held interaction.

## 1 Introduction

Smart hand-held devices nowadays come in many different sizes: from smart phones (10 – 15 cm in diagonal length) to larger tablet devices and hand-held computers (18 – 25 cm). There have been increasing interests in improving the user experience on hand-held devices by providing richer and more dynamic feedback through vibro-tactile actuators [6,11,12]. To avoid resorting to vibro-tactile arrays which would require a sizable and continuous contact area with the hand or body parts, researchers have proposed to apply tactile illusion techniques so that similar effects can be realized with only a few number of vibration motors [6, 11]. In addition, Miyazaki has demonstrated that tactile illusory feedback could be elicited at the “out of the body” space [8]. Kim has applied the “out of the body” tactile illusory feedback for a small-sized smartphone (9.4 cm in diagonal length), and demonstrated the illusory feedback was indeed applicable and extendable for interaction in 2D screen space using only four vibrators [6].

<footnote n="1">¹ Corresponding author</footnote>
In this paper, we investigate whether the “out of body” tactile illusion can be ex-
tended or applied also to larger hand-held devices such as tablet devices. The exten-
sion is not expected to be straightforward for few reasons. First the original “out of
body” tactile illusion was only validated when the body parts (e.g. fingertips) were
stimulated directly and set apart by 7–8 cm span [6], similar to when the phenomenon
was originally discovered [8]. With tablet devices, the operating conditions are quite
different (see Figures 1(c) and (d)). For one, the vibratory stimulations are indirect,
and the hands/fingers may not be in equidistance from the actuators. No relevant
study has been reported for when the size of the hand-held object goes larger than the
“8cm” span, not to mention for the 2D extension.

<figure><img src="image_3.png" /><figcaption>**Fig. 1.** The concept of “out of body” tactile experience from a hand-held object. (a) When vibra-tory stimulations are given to the fingertips properly, phantom tactile sensations can be felt as if occurring from the middle of the hand-held object [7]. The 1D tactile illusion is extended to 2D for (b) a small smart phone where fingers are in direct contact with the vibration motors [6], (c) a small tablet (e.g. *iPad mini*), and (d) a larger tablet (e.g. *iPad*).</figcaption></figure>

Thus, to find out how to apply the “out of body” tactile illusion to tablet devices,
we ran a series of pilot exploratory experiments to first observe the effects of the tac-
tile illusion e.g. with regards to various operational conditions such as the size of the
device, holding position, minimal vibration amplitudes and effects of accompanying
visual feedback for a fair perception of the illusion. The findings and observations
would form the basis for designing a proper 2D tactile illusory rendering method for
the tablet device. We shortly describe these experiments, report the newly derived
guidelines, propose for a 2D tactile illusory rendering method for a tablet, and empiri-
cally validate its effectiveness.

## 2 Related Work

Funneling is one of the major perceptual illusion techniques specifically for vibro-
tactile feedback (along with saltation which we do not consider in this work for now).
It refers to stimulating the skin at two different locations simultaneously with differ-
ent amplitudes and eliciting phantom sensations in between the two (i.e. 1D phenom-
enon) [1, 3]. The intended location of the phantom sensation (ITL) can be changed by
modulating the stimulation intensity.

Several researchers have applied this phenomenon to human interfaces, experi-
menting with different ways of modulating the vibration amplitudes for detailed con-
trolling of the target phantom sensation locations [1, 2, 5, 9, 10]. For example, the
SemFeel was a vibro-tactile feedback system for a mobile touch screen device [11] using five vibrators attached to the backside of the device (4 corners and 1 center). It was capable of producing 10 different perceptible vibration patterns for expressing richer semantic information compared to the usual single vibrator scheme.

Recently, researchers have discovered such phantom sensations can be extended to the “out of body” [8] (see Figure 1(a)), thus making it possible to generate phantom tactile sensations as if coming from an external object. This result is appealing in its application possibility to mobile devices. That is, there now is a potential method to make a user feel (phantom) sensations indirectly emanating from the hand-held device (e.g. middle of the display), but supplying actual vibrations only to the “natural” holding finger/hand locations (Figure 1). Kim demonstrated that the aforementioned phenomenon could be extended to 2D by employing vibrators both horizontally and vertically on a small sized (3.5 inch screen) smart phone [6]. But no general guidelines have been reported for further applications to a variety of today’s hand-held media devices.

Finally, many human computer interfaces also rely on cross modal illusions to enrich user experience by fusing feedbacks of different modalities. For instance, a mere single vibrator based tactile feedback (as adopted by most current smart phones), when combined with other modalities, can induce phantom directional haptic feedback and rich user experience [12]. We posit that the tactile illusion feedback, when combined with other usual modalities such as the visual and aural, can similarly improve the interaction experience even further.

Thus, to summarize, the distinguishing point of our work is that we seek to newly extend funneling to an enlarged 2D space, and identify important design guidelines. This would open doors for a more flexible design of vibro-tactile feedback techniques for a wide variety of hand-held devices.

# 3 Pilot Experiments: “Out of body” Funneling Effects along Longer 1D Separation Span

Due to lack of space, we only provide a very short description of how the exploratory pilot experiments were run. Funneling stimulations were given through vibrators attached to an *iPad mini* and *iPad*, along three horizontal (along the top/middle/bottom row) or vertical directions (along the left/middle/right column) using two vibrators attached at the respective ends (i.e. six vibrators used for each case). Sixteen participants were asked to report the existence of any illusory tactile perception and indicate where they came from, if any (along the respective row or column). Various rendering parameters were changed and results observed.

## 3.1 Perception resolution and extent

Our data showed the smallest perceptible linear interval size to be comparable to that of the smart phone case (~1.6 cm) [6], “regardless of the size of the device (or distance between the vibration motors),” but only so in the middle part (40–60% of the
bands). The perception resolution was notably lower (or discernable grid size "stretched", e.g. to as long as 7 cm depending on where the device was held) near the horizontal or vertical ends (see Figure 2). This was more evident in the vertical direction, because the hands were holding the device along the horizontal direction.

It seems the increased size of the devices and the fact that the hands/fingers are not in direct contact with the vibration motors somehow affect the “out of body” illusory perception near the holding positions. Based on this finding, we conjectured that for 2D situation, the perception resolution pattern and extent would look like what are depicted in Figure 2 (dotted lines).

<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Perceptible grid sizes in horizontal and vertical directions (a) smart phone: ~1.6cm [6] (b) iPad mini: 2–3.7 cm, (c) iPad (held in the middle): 1.5–4.5 cm, (d) iPad (held in the lower part): 1.3–7 cm. The grids and the dotted area on the iPad mini and iPad show the “projected” effective perception area in 2D. Shaded regions represent the “stretched” grids.</figcaption></figure>

## 3.2 Tactile rendering function and interpolated stimulation strength

In the pilot experiment described in Section 3.1, we initially tested the three most popular and arguably reliable (in terms of eliciting the phantom sensation from the intended location) stimulation rendering methods for funneling, namely, interpolating the two vibration amplitudes using linear, logarithmic, and tangent functions [1]. Among the three, the tangent based interpolation showed the widest perception band (e.g. 6 intervals vertically and 9 horizontally on an iPad as shown in Figure 2). However, in the later main experiment, this finding did not carry over to the 2D case. The responses were inaccurate and irregular. This was due to the fact that when tangent-based interpolation was applied in 2D, the interpolated amplitudes at the middle region became relatively too low (compare the two interpolating functions in Figure 3). Thus, the logarithmic interpolation was chosen to render the tactile vibration amplitude with slightly less effective perception resolution (5×7) but with higher accuracy and response regularity.

## 3.3 Hand location

Finally, the finger/hand grabbing positions were varied between the middle and lower part on the sides of the tablets. When the users held the lower part of the tablets, the middle effective perception region also shifted in the lower direction and toward the user (see Figure 2). In the main experiment, we asked the participants to hold the middle part of the device.
<figure><img src="image_5.png" /><figcaption>**Fig. 3.** The logarithmic (dotted) and tangent based (full) stimulation interpolation functions (left) and the user's responses (right). The tangent-based interpolation stimulation produces a near-linear user response (black).</figcaption></figure>

# 4 Main Experiment: Extending “Out of Body” Funneling to Tablets

## 4.1 Experimental Design and Set up

In the main experiment, we used four single vibrators attached in the four rear corners of the *iPad* (see Figure 1). We used the same rendering algorithm as in Kim's study for the smart phone [6] but with the few parameters adjusted according to our pilot experiment findings described in the previous section. The main purpose of the experiment was (1) to assess the illusory tactile perception behavior and effectiveness of the newly adjusted rendering algorithm with the larger tablet device and (2) how combining the illusory tactile feedback with the visual would further improve the accuracy and user experience. Two cases of 2D illusory tactile renderings were compared: without any visual feedback (NV), and with visual feedback (VF).

For NV, users were given 2D illusory tactile stimulations on an *iPad* held in the middle part (e.g. as shown in Figure 1(d)) at the resolution of 5×7 (conservatively chosen based on the results of the pilot experiment). Participants were asked to pinpoint (using a stylus pen) where they felt the tactile feedback to be coming from.

For VF, when the stimulation was given, 5 visual icons (matching feedback), only one of them corresponding to the true intended target location (ITL), were shown and the user had to choose the correct one.

Vibrators were controlled by a voltage input using a PWM signal with an amplitude between 0 to 5V, which in turn produced vibrations with frequency between 0 and 250 Hz and associated amplitudes between 0 to 2G (measured in acceleration, or 0 to 18µm in position) respectively.

## 4.2 Detailed Procedure

Seven paid participants (4 men and 3 women) participated in the experiment with the mean age of 24.7. After collecting one's basic background information, the partici-
pant was briefed about the purpose of the experiment and instructions for the experi-
mental task. A short training was given for the participant to get familiarized to the
experimental process. The participants wore ear muffs to prevent any bias from the
sounds of the vibration. For both NV and VF conditions, the funneling stimulation
was given by activating 4 vibrators simultaneously with logarithmically interpolated
amplitudes according to the intended target location of phantom sensation (Figure 4).

<figure><img src="image_6.png" /><figcaption>**Fig. 4.** Funneling stimulations for the 5×7 grid on an *iPad*. Amplitudes of four vibrators in the corners are logarithmically interpolated to produce an illusory sensation in the 2D screen space.</figcaption></figure>

5 Results

**5.1 NV Experiment**

ANOVA was applied for analyzing the perception resolution illustrated in the hori-
zontal and vertical directions separately for simplicity. Figure 5 shows the cases for
the first row and column where 5 and 3 distinct intervals were perceived respectively.
Other rows and columns showed similar statistics. This lead us to project that the 2D
perception resolution to be about only 3×5, lower than the expected (Figure 2). The
graphs in Figure 5 show a similar pattern of lowered resolution at the periphery of the
screen, i.e. elongated/enlarged grid size (e.g. ITLs at 1, 2 and 6, 7 in Figure 5(a) and
1, 2 and 4, 5 in 5(b)). Also note that the user response was not linear as desired, an
artifact of resorting back to using the logarithmic interpolation and a possible interac-
tion among the stimulations in 2D. Thus, we believe that better and linear responses
can be obtained by employing vibration motors with larger amplitude capabilities and
a “mixed” interpolation technique (e.g. logarithmic + tangent-based). Although the
main purpose of NV experiment was observing user behavior for 2D response in 2D
tactile feedback, correct responses for each ITL grid were recorded (Figure 6).

**5.2 VF Experiment**

As for the case of VF, Figure 6 shows the percentages of correct response for each
ITL grid with an average of 78.6% (right part of Figure 6). Note that with the user
having to select from five possible responses, the pure chance performance was only
20%. Thus the average of near 80% is considered a psychophysically meaningful response [4, 7]. While the performance is relatively lower in the middle upper and middle lower region (denoted darker), we claim that with the accompanying matching visual feedback, the non-regular and distorted perception pattern was much rectified and resolution improved as well (compare the left and right parts of Figure 6). In both NV and VF treatments, participants were surveyed about the localizability of the feedback, its clarity, task difficulty, and one's confidences in their responses on a 7-Likert scale. In all cases, the VF condition exhibited much more positive responses.

<figure><img src="image_7.png" /><figcaption>**Fig. 5.** Average percentage of correct responses in the horizontal (left) and vertical directions (right). The graphs show the same pattern of lowered resolution at the periphery of the screen (horizontal ITLs at 1, 2 and 6, 7 and vertical ITLs at 1, 2 and 4, 5).</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>**Fig. 6.** Overall performance of the NV (left) and VF (right) experiments. The percentages of the correct response are indicated in the grid.</figcaption></figure>

# 6 Conclusion and Future Work

In this paper, we carried out pilot experiments to explore how to extend 2D illusory tactile rendering to tablet devices. Tablet devices exceed in length for which “out of body” tactile illusion is known to work, and the stimulations become more indirect and their interactions in 2D more complex. Based on the experimental findings, we have devised a 2D illusory tactile rendering algorithm and have found that for an *iPad* sized device, approximately 5×7 regular resolution (grid size = ~2.5 cm) of illusory perception was achievable, when a matching visual feedback was provided.
For future work, we will apply our findings to real world examples such as inter-active videos and mobile games, and assess whether the UX can be improved by the proposed vibro-tactile rendering method. At the same time, we plan to investigate additional ways to improve the 2D sensation localization accuracy by adjusting vari-ous rendering parameters, combining with the aural modality and employing other vibro-tactile illusion techniques such as the saltation and apparent motion.

**Acknowledgements.** This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT & Future Planning (No. 2011-0030079) and funded by the Forensic Research Program of the National Forensic Service (NFS), Ministry of Government Administration and Home Affairs, Korea (NFS-2015-DIGITAL-04).

## References

1. Alles, D. S. Information transmission by phantom sensations. IEEE Transactions on Man-Machine Systems, (1970), 11(1), 85-91.

2. Barghout, A., Kammerl, J., Cha, J., Steinbach, E., and El Saddik, A. Spatial resolution of vibrotactile perception on the human forearm when exploiting funneling illusion. Proc. IEEE Intl. Workshop on HAVE, (2009), 19-23.

3. Bekesy, G. V. Funneling in the nervous system and its role in loudness and sensation intensity on the skin. The Journal of the Acoustical Society of America, (1958), 30(5), 399-412.

4. Geshelder, G. Psychophysics: The Fundamentals. Lawrence Erlbaum Associates, (1997).

5. Israr, A., and Poupyrev, I. Control space of apparent haptic motion. IEEE World Haptics Conf., (2011).

6. Kim, Y., Lee, J., and Kim, G. J. Extending “Out of the Body” Saltation to 2D Mobile Tactile Interaction. Proc. of APCHI, (2012), 67-74.

7. Kingdom, F., and Prins, N. Psychophysics: A Practical Introduction, Elsevier-Academic Press, (2010).

8. Miyazaki, M., Hirashima, M., and Nozaki, D. The “Cutaneous Rabbit” hopping out of the body. The Journal of Neuroscience, (2010), 30(5), 1856-1860.

9. Mizukami, Y., and Sawada, H. Tactile information transmission by apparent movement phenomenon using shape-memory alloy device. Intl. Journal on Disability and Human Development, (2006), 5(3), 277-284.

10. Seo, J., and S. Choi. Initial study for creating linearly moving vibrotactile sensation on mobile device. Proc. IEEE Haptics Symposium, (2010), 67-70.

11. Yatani, K. and Truong, K. SemFeel: a user interface with semantic tactile feedback for mobile touch-screen devices · Proc. of UIST, (2009), 111-120.

12. Nokia, Bouncing ball on Nokia N900. http://bit.ly/x2-bouncingball, (2013).