JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

1

Back-of-Device Force Feedback Improves
Touchscreen Interaction for Mobile Devices
Jens Maiero, David Eibich, Ernst Kruijff, André Hinkenjann, Wolfgang Stuerzlinger, Hrvoje Benko,
Gheorghita Ghinea
Abstract—Touchscreen interaction suffers from occlusion problems as fingers can cover small targets, which makes interacting withsuch targets challenging. To improve touchscreen interaction accuracy and consequently the selection of small or hidden objects weintroduce a back-of-device force feedback system for smartphones. We introduce a new solution that combines force feedback on theback to enhance touch input on the front screen. The interface includes three actuated pins at the back of a smartphone. All three pins
are driven by micro servos and can be actuated up to a frequency of 50Hz and a maximum amplitude of 5mm. In a first psychophysicaluser study, we explored the limits of the system. Thereafter, we demonstrate through a performance study that the proposed interface
can enhance touchscreen interaction precision, compared to state-of-the-art methods. In particular the selection of small targets
performed remarkably well with force feedback. The study additionally shows that users subjectively felt significantly more accuratewith force feedback. Based on the results, we discuss back-to-front feedback design issues and demonstrate potential applications
through several prototypical concepts to illustrate where the back-of-device force feedback could be beneficial.
Index Terms—User interfaces - Haptic interfaces, Human computer interaction - Back-of-device interaction, Mobile applications

F

1

I NTRODUCTIONR

ESEARCHERS have been fascinated with the possibilities
of back-of-device (BoD) interaction, since the appearance of the first prototypes about a decade ago. Motivatedby efforts to minimize screen occlusion through fingers,
early prototypes like LucidTouch [1] made use of pseudotransparent displays. At the back, users could interact withthe full screen content using touch input without occluding
content. One hope was that back interaction could compensate for poor pointing performance, especially for theselection of smaller objects.
Within this paper, we take an alternative approach to
BoD interaction. Instead of touch input on the back, we
explore the potential of force feedback at the back to enhancethumb-based touch interaction on the front-of-the-device(FoD) screen. Adding force feedback to smartphones has
great potential, as many force events currently are substituted through tactile (vibration) or visual-only feedback
mechanisms, which affects the perception of those events[2]. Consider pressing a button: in real life, we receive
physical (force) and tactile (surface) feedback while pressing
the button down. With current smartphones, usually audiotactile feedback (”click”) and a change in button color indicates that a button is pressed, which is not compliant with

•

•
•
•

Jens Maiero, David Eibich, Ernst Kruijff and André Hinkenjann arewith the Institute of Visual Computing and the Department of Computer
Science at the Bonn-Rhein-Sieg University of Applied Sciences, Germany.
E-mail: forename.surname@h-brs.de
Wolfgang Stuerzlinger is with the School of Interactive Arts + Technology
at Simon Fraser University in Vancouver, Canada.
E-mail: w s@sfu.ca
Hrvoje Benko is with Facebook Reality Lab, US.
E-mail: hbenko@gmail.com
Gheorghita Ghinea and Jens Maiero are with Department of Computer
Science, Brunel University, London, U.K.
E-mail: george.ghinea@brunel.ac.uk, jens.maiero@brunel.ac.uk

Manuscript received September 18, 2018; revised February 10, 2019.

real-world interaction. Based on previous work, we assume
that force feedback could affect performance positively, inparticular for thumb-based interaction that often suffersfrom occlusions. Moreover, as we will investigate in ourstudies, force feedback might improve performance in tasks
that, in the real world, do not depend on such feedback.
Studies have shown that tactile feedback can increase
performance in target selection tasks [3] [4]. However, theexploration of force feedback in smartphones is rare. A
major cause is that adding force feedback actuators to thefront display has many physical form factor constraints,
which makes physical construction challenging. For example, mounting an actuator on the front would occlude partof the display, and likely hinder input. In contrast, themain research problem our approach faces is how well force
feedback at the BoD works.
1.1

HapticPhone contributions

Through a novel device for exploring the haptic design
space on the back of smartphones, this paper presents a
new BoD approach (see Figure 1). We use BoD feedback
to overcome physical form-factor challenges for adding
force feedback to the front of the device and explore thisnovel design space by looking into the human and technicalaspects of our new interface approach. The direction of thishaptic feedback method is orthogonal to the back of themobile phone, similar to a pin moving out of the back andpushing the finger away.
The four core contributions of this paper are:•

A novel interface and feedback metaphor, which extends
a smartphone with relayed back-to-front force feedback to enhance touch-based interaction on the front
screen.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015•••

A psychophysical perception study that confirmed ourforce feedback mechanisms produces a Weber Fraction (8.29 %) comparable to similar force devices,confirming the rendering effectiveness of our design.
A performance study, which illustrates that relayed
back-to-front force feedback can improve accuracy,even in comparison to vibration. The performed
study shows that our approach can improve interaction accuracy up to a factor of 1.5 (selection) andup to 1.3 (drag-and-drop), compared to conditions
without additional feedback (visual-only).
Several applications that explore how back-of-device
force feedback can be used for higher-level functionality.

2

R ELATED W ORK

2.1

Back-of-Device Interaction

Touch is a compelling input modality for smartphones. Yet,it suffers from limitations as the user’s finger can easily
occlude screen content. To overcome this limitation, Wigdor
et al. presented a system called LucidTouch to explore how
BoD interaction can overcome the occlusion problem [1].
Extending results from earlier prototypes, this paper ignitedinterest in BoD interaction. Similar systems include Wigdor
et al.’s ”under the table” interaction [13], the HybridTouchsystem [5] that explored manipulation on the front and theback of a PDA, and the Behind Touch system [6] studying
text input at the back. Furthermore, LimpiDual offered atransparent display system with two touchscreens, one at
the front and one at the back [7]. Fukumoto introduceda passive soft-gel based transparent film that provides abutton-push feeling, which can be attached to either thefront or back of the device [14].
Through an initial study, Wigdor et al. showed that with
their pseudo-transparent display approach, users preferred
BoD interaction due to reduced occlusion, higher precisionand the possibility of multi-finger input [1]. Since then, various studies have looked into different interaction aspectsof BoD interaction, including gestural interaction [15] [16],the performance of dual touch input [7], the performanceof hand postures in front and back interaction [17], andtext input [18]. Others have explored simultaneous backand-front-of-device interaction [9], [19]. BoD interaction also
affords the creation of smaller scale interactive display devices, as identified in [8]. Overall, studies underlined thatBoD interaction can improve performance in a number of
aspects.
Finally, UniPhone explored the use of auxiliary finger
input in one-handed mobile interaction, which showed thatpressure-sensing areas at the edge (instead of the back) of asmartphone can improve one-handed interaction [20].
2.2

Haptic Feedback on Mobile Devices

Combining force feedback with smartphone interaction is alargely unexplored area. While flexible media systems suchas Flexpad [21] and jamming interfaces [22] have exploredthe potential of haptic and elastic input [23], these systems
have limitations in terms of actuation and feedback. Moreaccurately controllable haptic-based methods have been

2

demonstrated for other actuated devices, especially mouselike input [24].
Other work has illustrated promising results for underfinger force actuators using rod-like systems [3]. Akamatsu
showed that tactile feedback can improve completion time
when selecting targets with a mouse [25]. Our approachis related to this way of actuation, as we support similar
finger-tip based feedback. In contrast to force feedback, tactile interaction has been explored far more intensely, mostly
using built-in vibrotactors of mobile devices or tactile screen
overlays. Tactile interaction has been shown to improvetask performance on touchscreen displays [26], for example
for virtual keyboard interaction [11], button clicks [27] andin combination with simultaneous screen operation [28].
Some researchers also demonstrated improvement in task
performance under cognitive load [29]. Furthermore, tactile
cues have been used to simulate various surface [30] [31] orfriction characteristics [32]. Finally, some researchers havelooked into passive shapes, like wedges or dimples at theback of a phone, to improve performance [33], which hassome relevance to the shape of our actuator used in ourpilot studies.
Another area that is closely related to BoD interaction aresystems that use tactile grids on smartphones, e.g., [4] [34].
The usage of tactile grids supports not only eyes-on, but alsoeyes-off interaction, i.e., blind operation of a smartphone.
Non-visual interaction has also been explored previously,e.g., [35] [36] [37] [38], showing the potential of using suchmethods for in-the-pocket operation of a smartphone, or forvisually impaired people [39]. Yannier et al. have shownthat haptic feedback on the back can improve children’s
reading experience and make it more memorable [40]. Finally, while we use servos for actuation, our work resembles
other actuated display systems such as TouchMover [41]and ForceTab [42]. However, due to the form-factor andway of actuation, our approach is principally different. A
few researchers have also looked into actuation at the sideof mobile devices. The Haptic Edge Display introduced alinear array of tactile pixels at the side of a smartphone andpresented a psychophysical study to measure ideal resolutions [12]. In general BoD haptic feedback has been used
for, e.g., shape recognition, in pocket interaction, and GUI
interaction. In contrast, our proposed prototype introduces
a novel set of application areas and a performance study todemonstrate that haptic feedback is able to improve touch
precision.
2.3

Thumb-based Interaction

In our interface approach, we build on previous findingsin thumb-based interaction [43] [44] [45]. Current research
focuses on improving interactive metaphors that consider
the physical limitations of the thumb. For instance, a novel
way to increase the thumb interaction space [46] was motivated by three problems of thumb-based interaction: reachability, occlusions and limited multi-touch interaction. Thisapproach also took the space above the smartphone into
account. Furthermore, the system presented by [43] uses
the contact size of the thumb on the touchscreen for singlehanded mobile interaction. We extend the state of the art
in this area by implementing a novel approach to relay

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

3

TABLE 1
Classification of state-of-the-art approaches to show how the back of the device is used (input/feedback) and which applications or goals aretargeted by each approach, presented in tabular format.
Input

ApproachFeedback

Research focus/Context

touchpressure

tactileforce

precision

occlusion

manipulation

typingeyes-free

Wigdor et al. [1]

x

-

-

-

x

x

x

-

-

Sugimoto et al. [5]

x

-

-

-

-

x

x

x

-

Hiraoka et al. [6]

x

-

-

-

-

x

-

-

-

Iwabuchi et al. [7]

x

-

-

-

-

x

-

-

-

Baudisch et al. [8]

x

-

-

-

x

x

-

-

-

Liang et al. [9]

x

-

-

-

-

-

x

-

-

Corsten et al. [10]

-

x

-

-

-

x

-

x

Corsten et al. [4]

-

-

x

-

-

-

-

-

Brewster et al. [11]

-

-

x

-

-

-

-

x

-

Jang et al. [12]

-

-

-

x(edge)

-

-

-

-

x

Our Approach

-

-

-

x

x

x

-

x

x

x

metaphors. We extend previous work by introducing a BoD
force feedback interface and approach to increase touch
precision and to enhance subjective user performance.

3

Fig. 1. Illustration of the prototype, a touchscreen for thumb interaction
is shown on top and the force feedback pins in landscape orientation are
depict below.

Here, we illustrate our system design and implementation,and describe the underlying design rationale. As the literature review shows that force feedback has improved
interaction [42] [12], our main design goal was to combine
BoD interaction and haptic feedback with thumb-based
interaction. We assumed that continuous, assisting force
feedback can provide a secondary cue for improved touch
performance and increase the user experience in terms ofsubjective performance. One-handed interaction is a wellestablished method of interacting with a smartphone [47].
Therefore our work explores thumb-based interaction, as
one-handed interaction uses the thumb for input control.
To do so, we have attached a force feedback mechanisms tothe back of the device that relays feedback to touch-input
on the screen at the FoD. As such, our design couples front
touch-based screen interaction and BoD feedback.
3.1

force feedback on the BoD to touch input on the front,
where feedback on the index finger (at the back) is syncedwith touch input (through the thumbs) on the front. Results
indicate that this approach can increase touch interaction
accuracy and it can also increase perceived performance,in terms of completion time, accuracy, and quality of the
feedback.
Table 1 gives an overview about the main research
results in the field of BoD input and feedback on mobile
devices. The table also shows that force feedback at the
BoD is not well researched yet. Most of the BoD systems
focus on finger occlusion issues and overcome this challenge
by moving touch input to the back. Other researchers use
BoD interaction to introduce novel manipulation and typing

H APTIC P HONE S YSTEM D ESIGNTechnical Design Rationale

To overcome physical form-factor constraints – such as
screen occlusion, input hindrance, and un-ergonomic grip
positions – at the front of smartphones, we explore thefeasibility of adding force feedback at the back of the device
to support relayed feedback for FoD touch input.
We designed the new system for both one-handed (portrait) and two-handed (landscape) interaction. Input is performed by either the left or the right thumb, or both, on the
FoD, while haptic feedback is relayed to the index fingers
at the BoD. After several hardware design iterations thatlooked into ease-of-use, grip stability, and reachability of
the feedback elements, and informed by the results of Le etal. [47] on appropriate finger locations for one-handed BoD
interaction, we chose a three pin layout. The arrangementand dimensions are shown in Figure 2. This layout allows

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

4

80
60
40
0

20

time [ms]120system latency

0mm
7.4V

1mm
7.4V
force

Fig. 2. Technical drawing of our interface and the chosen pin layout, withdimensions for the top and side views.

users to use the outer pins comfortably while interacting inlandscape mode, while the middle or top pin is used for
one-handed portrait interaction.
The remaining fingers grasp the ergonomically shaped
bottom part of the device, to stabilize and balance the device
during interaction and feedback. To improve the grip on
the device, we attached a rubber based grip tape aroundthe edge of the unit. The grip has also been designed toafford comfortable device usage in both portrait and landscape mode. However, one-handed usage may suffer fromlimitations, as previous work has shown that one-handed
interaction can lead to more fatigue, less precision, and aless secure grip of the device [43].
Force feedback is provided by three actuated pins at theback of the device (see Figure 2), at the locations where theleft or the right index finger naturally rest. Three dimples in
the housing help to keep the fingers at the ideal locations,where force feedback is provided.
3.2

HardwareOur interface comprises three high voltage (7.4V) micro
servos (BMS-22HV), each measuring 23.0 x 12.0 x 25.4 mm,
running with a speed of 50 ms per 60◦ rotation at no load.
Each servo has a pulse width of 1200 µm and a maximum
torque of 0.245 N/m. We estimate that the servos have aresolution of 0.2◦ as follows: a pulse width of 1200 µm for120◦ results in 1200 steps for 120◦ . With a dead band widthof 2 µs this results in 600 controllable steps for 120◦ . All inall, the servos can be controlled with a resolution of 0.2◦ ,which corresponds to a radian measure of 0.049 mm withan arm length of 14 mm(see Figure 4 left).
For each of the three pins a modified scotch-yoke mechanism was developed to convert the servos rotational movement into a linear movement (see Figure 4). For smaller pin
movements, e.g., amplitude 1 mm, our approach achieves afrequency of 50 Hz. In idle mode, when no counter-force isapplied, the power consumption of the prototype is about
10mA. With a strong counter-force the consumption is about
500mA.
The design enables a feedback range along a single axis
from 0 up to 5 mm. All hardware components are mounted
in a self-contained unit at the back of an Android 7 Huawei
P9 Plus mobile, which also supports pressure-based input.
All other parts, such as the case or the scotch-yoke mechanism, are 3D printed, which ensures easy reproducibility.

0mm
6.0V

1mm
6.0V
tactiledisplay

Fig. 3. System latency measurements (in milliseconds) for force, tactileand display, shown as a bar plot. The actuators were operated eitherwith 6 V or with 7.4 V and latency was measured with 0 mm or 1 mm pinheight.

The entire prototype measures 157.6 x 80.6 x 28.5 mm,weighs about 200 g and has a resolution of 1920x1080. A
16-bit Adafruit Feather (M0 Basic Proto) micro controller,
directly connected via USB (OTG) to the smartphone, handles the communication between feedback elements andthe application, which means that touch input is forwarded
with low latency to the pins.
The current prototype relies on a cabled solution. However a completely self-maintained unit can be created using
an Arduino Nano and communication over Bluetooth. Servo
motors could be powered with a buck-boost converter,
which can increase or decrease the input voltage and which
can permit the use of small battery packs.
Figure 3 lists the latency of the systems hardware components in milliseconds. In a experimental setup, latencyof the touchscreen (visual), the vibration (tactile), and the
proposed feedback (force) were measured. Force feedback
was measured under multiple conditions, with 6 V vs 7.4 V
and pin heights of 0 mm vs 1 mm to show the influenceof the described parameters on latency. A two pole relay
was used to trigger a touch screen using a microcontroller,
similar to Deber et al. [48]. After triggering the touchscreen
a photodiode (Tru Components 5013M1C) was used tomeasure the time between touchscreen event and visualfeedback. To obtain tactile feedback latency, the photodiode was replaced by a vibration sensor (Phidgets 1104).
To obtain force feedback latency, an electric circuit wasmechanically closed. This mechanism allows to measure theelapsed time of the touch screen event and its associatedfeedback, known as latency. For each feedback modality,
100 measurements were performed automatically. Figure
3 shows the average measured latency and the standard
deviation of the measurement data. Opinions differ on theinfluence of latency on performance [49] [50] [51]. Yet, since
the measured values are very close to each other and mostly
below 100 ms, we believe that a performance comparison of
the three components is valid and sensible.
3.3

Implementation

In our implementation, we support two different types oftouch input, namely position- and force-based interaction.
The latter makes use of the pressure-sensitive touchscreenof the smartphone.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

5

Fig. 4. Modified scotch-yoke mechanism to translate rotational into linear movements. An exploded diagram to introduce the used components(right), a dimensioned front view (middle), and the parameters for controlling the pins height (left). Height is determined by h = k ∗ tan(θ).

In contrast to conventional tactile feedback through vibration on the screen, the introduced force feedback can
support several clearly distinguishable levels of intensities,
as our psychophysical study will demonstrate below. These
intensities can be rendered in a continuous and discreteway. For the proposed haptic feedback two main software
components are important. First, the application on thesmartphone, which takes the orientation of the smartphone
into account and offers developers the possibility to address
the three pins individually. The smartphone communicateswith a micro controller via I2 C. The second component,
the software on the micro controller, receives data from thesmartphone and passes it directly to the motors.

4

P SYCHOPHYSICAL P ERCEPTION S TUDY

We conducted this study to determine the relationship between stimuli and sensation at the index finger using the
introduced interface. The study examines haptic perception
aspects and constraints of the BoD system. We performed ajust noticeable difference (JND) experiment to address thequestion - which force signals can users easily distinguish?
In addition, we studied the relationship between thumband index finger movements through a second, spatialcompliance, task. This investigates if it is possible to transfer
movements of the index finger to the thumb and how theusers perceive such movements.
4.1

Participants

12 users (3 female, age M=30.7/SD=5.8) volunteered toparticipate in this study. All were right handed. Participants
had various experiences with force feedback, ranging fromno to regular experience. All participants used smartphones
regularly. Each participant completed all 60 trials, which
took on average 12 minutes. The index fingers of the subjects
were 88.3 mm (SD=9.6 mm) and the thumbs were 64.0 mm(SD=4.9 mm) long on average.
4.2

Apparatus

Users operated the mobile system as described in the systemdesign section, while being seated comfortably at a desk.
Users could rest their arms on the desk, and were allowed
to take small breaks between tasks. During the whole user

study, users wore noise-canceling headphones that played
white noise to prevent users from hearing the servo actuation, as the servo sound could provide some information
about the force feedback.
4.3

Tasks and Procedure

The psychophysical perception study consists of two tasks.
The first task investigates the just noticeable difference offorce stimuli afforded by the pins, while the second task
looks at spatial compliance. Both tasks were performed ina counterbalanced order. Participants were asked to use
their dominant hand for interacting with the system. This
ensured that the dominant thumb was used for touch interaction and the index finger of the same hand for therelayed feedback. To do so users were asked to place theirindex finger on one of the outer two pins, while holding thedevice two-handed in landscape mode. In addition, users
were advised not to apply force to the pins.
The JND experiment was performed based on the principle of constant stimuli with a two-alternative forced choice
protocol, similar to Geschneider [52]. This protocol specifies
that n times a randomly chosen stimulus from an appropriate interval must be compared to a standard stimulus. Thiscan take place either spatially or temporally shifted.
In the experiment the standard stimulus S was set to
1 mm while the comparison stimuli Cn were one of 1.2, 1.4,
1.6, 2.0 or 2.4 mm. Stimuli were defined by the distance thepin was moved towards the index finger, i.e., away fromthe device. Each participant completed 10 repetitions. Each
repetition was fully randomized, resulting in a total of 60
trials. For each trial, users were prompted to press a start
button, to enable them to prepare themselves for the task.
Each stimulus was presented for 2 seconds, with a pauseof 1 second in between. The order of stimuli was chosenrandomly, meaning the standard stimulus was presented
either first or second. Users then had to determine which of
the two stimuli was higher. Before the experimental tasks,
users were allowed to do three practice trials. All training
trials were marked visually so that the users knew when theactual study started.
The other task focused on the psychophysical perceptionof depth and height, which combines visual feedback andtouch interaction (thumb) on the screen with BoD force

6

1

1

1

1

1

0.95

0.95

0.95

0.95

0.95

0.9

0.9

0.9

0.9

0.9

0.9

0.85

0.85

0.85

0.85

0.85

0.85

0.8

0.8

0.8

0.8

0.8

0.75

0.75

0.75

0.75

0.75

0.7

0.7

0.7

0.7

0.7

0.65

0.65

0.65

0.65

0.65

0.6

0.6

0.6

0.6

0.6

0.55
0.5

0.55

0.55

0.8

1

1.2

1.4

1.6

1.8

2

2.2

1
0.95

0.5

0.8

1

1.2

1.4

1.6

1.8

2

2.2

0.5

0.55

0.8

1

1

0.95

0.95

0.9

0.9

0.9

0.85

0.85

0.85

0.8

0.8

0.8

0.75

0.75

0.75

0.7

0.7

0.7

0.65

0.65

0.65

0.6

0.6

0.6

0.55

0.55

0.55

0.5

0.5

0.8

1

1.2

1.4

1.6

1.8

2

2.2

0.8

1

1.2

1.4

1.6

1.8

2

2.2

0.5

1

1.2

1.4

1.6

1.8

2

2.2

0.5

1

1.2

1.4

1.6

1.8

2

2.2

1.2

1.4

1.6

1.8

2

2.2

0.7
0.65

0.55

0.8

1

1.2

1.4

1.6

1.8

2

0.5

2.2

0.9

0.9

0.85

0.85

0.8

0.8

0.75

0.75

0.7

0.7

0.65

0.65

0.6

0.6

0.6

0.55

0.55

0.55

0.5

0.5

1

1.2

1.4

1.6

1.8

2

2.2

1

1.2

1.4

1.6

1.8

2

2.2

0.8

1

1.2

1.4

1.6

1.8

2

2.2

0.95

0.95

0.9
0.85

0.8

0.8

1

1

0.95

1

0.5

0.8
0.75

0.6

0.55

0.8

1

0.8

Percent Correct

1
0.95

Percent Correct

Percent Correct

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

0.8

1

1.2

1.4

1.6

1.8

2

2.2

0.8
0.75
0.7
0.65

0.5

Stimulus Level [mm]Fig. 5. Results of the psychophysical perception study. The pyschometric curves of all 12 participants are plotted in this Figure. Obtained data of
the individual observers were fitted using Weibull function. Each plot shows the stimulus level on the x-axis and the percentage of correct stimuli
detected on the y-axis.

feedback on the index finger. With this task we exploredthe spatial compliance between index finger and thumb,where we assume that the thumb can “perceive” heightdata from the index finger. This addresses the question if
users perceive a valley at the thumb when the distance
between thumb and index finger increases or if participants
will perceive a hill and vice versa. In this task, participants
were asked to explore a 3D Gaussian-like shape with the
thumbs on the touchscreen. Since the screen has 2 physicaldimensions, the 3rd dimension was mapped to the index
finger, like a height map.
The maximum feedback (height/depth) was set to
2.4 mm whenever the thumb reached the maximum of the
shape displayed on the screen. While visual feedback remained constant over all trials, force feedback was designed
so that either the distance between the index finger andthe thumb became smaller or larger. Because the task was
simple and easy to understand, only 8 trials were performed
in random order per user. After each trial, users were asked
if they felt that the 3D Gaussian-like shape was directed
either into (valley) or out (hill) of the display.
4.4

Results

The JND experiment was analyzed using the psignifit MATLAB toolbox, similar to previous work [12]. Three relevant
values were determined: point of subjective equality (PSE)and stimulus values at 25 % and 75 % probability. With thesethree values the JND and the Weber Fraction (WF) were
determined. The results of the psychophysical perception
study are summarized in Table 2 and the psychometriccurves are plotted in Figure 5. The (index finger) depth
perception threshold of our system is on average 0.1 mm.
Our results are in line with what Jang et al. [12] reported,
who found an average JND of 0.15 mm.
The evaluation of the answers shows that in the second
part of the study 83 % of all subjects felt a valley whenever
the distance between index finger and thumb decreased and
91 % felt a hill whenever the distance increased. Since users
were asked what they perceived with their thumbs on thetouchscreen this indicates that users seem to experience thesensation that their thumbs sink into the smartphone when
the distance between index finger and thumb increases andvice versa.

TABLE 2
List of the results of the psychophysical perception experiment

depth perception
participant

5

PSE [mm]

JND [mm]

WF [%]

1
2
3
4
5
6
7
8
9
10
11
12

1.20
1.28
1.06
1.23
1.21
1.21
1.29
1.23
1.21
1.15
1.10
1.14

0.11
0.07
0.04
0.13
0.11
0.12
0.12
0.10
0.06
0.17
0.08
0.04

8.89
5.88
3.90
10.80
9.43
9.62
9.62
7.82
4.91
14.74
7.01
3.83

Mean
SD

1.18
0.06

0.10
0.04

8.29
3.43

TOUCH ACCURACY S TUDYThis study mainly examined touch accuracy of the BoD
feedback mechanisms, comparing force with tactile andvisual-only feedback. Since thumb interaction offers a relatively smaller interaction area due to physical constraints
for reaching compared to using the index finger, a higher accuracy might offer the possibility of comfortably controllingdenser interaction elements on a smaller area. To determine
touch accuracy we used two tasks, a drag-n-drop and a selection task. The drag-n-drop task is designed to determine
the accuracy of a constant force feedback stimulus and tocompare this with other feedback metaphors. The selection
task pursues the same goal, however, the intensity of theforce feedback was varied as well in this task. Furthermore,in a questionnaire we explored the user experience in termsof subjective performance of completion time, precision,and quality of the feedback. Since our proposed system
employs thumb-based interaction, we excluded multi-touch
metaphors from the study.
5.1

Participants

12 users (5 female, age M=29.8/SD=4.9) volunteered to participate in the laboratory experiment. Again, all participants

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

7

Fig. 6. Exemplary illustration of the drag-and-drop study. Users had todrag the orange disk onto the grey one. The black arrow was not visible
to the participants.

Fig. 7. Illustration of the selection task. Users had to select a specificcell in the stimulus matrix. Cells have alternating intensities (tactile) orheights (force).

were right handed. Participants had varied experience withforce feedback, ranging from none to regular experience.
Each participant completed all of the 90 trials, which took
an average of 12 minutes.

After the study, users completed a questionnaire, inwhich we queried them about the three parameters accuracy, completion time, and quality for each feedback modality using a 7-Point Likert scale, ranging from “strongly
disagree” to “strongly agree”, through statements such as: I
could perform the task precisely, I could complete the task quickly,
and The quality of the feedback was excellent.

5.2

Apparatus

The apparatus for landscape interaction was similar to theone used in the first study reported above. In addition,
we investigated interaction in portrait mode in this study,
where the thumb of the dominant hand was used to handle
the touch input and the corresponding index finger receives
the force feedback through the middle pin. Users wereallowed to use the other hand to keep the smartphonestable.
5.3

Design and Procedure

Before the actual study began, users were encouraged to
explore the new feedback mechanism to understand theprinciple of the feedback. Participants had to complete
two tasks, a drag-and-drop and a selection task. Since theproposed BoD approach is comparable with more common
feedback metaphors, users were asked to use all three different feedback metaphors (force, tactile, visual-only) to gain
an understanding of each feedback modality and their differences. The order of the two tasks was chosen randomly.
For the drag-n-drop study, the center of the rectangularinteraction area (30 mm x 45 mm) was above the corresponding pin, the size was chosen so that all participants
could reach all elements in this area. The size of the interaction area for the selection task was determined by the sizeof the stimulus matrix, which was at most 30 mm x 30 mm.
Can interaction of the thumb be influenced by controlled
movements induced on the index finger? To investigate thisquestion, this study focused on the interplay of relayed
feedback from the index finger at the BoD to the thumb withthe aim of increasing accuracy of thumb interaction on the
FoD. In addition, since the offset between fingers could also
affect touch accuracy, we also investigated the influence ofthe offset in the axis orthogonal to the touchscreen between
the thumb position on the touchscreen and index finger on
the back. All tasks were performed in landscape and portrait
mode, as both modes are common in every-day smartphoneinteraction.

5.3.1

Drag-and-Drop Task

In this study, participants were asked to perform a dragand-drop task by moving a disc as accurately as possible
onto another. We used a within-subject design to examine
touch accuracy and user experience, through a (3x3x2) factorial design with 3 independent variables: screen orientation(levels: landscape and portrait), target disc size (levels: 1.0,
1.5, 2.0 mm) and feedback type (levels: force, tactile andvisual-only). The factors disc size and feedback type were
fully randomized over the trials, whereas the factor screen
orientation was counterbalanced. The dependent variables
were touch accuracy and user experience. Touch accuracy
was measured with an error rate in mm, representing the
distance to the center of the target disc. The user experience
was measured through the questionnaire described above.
To examine touch accuracy, users had to drag one disc(the action disc) and drop it onto another (target) disc.
Participants were asked to do this as accurately as possible.
Throughout the entire study the size of the action disk
(diameter=4 mm) was the same. The size of the target discvaried (diameter=1.0, 1.5, 2.0 mm), with the largest target
size chosen so that users still had a chance to see or estimate
where the disc is located, without completely covering
the target. In contrast, the smallest target was completely
covered by the thumb when both centers intersect. Bothdiscs appeared at random, but reachable, locations. Once
participants assumed, that the center of the action disc wasabove the center target disc, participants had to submit theirposition result by pressing a button with the other thumb.
We designed the visual feedback to be similar acrossdifferent feedback types, while the perceived stimuli atthe index finger/thumb were different. To initially selectthe action disc, users had to select it for at least 50 ms.
Thereafter, participants were able to drag the disc with theirthumb. In the ’visual-only’ mode there was only visualfeedback. In the other conditions, tactile respectively force

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

feedback was enabled whenever the center of the action discwas inside the target disc, which means that either a 2 mmforce feedback event was rendered on the BoD or standard
vibration was turned on. Then force or vibration stayed on
until the action disc center went outside the target disc or
the user confirmed the location with their other thumb.
5.3.2 Selection Task
In the selection task a within-subject design was used to
examine touch accuracy and user experience. We used a
3x3x2 factorial design, with 3 independent variables: cellsize (levels: 3, 5, 7 mm), feedback type (levels: force, tactile
and visual-only) and screen orientation (levels: landscapeand portrait), . Each user had to complete 3 repetitions,overall 54 trials. The factors cell size and feedback typewere fully randomized over the trials, whereas the factor
screen orientation was counterbalanced, so that participants
completed all trials for the same screen orientation in one
block. This avoids users having to constantly rotate thescreen, which could lead to unnecessary fatigue and increased stress. The dependent variables were touch accuracy
and user experience. Touch accuracy was measured with anerror measurement in mm, corresponding to the distanceof the touch center to the center of the selected cell. User
experience was assessed with the questionnaire describedabove.
We used a stimulus matrix of abstract targets, as arepresentation similar to common interaction targets, suchas menus, toolbars, tool palettes or links. Users were asked
to search and select an element in a 6x6 matrix, in landscapeas well as portrait mode (see Figure 7). The center of thestimulus matrix was located exactly above the feedback pin.
For each cell of the matrix a unique number from 0 to 35
was randomly generated. The number to be selected wasvisually marked. Users could then use the thumb of their
dominant hand to select the cell as best they could, as close
as possible to the center of the cell. To submit the target
position participants had to press a button with the other
thumb. We chose three cell sizes of 3, 4 and 5 mm for thisstudy, based on earlier research on appropriate target sizes
for thumb interaction [46], [53].
globallocal

Fig. 8. Illustration of the 2 different feedback approaches for a 6x6
stimulus matrix. Right: constant feedback intensities (pin heights) over
the whole matrix and Left: feedback intensities increasing towards the
middle to help the user perceive the global position of the matrix.

Depending on the feedback type, we implemented different approaches to search for and select a cell. For visualonly feedback, as the name suggests, no additional cues

8

were rendered. For tactile feedback, we provided vibration
via the in-built functionality of the touchscreen device.
This means that the feedback was rendered to the wholedevice, including the front screen. Since Android does notnatively offer the possibility to change the intensity of thevibration, we used a binary approach, i.e., only on or off.
Thus, vibration feedback was either continuously switchedon and off, alternating with the cells (see Figure 7). This
enables the user to sense the transitions between cells. Forforce feedback, two approaches were introduced. First, local
feedback, an alternating feedback per cell with a constantpin height of 2mm (for yellow cells) and 0 mm (for grey
cells). Second, global feedback, an alternated feedback per thecell, but with an increase for the intensity of the feedback
towards the center of the (whole) stimulus matrix, up toa maximum of 2.4 mm in discrete steps, depending on thesize of the matrix. For example, the step size was 0.8 mm forthe 6x6 matrix. We chose to increase the feedback towards
the middle of the matrix so that users could not only sense
the transitions from cell to cell (local position), but also
determine the (global) position within the stimulus matrix
based on the intensity. At the end of the study, a question
was asked about the perception of localization in the matrix.
Pin heights (feedback intensities) for local and global force
feedback are shown schematically in Figure 8.
Before each experiment, users were asked to explore eachof the feedback modalities for about a minute. Completiontime was measured per trial and logged for both tasks.
5.4

Results

5.4.1 Drag-and-DropAs the data were not normally distributed, we used a
Kruskal-Wallis test to analyze the data of the drag-and-drop
task. For the sake of completeness, the completion times inmilliseconds per trial and feedback are listed in Table 3.
TABLE 3
Mean time and standard deviation per trial for the drag-and-drop task inseconds (standard deviation in brackets).

visual-only
tactile feedback
force feedback

portrait

landscape3.17 (1.56)
3.95 (3.01)
3.84 (2.10)

3.79 (3.16)
3.87 (2.11)
4.01 (2.53)

Through the error distance measure we identified thatusers performed the task significantly more precisely withtactile and force feedback for both landscape (χ2 = 9.81, df =2, p < .01) and portrait mode (χ2 = 18.34, df = 2, p < .001),
see Figure 9. A pairwise comparison identified a significant
effect between tactile and no feedback (landscape p < .05/ portrait p < .001) and force and no feedback (landscapep < .01 / portrait p < .001). There was no significant effectbetween tactile and force feedback according to a Nemenyi
post-hoc test.
The results for the questionnaire data (see Figure 10)
showed that force and tactile feedback were perceived to be
more precise. The Fisher Exact test identified a significant
effect (p < .001) for the accuracy ratings. Participants felt
more accurate with force (M = 6.3/SD = 0.7) and tactile

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

9results drag−and−drop task
100

distance [px]80

subjective drag−and−drop performance

feedback
visual−onlytactilefeedback

force

landscapevisual−onlyportraittactile

force

7

60

6
5

40

4
3

20

2
1

0
2

1.5

1

2

1.5

1

0
precision

time

qualitytarget disc size [mm]Fig. 9. Box-plot for touch accuracy in the drag-and-drop study. For
smaller target sizes, force and tactile feedback have a significantly lower
error than visual-only.

feedback (M = 5.4/SD = 1.0) than with visual-only (M =3.6/SD = 0.9).
The responses for the completion time question were notsignificant. Users felt faster with force (M = 5.6/SD = 0.8)
and tactile feedback (M = 5.4/SD = 0.9) than with visual
only (M = 4.7/SD = 0.9), but not significantly so.
In conclusion, in both landscape and portrait mode tactile and force feedback were on average 1.33 times moreaccurate compared to visual-only feedback. Considering
only small targets where the thumb covers the entire disc,the average error in touch accuracy of force and tactile
feedback compared with visual-only is about 1 mm. The
results also indicate a higher variance for the average accuracy for visual-only feedback compared to other feedback
modalities. This can be interpreted as that the users felt more
confident in their selection with tactile and force feedback
than with visual-only feedback. Additionally, any potential
offset between touch and pin position seems to have had noobservable influence on accuracy and time per trial in the
study.
5.4.2 Selection
We were primarily interested in touch accuracy for theselection task. As the proposed BoD force feedback condition involved several intensity levels, this meant thatparticipants were able to determine not only their local
but also the global position in the matrix. Therefore, we
assumed that the touch accuracy would be higher with force
feedback compared to the visual-only or tactile conditions.
As the data were not normally distributed, we applied a
Kruskal-Wallis-Test instead of an ANOVA. We conducteda Nemenyi-Test to calculate pairwise comparisons between
feedback groups for the post-hoc analysis. For sake of completeness, the means and standard deviations of the completion times per trial are listed in Table 4.
We found a significant effect of touch accuracy for small
cell sizes (3mm) in landscape mode (χ2 = 10.626, df = 3,
p < .01). A pairwise comparison with the Nemenyi posthoc test between visual-only, tactile, force local and global
feedback showed a significant difference for all 3 feedback

Fig. 10. Subjective drag-and-drop performance in terms of precision,
completion time and feedback quality.

modalities compared to visual-only (force global/p < .05,
force local/p < .05, tactile/p < .05).
The analysis of the selection accuracy in portrait mode
also indicated a significant effect for the selection of small
sized cells (3mm) (χ2 =8.3779, df = 3, p < .05). The post-hoc
analysis for portrait mode reveals a significant differenceof touch accuracy for force global and visual-only feedback(p < .05). This shows that BoD force feedback can increase
touch accuracy in both conditions.
Our results illustrate that small targets can be selected
more precisely with a secondary cue (force and tactile).
Users were able to select all cell sizes (3 mm, 4 mm and5 mm) with a similar probability of 69.8 % (force global),
70.8 % (force local), 66.2 % (tactile) and 59.3 % (visual-only)
in both modes (landscape and portrait). Considering only
the smallest cell size, (3 mm), users could select the cells
correctly with a probability of 54.2 % (force global), 54.2 %(force local), 50.0 % (tactile) and 36.1 % (visual-only). Figure
11 depicts the results for all cell sizes.
TABLE 4
Mean time and standard deviation per trial in seconds for the selection
task (standard deviation in brackets).

visual-only
tactile feedback
force local feedback
force global feedback

portrait

landscape3.65 (2.18)
3.92 (2.19)
4.55 (3.10)
3.91 (2.46)

3.78 (2.29)
4.05 (3.0)
5.27 (5.70)
4.82 (4.70)

Figure 11 shows that the pure visual state has the highest error rate, whereas secondary feedback, whether tactile
or force, performs much better. The comparison between
global and local force feedback in the selection task did notreveal any statistical differences. However, in the questionnaire right after the study, users rated the global feedback
as more positive and supportive than the local feedback (M= 4.8/SD = 1.4).
The questionnaire data was evaluated using the Fisher
Exact test, which indicated that both accuracy (p < .001)
and time per trial (p < .01) were significant (see Figure 12).

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 201510

results selection task
100

subjective selection performance
feedback

portraitvisual−onlytactile

force

60

7
6

40
20

5

feedback
visual−onlytactileforce local
force global

4
3
2
1

0

selected cells [%]80

landscape

0

3

4

5

3

4

5

precision

time

qualitycell size [mm]Fig. 11. Percentage of successfully selected cells per size and mode.
The influence on accuracy for visual-only feedback decreases strongly
as soon as the thumb covers the target.

Fig. 12. Subjective selection performance in terms of precision, completion time and feedback quality. For all three parameters participants liked
force feedback most.

Participants felt more accurate with force (M = 6.1/SD = 0.9)
and tactile feedback (M = 5.6/SD = 0.8) than with visualonly (M = 3.0/SD = 1.2). Additionally, users felt also fasterwith force (M = 5.6/SD = 1.1) and tactile (M = 5.3/SD = 0.6)
feedback than with visual-only (M = 4.1/SD = 1.1).
The improved selection accuracy of smaller targets is
an important finding, extending previous research in BoD
interaction, such as [1], showing that haptic feedback is also
a valuable method to overcome occlusion problems. Whenlooking at the smallest cell size of 3 mm, force and tactile
feedback can on average increase accuracy by a factor of
1.5 compared to visual-only feedback. This means that thepresented feedback mechanism can improve selection tasks,
including text selection, considerably.
Participants filled out the SUS questionnaire to evaluate
system usability. They rated our new system on average at
81 with a standard deviation of 13. According to Bangor etal. [54] this means that the average is between ”good” and”excellent” usability.

Height perception. We created a Map Exploration application that supports simultaneous exploration of multivariate, map-related data. With normal map viewers, exploring multivariate data in parallel can become difficult.
Thus, complex visualization methods are normally needed
for such scenarios. In our application, we pass different
geographic data to different feedback elements, e.g., output elevation profiles via a servo and map environmental
pollution data to tactile stimuli. This choice effectively implements multi-channel feedback to provide feedback about
different aspects of the data. The application builds on the
JND results, which showed that the users can interpret finedifferences through force feedback. We assume this accuracy
can assist users in finding, e.g., a location on a map that isdesirable in multiple dimensions.
Gaming experience. Finally, in a 2D Racing Game, we
explore a combination of pressure input and height perception. Here, the car is steered through the touch position andinput pressure is mapped to speed, while force feedback
gives indication of the speed, by mapping the speed to theforce pin. While force feedback has been shown to improve
fun and immersion [55], we specifically targeted subjective
control accuracy here.

6

A PPLICATIONS

To show how HapticPhone supports higher-level tasks, we
implemented three applications that extend the basic (lowlevel) functionality demonstrated in the user studies (seeFigure 13). These applications demonstrate other instanceswhere force feedback at the BoD is also useful.
Pressure input. We implemented a 3D Volume Viewer
application based on stacked images that can be exploredthrough pressure input. In the first iteration, the appliedpressure was directly transferred to the intensity of the servo
motor. In this case force feedback was used to enhance scenenavigation, as force feedback indicates where the user is in
the scene: feedback is spatially compliant, as it is based on
displacement with more/less pressure. Specifically, the user
can directly feel if the currently viewed slice is further up or
down in the 3D volume stack, aligned with the displacement
axis. Thus, we assume that force feedback can be useful toafford more precision for finding a given layer in a volume.

7

D ISCUSSION

In our first study, we assessed if relayed feedback at the backof a device can support touch-based interaction on the front.
Doing so, we explored different back-to-front feedback mappings to investigate both perception (psychophysical limits)and performance. Through a second user study, we showedthat relayed feedback can indeed improve both objective
and subjective performance. In some instances results arecomparable to tactile feedback, while in others (selection
accuracy of small targets) objective and subjective performance and preference are better, which demonstrates thepotential of our approach.
Here, we discuss important factors that need to be considered when applying back-to-front relayed feedback inapplications.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 201511

Fig. 13. Exemplary application scenarios for the proposed BoD force feedback approach: (top), a volume viewer, a racing game, a map viewer,
all supported by force feedback on the back of the smartphone. For the Volume viewer, for example, the pressure on the touchscreen is directlytranslated to the pins’ height. When mapping the position, the information at the point of contact is taken (e.g. elevation data) and transferred to thepin.

Design space. In our device design and studies, we only
explored a subset of the whole design space afforded byinteraction on the back of a smartphone. A smartphone
provides many different input and output modalities atdifferent locations, which can be combined to create different mappings. E.g., a microphone, speaker, buttons, ornotification LED are normally located at the front, while afingerprint scanner, torchlight, or camera, are often locatedat the back of the device. Consequently, the design space
has many dimensions, as the same hardware mapping canbe used to elicit different back-to-front feedback N:M mappings, especially for force feedback at the BoD.
Direct vs. indirect mapping. Feedback can be directly or
indirectly mapped. While we studied only directly mapped
feedback, where a direct connection between the type ofaction and feedback exists, indirectly mapped back-to-front
actions are also possible. Moreover, similar to tactile or auditory warnings, back-of-device feedback can communicate
information about other processes than the one the user isinvolved in, for example to draw attention to a notification.
Additionally, the presented prototype could also be used toenable in-pocket or other eyes-off interaction.
Spatial compliance. In our work, we explored spatially compliant and non-compliant feedback. Our feedback
mechanism provides displacement along a single axis, afforded by the mechanical constraints of the servo. Whenusers press down on the screen, force feedback that moves
the index finger away from the phone is spatially consistent,
which matches the direction of the touch input action. Our
analysis of the results indicate that users could interpret
such spatially consistent feedback well, also because suchconsistency was preferred by users. Users stated that spatially inconsistent feedback, for example mapping up/down
dimple displacement to sliding left/right, was more difficult
to interpret.
Integration or separation. Spatial compliance also directly affects one of the key issues for back-to-front feedback, namely the physical separation of feedback (on the index finger) from the actual input channel (the thumb). This
also opens the question if integrated feedback is possible atthe back. While we currently only made use of feedback at
the back of the device, the back could be extended withsensing hardware, e.g., through force sensitive resistors.
Then, feedback does not have to be relayed. While this isdifferent from our core idea of back-to-front feedback relay,

it opens up an interesting venue for systems that couple
front-input or front-display with back-input-and-feedback.
Single- vs. multi-channel feedback. Another issue is thenumber of input or feedback channels being used. Here, weused a type of single-channel, integrated solution: the same
type of feedback was mapped to both left and right fingers,
where input location defines which finger was actuated.
However, and as discussed in the next section on applications, a system could separate both feedback channels, forexample by providing different kinds of force feedback at
the actuators at the back for different types of events.
Range and resolution Remapping values to work within
the provided (force) range is an issue for spatial consistency, in particular when fine-grained feedback is required.
The resolution of actuation is both a technical and humanissue. Technical limitations may affect the physical range
and levels of feedback one can provide. With our current
device, the range was a physical displacement of 5 mm,with a resolution of 0.2◦ , which corresponds to a radianmeasure of 0.049 mm. Through the just-noticeable difference
experiment, we showed that users can detect differences ofaround 0.1 mm.

8

C ONCLUSION

In this paper, we presented the concept of back-to-front
relayed feedback on smartphones and discussed the resultsof our evaluations. The studies involved HapticPhone, anovel smartphone interface that provides servo-actuatedforce feedback at the back of the device, which overcomes
physical form-factor constraints for providing force feedback on the front. Through our evaluations, we showedthat relayed feedback can improve interaction on the frontal
display. Based on the results, we discussed several factors
affecting the relayed back-to-front feedback paradigm. We
briefly summarize our main findings:••

Psychophysical analysis revealed that perception offorce stimuli using the presented mechanisms resulted in a Weber Fraction (8.29 %), which is comparable to other devices and confirms that the feedback
works appropriately and affords the perception offine differences.
Both selection and drag-and-drop task show encouraging results. In most situations, force and tactile
feedback performed about equally well. For both

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015selection and drag-and-drop, as expected, feedbackyielded better objective performance than without.
Accuracy for small targets was increased by factor of
1.5 (for selection) and 1.3 (for drag-and-drop). Especially for selection of smaller targets, force feedback
can produce better results than tactile, extending
previous findings that showed the potential of backof-device interaction for selection of smaller targets[1]. Users preferred force feedback over tactile or no
feedback, feeling more accurate, yet not faster. Thus,in terms of subjective performance, force feedback
was the best option.
As a next step, we intend to look more closely into
multi-channel, separated back-to-front feedback, which relays both force and tactile feedback about different events
controlled at the front of the display. Also, the discussion of
the factors affecting relayed feedback points to the potential
for many different types of devices, for example those thatcan flip or move. We did not discuss these options, as thisis beyond the scope of back-to-front feedback relay. Finally,as HapticPhone already is a fairly compact design, we areconsidering if it can be offered as a compact extension toexisting smartphone setups.

R EFERENCES
[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

D. Wigdor, C. Forlines, P. Baudisch, J. Barnwell, andC. Shen, “Lucid touch: A see-through mobile device,” inProceedings of the 20th Annual ACM Symposium on User
Interface Software and Technology, ser. UIST ’07. New York,NY, USA: ACM, 2007, pp. 269–278. [Online]. Available:
http://doi.acm.org/10.1145/1294211.1294259
K. A. Kaczmarek, J. G. Webster, P. B. y Rita, and W. J. Tompkins,“Electrotactile and vibrotactile displays for sensory substitution
systems,” IEEE Transactions on Biomedical Engineering, vol. 38, no. 1,
pp. 1–16, Jan 1991.
M. Akamatsu and S. Sato, “A multi-modal mouse withtactile and force feedback,” Int. J. Hum.-Comput. Stud.,vol. 40, no. 3, pp. 443–453, Mar. 1994. [Online]. Available:
http://dx.doi.org/10.1006/ijhc.1994.1020
C. Corsten, C. Cherek, T. Karrer, and J. Borchers, “Hapticase:Back-of-device tactile landmarks for eyes-free absolute indirect
touch,” in Proceedings of the 33rd Annual ACM Conferenceon Human Factors in Computing Systems, ser. CHI ’15. New
York, NY, USA: ACM, 2015, pp. 2171–2180. [Online]. Available:
http://doi.acm.org/10.1145/2702123.2702277
M. Sugimoto and K. Hiroki, “Hybridtouch: An intuitive
manipulation technique for pdas using their front and rearsurfaces,” in Proceedings of the 8th Conference on Human-computer
Interaction with Mobile Devices and Services, ser. MobileHCI ’06.
New York, NY, USA: ACM, 2006, pp. 137–140. [Online]. Available:
http://doi.acm.org/10.1145/1152215.1152243
I. M. Shigeo Hiraoka and K. Tomimatsu, “Behind touch: A text
input method for mobile phone by the back and tactile senseinterface,” in INTERACTION. Information Processing Society ofJapan (in Japanese): ACM, 2013, pp. 131–146.
M. Iwabuchi, Y. Kakehi, and T. Naemura, “Limpidual touch:Interactive limpid display with dual-sided touch sensing,” in
ACM SIGGRAPH 2008 Posters, ser. SIGGRAPH ’08. New
York, NY, USA: ACM, 2008, pp. 87:1–87:1. [Online]. Available:
http://doi.acm.org/10.1145/1400885.1400978
P. Baudisch and G. Chu, “Back-of-device interaction allows
creating very small touch devices,” in Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems, ser. CHI’09. New York, NY, USA: ACM, 2009, pp. 1923–1932. [Online].
Available: http://doi.acm.org/10.1145/1518701.1518995
M. S. W. S. Hai-Ning Liang, Cary Williams and P. Irani, “Aninvestigation of suitable interactions for 3d manipulation of distant objects through a mobile device,” in International Journal on
Innovative Computing, Information and Control, 2013, pp. 4737–4752.

12[10] C. Corsten, B. Daehlmann, S. Voelker, and J. Borchers,“Backxpress: Using back-of-device finger pressure to augment
touchscreen input on smartphones,” in Proceedings of the 2017
CHI Conference on Human Factors in Computing Systems, ser. CHI’17. New York, NY, USA: ACM, 2017, pp. 4654–4666. [Online].
Available: http://doi.acm.org/10.1145/3025453.3025565[11] S. Brewster, F. Chohan, and L. Brown, “Tactile feedback formobile interactions,” in Proceedings of the SIGCHI Conferenceon Human Factors in Computing Systems, ser. CHI ’07. New
York, NY, USA: ACM, 2007, pp. 159–162. [Online]. Available:
http://doi.acm.org/10.1145/1240624.1240649[12] S. Jang, L. H. Kim, K. Tanner, H. Ishii, and S. Follmer, “Haptic
edge display for mobile tactile interaction,” in Proceedings of the2016 CHI Conference on Human Factors in Computing Systems,
ser. CHI ’16. New York, NY, USA: ACM, 2016, pp. 3706–3716.
[Online]. Available: http://doi.acm.org/10.1145/2858036.2858264[13] D. Wigdor, D. Leigh, C. Forlines, S. Shipman, J. Barnwell,
R. Balakrishnan, and C. Shen, “Under the table interaction,”in Proceedings of the 19th Annual ACM Symposium on User
Interface Software and Technology, ser. UIST ’06. New York,NY, USA: ACM, 2006, pp. 259–268. [Online]. Available:
http://doi.acm.org/10.1145/1166253.1166294[14] M. Fukumoto, “Puyosheet and puyodots: Simple techniques
for adding ”button-push” feeling to touch panels,” in CHI ’09
Extended Abstracts on Human Factors in Computing Systems, ser.
CHI EA ’09. New York, NY, USA: ACM, 2009, pp. 3925–3930.
[Online]. Available: http://doi.acm.org/10.1145/1520340.1520595[15] K. Wolf, R. Schleicher, and M. Rohs, “Ergonomic characteristics ofgestures for front- and back-of-tablets interaction with grasping
hands,” in Proceedings of the 16th International Conference on
Human-computer Interaction with Mobile Devices &#38; Services, ser.
MobileHCI ’14. New York, NY, USA: ACM, 2014, pp. 453–458.
[Online]. Available: http://doi.acm.org/10.1145/2628363.2634214[16] X. Xiao, T. Han, and J. Wang, “Lensgesture: Augmenting mobileinteractions with back-of-device finger gestures,” in Proceedings ofthe 15th ACM on International Conference on Multimodal Interaction,
ser. ICMI ’13. New York, NY, USA: ACM, 2013, pp. 287–294.
[Online]. Available: http://doi.acm.org/10.1145/2522848.2522850[17] J. O. Wobbrock, B. A. Myers, and H. H. Aung, “Theperformance of hand postures in front- and back-of-device
interaction for mobile computing,” Int. J. Hum.-Comput. Stud.,vol. 66, no. 12, pp. 857–875, Dec. 2008. [Online]. Available:
http://dx.doi.org/10.1016/j.ijhcs.2008.03.004[18] O. Schoenleben and A. Oulasvirta, “Sandwich keyboard: Fast
ten-finger typing on a mobile device with adaptive touch sensing
on the back side,” in Proceedings of the 15th International Conferenceon Human-computer Interaction with Mobile Devices and Services, ser.
MobileHCI ’13. New York, NY, USA: ACM, 2013, pp. 175–178.
[Online]. Available: http://doi.acm.org/10.1145/2493190.2493233[19] H.-N. Liang, C. Williams, M. Semegen, W. Stuerzlinger,
and P. Irani, “User-defined surface+motion gestures for 3dmanipulation of objects at a distance through a mobile
device,” in Proceedings of the 10th Asia Pacific Conference on
Computer Human Interaction, ser. APCHI ’12. New York,NY, USA: ACM, 2012, pp. 299–308. [Online]. Available:
http://doi.acm.org/10.1145/2350046.2350098[20] D. Holman, A. Hollatz, A. Banerjee, and R. Vertegaal, “Unifone:
Designing for auxiliary finger input in one-handed mobileinteractions,” in Proceedings of the 7th International Conference on
Tangible, Embedded and Embodied Interaction, ser. TEI ’13. New
York, NY, USA: ACM, 2013, pp. 177–184. [Online]. Available:
http://doi.acm.org/10.1145/2460625.2460653[21] J. Steimle, A. Jordt, and P. Maes, “Flexpad: Highly flexible bending
interactions for projected handheld displays,” in Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems, ser.
CHI ’13. New York, NY, USA: ACM, 2013, pp. 237–246. [Online].
Available: http://doi.acm.org/10.1145/2470654.2470688[22] S. Follmer, D. Leithinger, A. Olwal, N. Cheng, and H. Ishii,“Jamming user interfaces: Programmable particle stiffness
and sensing for malleable and shape-changing devices,” inProceedings of the 25th Annual ACM Symposium on User
Interface Software and Technology, ser. UIST ’12. New York,NY, USA: ACM, 2012, pp. 519–528. [Online]. Available:
http://doi.acm.org/10.1145/2380116.2380181[23] E. Kruijff, S. Biswas, C. Trepkowski, J. Maiero, G. Ghinea, and
W. Stuerzlinger, “Multilayer Haptic Feedback for Pen-Based Tablet

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

Interaction.” in In Proceedings of the ACM conference on Human
Factors in Computing Systems (CHI), 2019.
[24] N. Marquardt, M. A. Nacenta, J. E. Young, S. Carpendale,
S. Greenberg, and E. Sharlin, “The haptic tabletop puck: Tactile
feedback for interactive tabletops,” in Proceedings of the ACM
International Conference on Interactive Tabletops and Surfaces, ser.
ITS ’09. New York, NY, USA: ACM, 2009, pp. 85–92. [Online].
Available: http://doi.acm.org/10.1145/1731903.1731922[25] M. Akamatsu, S. MacKenzie, and T. Hasbroucq, “A comparisonof tactile, auditory, and visual feedback in a pointing task using amouse-type device,” in Ergonomics 38, 1995, pp. 816—-827.
[26] I. Poupyrev and S. Maruyama, “Drawing with feeling: Designing
tactile display for pen,” in ACM SIGGRAPH 2002 Conference
Abstracts and Applications, ser. SIGGRAPH ’02. New York,NY, USA: ACM, 2002, pp. 173–173. [Online]. Available:
http://doi.acm.org/10.1145/1242073.1242186[27] M. Fukumoto and T. Sugimura, “Active click: Tactile feedback
for touch panels,” in CHI ’01 Extended Abstracts on Human
Factors in Computing Systems, ser. CHI EA ’01. New York,NY, USA: ACM, 2001, pp. 121–122. [Online]. Available:
http://doi.acm.org/10.1145/634067.634141[28] S. Khurelbaatar, Y. Nakai, R. Okazaki, V. Yem, and H. Kajimoto,“Tactile presentation to the back of a smartphone withsimultaneous screen operation,” in Proceedings of the 2016 CHI
Conference on Human Factors in Computing Systems, ser. CHI’16. New York, NY, USA: ACM, 2016, pp. 3717–3721. [Online].
Available: http://doi.acm.org/10.1145/2858036.2858099[29] R. Leung, K. MacLean, M. B. Bertelsen, and M. Saubhasik,“Evaluation of haptically augmented touchscreen gui elements
under cognitive load,” in Proceedings of the 9th International
Conference on Multimodal Interfaces, ser. ICMI ’07. New York,NY, USA: ACM, 2007, pp. 374–381. [Online]. Available:
http://doi.acm.org/10.1145/1322192.1322258[30] C. Harrison and S. E. Hudson, “Texture displays: A passive
approach to tactile presentation,” in Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems, ser. CHI’09. New York, NY, USA: ACM, 2009, pp. 2261–2264. [Online].
Available: http://doi.acm.org/10.1145/1518701.1519047[31] S.-C. Kim, A. Israr, and I. Poupyrev, “Tactile rendering of 3dfeatures on touch surfaces,” in Proceedings of the 26th Annual ACM
Symposium on User Interface Software and Technology, ser. UIST ’13.
New York, NY, USA: ACM, 2013, pp. 531–538. [Online]. Available:
http://doi.acm.org/10.1145/2501988.2502020[32] J. Mullenbach, C. Shultz, A. M. Piper, M. Peshkin, and J. E.
Colgate, “Surface haptic interactions with a tpad tablet,” inProceedings of the Adjunct Publication of the 26th Annual ACM
Symposium on User Interface Software and Technology, ser. UIST ’13
Adjunct. New York, NY, USA: ACM, 2013, pp. 7–8. [Online].
Available: http://doi.acm.org/10.1145/2508468.2514929[33] Y. Fukatsu, H. Hakoda, A. Noguchi, B. Shizuki, and J. Tanaka, “A
study of touch accuracy using smartphone cases with a dimple ora wedge shaped attachment,” Computer Software, vol. 31, no. 3, pp.
3325–3335, 2014.
[34] R. Kincaid, “Tactile guides for touch screen controls,” inProceedings of the 26th Annual BCS Interaction Specialist Group
Conference on People and Computers, ser. BCS-HCI ’12. Swinton,
UK, UK: British Computer Society, 2012, pp. 339–344. [Online].
Available: http://dl.acm.org/citation.cfm?id=2377916.2377963[35] T. Guerreiro, J. A. P. Jorge, and D. J. V. Gonçalves, “Exploring thenon-visual acquisition of targets on touch phones and tablets,” in
ACM SIGCHI 2nd Workshop on Mobile Accessibility, Sep. 2012.
[36] Y. Jansen, T. Karrer, and J. Borchers, “Mudpad: Tactile feedback
and haptic texture overlay for touch surfaces,” in ACM
International Conference on Interactive Tabletops and Surfaces, ser.
ITS ’10. New York, NY, USA: ACM, 2010, pp. 11–14. [Online].
Available: http://doi.acm.org/10.1145/1936652.1936655[37] M. Pielot, A. Kazakova, T. Hesselmann, W. Heuten, and S. Boll,“Pocketmenu: Non-visual menus for touch screen devices,” inProceedings of the 14th International Conference on Human-computer
Interaction with Mobile Devices and Services, ser. MobileHCI ’12.
New York, NY, USA: ACM, 2012, pp. 327–330. [Online]. Available:
http://doi.acm.org/10.1145/2371574.2371624[38] Y. Wang, C. Yu, J. Liu, and Y. Shi, “Understanding performanceof eyes-free, absolute position control on touchable mobile
phones,” in Proceedings of the 15th International Conference on
Human-computer Interaction with Mobile Devices and Services, ser.

13

MobileHCI ’13. New York, NY, USA: ACM, 2013, pp. 79–88.
[Online]. Available: http://doi.acm.org/10.1145/2493190.2493215[39] S. K. Kane, C. Jayant, J. O. Wobbrock, and R. E. Ladner,“Freedom to roam: A study of mobile device adoption andaccessibility for people with visual and motor disabilities,”in Proceedings of the 11th International ACM SIGACCESS
Conference on Computers and Accessibility, ser. Assets ’09. New
York, NY, USA: ACM, 2009, pp. 115–122. [Online]. Available:
http://doi.acm.org/10.1145/1639642.1639663[40] N. Yannier, A. Israr, J. F. Lehman, and R. L. Klatzky,“Feelsleeve: Haptic feedback to enhance early reading,” inProceedings of the 33rd Annual ACM Conference on Human
Factors in Computing Systems, ser. CHI ’15. New York,NY, USA: ACM, 2015, pp. 1015–1024. [Online]. Available:
http://doi.acm.org/10.1145/2702123.2702396[41] M. Sinclair, M. Pahud, and H. Benko, “Touchmover: Actuated 3d
touchscreen with haptic feedback,” in Proceedings of the 2013 ACM
International Conference on Interactive Tabletops and Surfaces, ser. ITS’13. New York, NY, USA: ACM, 2013, pp. 287–296. [Online].
Available: http://doi.acm.org/10.1145/2512349.2512805[42] J. Maiero, E. Kruijff, A. Hinkenjann, and G. Ghinea, “Forcetab:
Visuo-haptic interaction with a force-senstive actuated tablet,” inIEEE International Conference on Multimedia and Expo (ICME), 2017,
pp. 1–6.
[43] S. Boring, D. Ledo, X. A. Chen, N. Marquardt, A. Tang,and S. Greenberg, “The fat thumb: Using the thumb’s contact
size for single-handed mobile interaction,” in Proceedings ofthe 14th International Conference on Human-computer Interaction
with Mobile Devices and Services, ser. MobileHCI ’12. New
York, NY, USA: ACM, 2012, pp. 39–48. [Online]. Available:
http://doi.acm.org/10.1145/2371574.2371582[44] S. MacKenzie and W. Soukoreff, “A model of two-thumb text
entry,” in Proceedings of Graphics Interface 2002, ser. GI 2002. University of Waterloo, Waterloo, Ontario, Canada: Canadian HumanComputer Communications Society, 2002, pp. 117–124.
[45] A. Oulasvirta, A. Reichel, W. Li, Y. Zhang, M. Bachynskyi,
K. Vertanen, and P. O. Kristensson, “Improving two-thumb text
entry on touchscreen devices,” in Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems, ser. CHI’13. New York, NY, USA: ACM, 2013, pp. 2765–2774. [Online].
Available: http://doi.acm.org/10.1145/2470654.2481383[46] K. Hasan, J. Kim, D. Ahlström, and P. Irani, “Thumbsup: 3d spatial thumb-reachable space for one-handed thumbinteraction on smartphones,” in Proceedings of the 2016
Symposium on Spatial User Interaction, ser. SUI ’16. New
York, NY, USA: ACM, 2016, pp. 103–106. [Online]. Available:
http://doi.acm.org/10.1145/2983310.2985755[47] H. V. Le, S. Mayer, P. Bader, and N. Henze, “Fingers’ range
and comfortable area for one-handed smartphone interaction
beyond the touchscreen,” in Proceedings of the 2018 CHI Conferenceon Human Factors in Computing Systems, ser. CHI ’18. New
York, NY, USA: ACM, 2018, pp. 31:1–31:12. [Online]. Available:
http://doi.acm.org/10.1145/3173574.3173605[48] J. Deber, B. Araujo, R. Jota, C. Forlines, D. Leigh, S. Sanders,and D. Wigdor, “Hammer Time!: A Low-Cost, High Precision,
High Accuracy Tool to Measure the Latency of Touchscreen
Devices,” in Proceedings of the 2016 CHI Conference on Human
Factors in Computing Systems - CHI ’16. New York, New York,
USA: ACM Press, 2016, pp. 2857–2868. [Online]. Available:
http://dl.acm.org/citation.cfm?doid=2858036.2858394[49] C. Jay and R. Hubbold, “Delayed visual and haptic feedback
in a reciprocal tapping task,” in First Joint Eurohaptics Conference
and Symposium on Haptic Interfaces for Virtual Environment andTeleoperator Systems. World Haptics Conference. IEEE, 2005, pp.
655–656.
[50] T. Kaaresoja, E. Hoggan, and E. Anttila, “Playing with tactile
feedback latency in touchscreen interaction: Two approaches,”in Human-Computer Interaction – INTERACT 2011, P. Campos,
N. Graham, J. Jorge, N. Nunes, P. Palanque, and M. Winckler, Eds.
Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 554–571.
[51] T. Kaaresoja, S. Brewster, and V. Lantz, “Towards
the Temporally Perfect Virtual Button: Touch-Feedback
Simultaneity and Perceived Quality in Mobile Touchscreen
Press Interactions,” ACM Transactions on Applied Perception,
vol. 11, no. 2, pp. 1–25, jun 2014. [Online]. Available:
http://dl.acm.org/citation.cfm?doid=2633908.2611387

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015[52] G. A. Gescheider, Psychophysics: the fundamentals.
Psychology
Press, 2013.
[53] P. Parhi, A. K. Karlson, and B. B. Bederson, “Target size study
for one-handed thumb use on small touchscreen devices,” inProceedings of the 8th Conference on Human-computer Interaction
with Mobile Devices and Services, ser. MobileHCI ’06. New
York, NY, USA: ACM, 2006, pp. 203–210. [Online]. Available:
http://doi.acm.org/10.1145/1152215.1152260[54] A. Bangor, P. Kortum, and J. Miller, “Determining what individualsus scores mean: Adding an adjective rating scale,” J. Usability
Studies, vol. 4, no. 3, pp. 114–123, May 2009. [Online]. Available:
http://dl.acm.org/citation.cfm?id=2835587.2835589[55] P. Lopes and P. Baudisch, “Muscle-propelled force feedback:Bringing force feedback to mobile devices,” in Proceedings of
the SIGCHI Conference on Human Factors in Computing Systems,
ser. CHI ’13. New York, NY, USA: ACM, 2013, pp. 2577–2580.
[Online]. Available: http://doi.acm.org/10.1145/2470654.2481355

Jens Maiero received the Diploma in mathematics from the Stuttgart Technology University ofApplied Sciences, Stuttgart, Germany, in 2006,
the M.Sc. degree in Computer Science from theBonn-Rhein-Sieg University of Applied Sciences(BRSU), Sankt Augustin, Germany, in 2009, andis currently working toward the Ph.D. degree in
Computer Science at Brunel University, London,
U.K., and BRSU.

David Eibich received his Bachelor in Computer Science from the Bonn-Rhein-Sieg University in 2017. He is currently doing the Masters
programm and his research interest are augemented reality and interactive environments.

Ernst Kruijff received the Ph.D. (Hons.) degreefrom Graz University of Technology, Graz, Austria, in 2007.
He is an Interim Professor for computer graphics and interactive environments with the Department of Computer Science, Bonn-Rhein-Sieg
University of Applied Sciences, Sankt Augustin,
Germany, where he is the Head of the 3-D Multisensory Interfaces Group. He has widely presented his work at the main ACM and IEEE conferences. He has coauthored the book 3D User
Interfaces: Theory and Practice (Addison-Wesley, 2004). His researchinterests include the human-factor-driven analysis and design and validation of 3D user interfaces, with a particular focus on multisensory
cues.

14

André Hinkenjann (M11) is a Research Professor for computer graphics and interactive
environments with the Department of Computer Science, Bonn-Rhein-Sieg University ofApplied Sciences (BRSU), Sankt Augustin, Germany, and an Adjunct Professor with the Department of Computer Science, University of
New Brunswick, Fredericton, NB, Canada. He
is the Founding Director of the Institute of Visual Computing, BRSU. His research interests
include high-quality rendering, interactive data
exploration, and high-resolution display systems.
Prof. Hinkenjann is a Member of ACM SIGGRAPH, Eurographics, andGerman GI. He is a regular reviewer for international conferences andjournals on the above mentioned topics.

Wolfgang Stuerzlinger is a leading researcherin spatial and three-dimensional user interfaces.
He got his Doctorate from the Vienna Universityof Technology, was a postdoctoral researcher
at the University of Chapel Hill in North Carolina, and professor at York University in Toronto.
Since 2014, he is a full professor at the Schoolof Interactive Arts + Technology at Simon Fraser
University in Vancouver, Canada.
He has been program chair for 8 scientificevents, has been a member of numerous program committee, and serves on several steering committees.

Hrvoje Benko is a research science manager
at Oculus Research in the field of Human Computer Interaction. Prior to his current role, he
was a researcher at Microsoft Research. Hisinterests span augmented and virtual reality, interactive projection mapping, haptics, new input
form factors and devices, as well as touch andfreehand gestural input.
He has been active in the Human-Computer
Interaction research community and He hasserved on various program committees (CHI,
UIST, SIGGRAPH, ISMAR, ITS/ISS) and was the general chair for ACM
UIST 2014 and the program chair for ACM UIST 2012. He is currently
on the ACM UIST steering committee and on the advisory board for
SmartHaptics. He is also the Information Director and Associate Editor
for ACM TOCHI journal.

Gheorghita Ghinea (M02) received the B.Sc.
and B.Sc. (Hons.) degrees in computer scienceand mathematics and the M.Sc. degree in computer science from the University of the Witwatersrand, Johannesburg, South Africa, in 1993,
1994, and 1996, respectively, and the Ph.D. degree in computer science from the University ofReading, Reading, U.K., in 2000.
He is a Reader with the Department of Computer Science, Brunel University, London, U.K.
He has more than 250 publications in leadinginternational conferences and journals. He has co-edited two books
entitled Digital Multimedia Perception and Design (IGI Global, 2006) andMultiple Sensorial Media Advance and Applications (IGI Global, 2011).
His research interests include the confluence of computer science,
media, and psychology. In particular, his work focuses on the area ofperceptual multimedia quality and building end-to-end communication
systems incorporating user perceptual requirements. Dr. Ghinea is a
Member of the British Computer Society.