International Conference on Artificial Reality and TelexistenceEurographics Symposium on Virtual Environments (2020)
A. Kulik and M. Sra (Editors)

PosterNovel tap operation on capacitive touch
screen for people with visual impairment
Kenji Funahashi 1 , Hayato Maki2 and Yuji Iwahori3
1,2 Nagoya Institute of Technology, Japan, 3 Chubu University, Japan

Abstract
We propose a novel tap method for a capacitive touch screen, which is called inverse tap (iTap). Because this is operated asfollows: a finger is removed from a screen first, and then put on it, different from normal tap. It enables the people with visual
impairment to find a braille on a touch screen and operate a button, after remembering a button and a function layout. It alsoenables drivers to operate a car audio and an A/C through a touch screen instrument panel after a quick look at it while driving
when they can not close watch on it.
CCS Concepts• Human-centered computing → Human computer interaction (HCI);

1. Introduction
There are two types of touch detection on a touch screen, pressure
sensitive and capacitive types. There is also a 3D Touch function,
which is installed on the Apple iPhone6s to iPhoneXS. A PC and acell phone always have physical buttons such as a keyboard and acall button, but a touch screen device usually does not. However, if
you move your finger to find the place you want to tap while touching a screen, it must be recognized as swipe. It means it is difficult
to operate through a capacitive touch screen without seeing and justafter a quick look at the screen. Note that, pressure sensitive screen
devices and 3D Touch applications can allow these operations. Of
course, research and development for the people with visual impairment are also performed (e.g. [LMH18], [WWJ∗ 09]), howevera simple capacitive touch screen device has already become popular as a smartphone, and also a car instrument panel. In this paper,
we propose a novel tap method for a capacitive touch screen, whichis called inverse tap (iTap). For this, a finger is removed from thescreen first, and then put back on it, different from a normal tap.
This enables the people with visual impairment to find and operate
a button on a touch screen, after remembering a button and a function layout. Of course, it may be necessary for sound and vibration
feedback to be provided to notify completion of the operation. In
addition, this process would also enable drivers to operate a car audio and an air conditioner through a touch screen instrument panel
after a quick look at it while driving when they can not close watch
on it.
2. New tap
First, some physical objects like braille are placed on a touch
screen, and normal tap and swipe operations are disabled. Whenc 2020 The Author(s)Eurographics Proceedings c 2020 The Eurographics Association.

&'()*'+,-.

/012
3-4,56-47!"#!$#!%#Figure 1: Novel iTap: (a) An experiment application, (b) Appearance of an experiment, (c) Next experiment in progress.

the following operations are performed, they are judged as iTap operations.
1. Touch a screen with finger(s)2. Move finger(s) to find a point to operate while touching
3. Release a finger from a screen at the point
4. Touch the same point on a screen within a certain time
Android Studio’s onTouchEvent is used to implement this idea.
There are several types of events, which are called actions, and theactions can be obtained by getActionMasked().
• ACTION_DOWN: one touch when there is no prior touch• ACTION_UP: no touch after one touch• ACTION_POINTER_DOWN: add a touch when touch(es)• ACTION_POINTER_UP: remove a touch when some touches• ACTION_MOVE: move while touch(es)Figure 2 shows a flowchart of iTap operation. The process is divided into two parts: a candidate processing part and a final judgment part. Variables u, d in the flowchart are sets of a touch po-Kenji Funahashi & Hayato Maki & Yuji Iwahori / Novel tap operation on capacitive touch screen for people with visual impairment

Table 1: Experiment result (times)!"#!$%&'()"*+,#-./086+,#-./06.-/#'7086+,#-./02.3'

subject
normal: wrong
normal: correct
iTap: wrong
iTap: correct+,#-./011+,#-./04.5/+,#-./06.-/#'704.5/%?"=>=?*)
E<!%)CC>#?E H$=IA)"*! 9:;%!!<=>"?*)@;*>A)# 9:;%!!<=>"?*)@;*>A)!"! # B;*&<)C&!D=B07011C2709D
64010E
1406F
2509G4115H2729I
5115ave.
2.44
4.78
0.44
8.33/!

G)C>#?E !E)<?*>!"F)"=A07011)"=Figure 2: iTap flowchart

speed, and swing direction. Before starting the experiment, eachsubject was instructed on how to operate iTap, and a training time
was provided to remember the button layout. The subjects werenine Japanese university students who were asked to wear a blindfolds (Figure 1(b)). The experiment procedure for both normal tapand iTap application is as follows:
1. repeat following steps in one minutesition coordinate and time as global variables, and initial values
are −∞ (this means x, y coordinates and time are the minimum values in the program). The inequality (d − u < threshold) indicatesthat the operation is iTap when the distance between the coordinates is less than the threshold value and the difference between
the times is less than the threshold value. Operating iTap continuously in a certain time is recognized as a double iTap. Figure
3 shows a flowchart of the double iTap operation. Variable a is aset of a center coordinate between u, d and average time of u, d as
global variables, and the initial value is also −∞. The inequality(average(u, d) − a < threshold) indicates whether the operation isdouble iTap or single iTap.

• take an operation task (ex. turn-up volume)• execute the task, and report
2. record the number of times of correct and wrong operation
The result of the experiment is shown in Table 1. The average number of wrong iTap operations was about two less than the normal
operations. More than half of the subjects did not make any mistakes in iTap operation. The average number of correct iTap operations was more than the normal operations by about 3.5 operations,and tasks were executed earlier in the second half of the iTap experiment. It is suggested that iTap is easier to operate compared tonormal tap in a blind situation.

3. ExperimentFirst, the screen is divided into 12 button areas by referring to thenumeric keypad of a phone and a calculator, and a physical object
like braille is pasted on the center of each button so that peoplewith visual impairment can find the button without looking at thescreen (Figure 1(a)). When any operation is performed, a vibration
notification is also given. In this experiment, it is assumed that inusing an audio device a subject selects the desired music, adjusts
the volume, and sets the play order for the audio. For an air conditioner controller, it is assumed a subject controls temperature, wind

4. Conclusion
In this research, we proposed a novel tap method for people withvisual impairment that can be operated without looking at a capacitive touch screen, and we confirmed usefulness of iTap in theexperiment. A person can iTap also while supporting their hand byanother finger on a screen. In the future, we would also like to consider new operation methods equivalent to swipe and press & hold.
We are also progressing an experiment while driving with a quick
look at a touch screen instrument panel in a car (Figure 1(c)).
Acknowledgement
This work was supported in part by JSPS KAKENHI Grant Number JP20K11918.

%)*+ =6#&>!"9#567'!(%)*+=6#&>!"9References*/!0*&!1"$#3(4 !
8(9:0!$:5'#;5<!$! -.(4 55! -.(*/!0*&!1"2#3!"#$%"&'!(%)*+,#567'!((%)*+,,

Figure 3: A part of double iTap flowchart[LMH18] L E H. V., M AYER S., H ENZE N.: InfiniTouch: Finger-awareinteraction on fully touch sensitive smartphones. Proc. the 31st Annual ACM Symposium on User Interface Software and Technology (UIST’18). Association for Computing Machinery, New York, NY, USA (2018),
779–792. doi:10.1145/3242587.3242605. 1[WWJ∗ 09] W EISS M., WAGNER J., JANSEN Y., J ENNINGS R.,
K HOSHABEH R., H OLLAN J. D., B ORCHERS J.: SLAP widgets: bridging the gap between virtual and physical controls on tabletops. theSIGCHI Conference on Human Factors in Computing Systems (2009),
481–490. doi:10.1145/1518701.1518779. 1c 2020 The Author(s)Eurographics Proceedings c 2020 The Eurographics Association.