Automated Data Gathering and Training Tool
for Personalized "Itchy Nose"
Juyoung Lee

Hui-Shyong Yeo

Thad Starner

Korea Advanced Instituteof Science Technology,
Daejeon, Republic of Korea
ejuyoung@kaist.ac.kr

University of St Andrews,
Scotland, United Kingdom
hsy@st-andrews.ac.uk

Georgia Institute of Technology,
Atlanta, Georgia, United States
thad@gatech.edu

Aaron Quigley

Kai Kunze

Woontack Woo

University of St Andrews,
Scotland, United Kingdom
aquigley@st-andrews.ac.uk

Keio University,
Yokohama, Japan
kai@kmd.keio.ac.jp

Korea Advanced Instituteof Science Technology,
Daejeon, Republic of Korea
wwoo@kaist.ac.kr

ABSTRACTIn “Itchy Nose” we proposed a sensing technique for detecting
finger movements on the nose for supporting subtle and discreet
interaction. It uses the electrooculography sensors embedded in
the frame of a pair of eyeglasses for data gathering and uses
machine-learning technique to classify different gestures. Here we
further propose an automated training and visualization tool for its
classifier. This tool guides the user to make the gesture in proper
timing and records the sensor data. It automatically picks theground truth and trains a machine-learning classifier with it. With
this tool, we can quickly create trained classifier that is
personalized for the user and test various gestures.

CCS CONCEPTS• Human-centered computing → Gestural input;

1INTRODUCTION

Smart eyewears are becoming popular nowadays. These devices
support speech recognition or using the touchpad on the device forgetting inputs from the user. However, these actions can bedisruptive, intrusive or socially unacceptable. Indeed, there aremuch related work on sensing the body gesture and use it as aninput. In particular, "Itchy Nose" [6] is a system that can detect
finger movements on the nose using electrooculography (EOG)
sensor embedded in the frame of the eyeglasses. As mentioned in
the paper, the gesture classifier requires personalized training forachieving better result. However, training takes much effort,
especially on the data gathering and ground truth generation. For
this reason, we propose a GUI based data gathering andautomated ground truth generation and training tool to help andassist the personalized training process.

KEYWORDS
Nose gesture; subtle interaction; EOG; wearable computer; smart
eyeglasses; smart eyewear; Training tool; online classification; 1Figure 1 J!ns Meme and its EOG electrode placement.

2RELATED WORK

There are various research work on detecting finger tapping on thebody, such as the Skinput [3], TapSkin [9], and Earput [7]. Bragi
Dash [2] is a pair of consumer earphones that supports a simple
tapping gesture on the face. However, these techniques rely onrather a strong tapping, which would not be practical for the nose.
Similarly, Bitey [1] explores tooth click gestures for hands-freeinterface control; Palpebrea Superioris [4] explores the design
space of eyelid gestures; Eartouch [5] explores the ear as input
surface, and CheekInput [8] explores the touch gestures on theAH2018, February 7-9, 2018, Seoul, Republic of Korea
cheek as input. All of these works suggest that the human face canprovide a rich medium for interaction with computing.
Finally, there is “Itchy Nose” [6] which use EOG sensor to detect
finger tapping on the nose and can classify to five differentgesture. It used a random decision forest classifier to classify
different gestures with very high accuracy and robustness. Yet,the authors mentioned that it requires personal data to improve theaccuracy.

Figure 2: Three proposed input gestures of Itchy Nose.

3
3.1

SYSTEM
Itchy Nose

The EOG sensors in the J!ns Meme eyeglasses are strategicallyplaced around the nose: two on the nose pads and one on the nose
bridge. We stream raw data from the J!ins Meme over Bluetooth
to a remote computer for real-time processing and classification.
We implemented the system in Python with Pygame forvisualization and Scikit-learn for training machine-learningclassifiers. We use five signals from the J!ns Meme: EOG left,right, horizontal, vertical, and the z-axis of the gyroscope. In ourlast paper “Itchy Nose”, we used 10 features including fivestatistical features. We removed three non-dominant statisticalfeatures and modified standard deviation to the frequency domain.
In result, we extracted 7 features for each signal: (i) root meansquare, (ii) standard deviation of frequency domain, the numberof (iii) positive, (iv) negative peak values (which are high on flickand rub), the numbers of values to cross a (v) positive thresholdand (vi) negative threshold, and (vii) the largest number of values
that exceed these thresholds consecutively. Also, we added thesum of all three axis values of the accelerometer and gyroscope torepresent the average movement. In total, we used 51 features per
signal window. For the classifier, we chose Random decision
forest similar to our previous implementation due to its highaccuracy and fast speed.

3.2

Training Tool

Following is the description of using the proposed automated tool.
When we start using the tool, we have to type all the gesturesname we want to train. For example, if we want to train same fivegestures as in the previous paper, then we should type: left flick,left push, right flick, and right push. Those will be used when thesystem prompts the target gesture on the screen. After typing all
the gestures, the system asks the users to press a button when they
are prepared to perform the gesture. When the button is pressed, it
countdowns three seconds and at the same time it shows the time
2J.Lee et al.
left on the screen and makes a beep sound when the countdown
ends. Then, we have to perform the target gesture within the next
three seconds. As same with start countdown, it also countdowns
to show when the recording ends. After the recording ends, wehave to press a button to confirm that the input was correctly
performed. If not, we can press the back button to do it again.
Above descriptions are the one trial’s procedure, and the systemshows 10 times per gesture in random order.
When all the gesture input trials are ended, it starts the training
process. First, it picked one-second length from three seconds ofthe recording session. It picked the signal by counting the root
mean square value of EOG signal, where it is most significant will
be selected. After picking the true gesture data, it generates a false
data which means there is no gesture. The false data is generated
from the time window between start button pressed and recording
start point, during the first countdown. After picking true and false
data, it extracts features from each window and trains the random
decision forest. When the training ends, it shows each gesture’s
possibility and confidence on screen with bar plot that changescontinuously in real-time. Hence, the user can try the gesture thatwas used in the training session and can distinguish whether theclassifier works appropriately.

Figure 3: A graphical guide of data gathering and online
classification screen after training.

4

CONCLUSIONS

Itchy Nose uses finger movements on the nose as input forinteracting with a wearable computer. It may allow users torespond to notifications quickly without distracting nearby
colleagues. With this tool, we can not only train and test thegesture for each person but also use various custom gesture suchas wiggling or pinching with it.

REFERENCES[1] Ashbrook, D. et al. 2016. Bitey. Proceedings of the 18th International
Conference on Human-Computer Interaction with Mobile Devices and Services- MobileHCI ’16 (New York, New York, USA, 2016), 158–169.
[2] Bragi Dash: https://www.bragi.com/thedash/.
[3] Harrison, C. et al. 2010. Skinput. Proceedings of the 28th international
conference on Human factors in computing systems - CHI ’10 (New York, New
York, USA, 2010), 453.

Automated Data Gathering and Training Tool for Personalized "Itchy Nose"

AH2018, February 7-9, 2018, Seoul, Republic of Korea[4] Jota, R. and Wigdor, D. 2015. Palpebrae superioris: exploring the design spaceof eyelid gestures. Proceedings of Graphics Interface 2015 (Toronto, Ontario,Canada, 2015), 273–280.
[5] Kikuchi, T. et al. 2017. EarTouch. Proceedings of the 19th International
Conference on Human-Computer Interaction with Mobile Devices and Services- MobileHCI ’17 (New York, New York, USA, 2017), 1–6.
[6] Lee, J. et al. 2017. Itchy nose. Proceedings of the 2017 ACM International
Symposium on Wearable Computers - ISWC ’17 (New York, New York, USA,2017), 94–97.
[7] Lissermann, R. et al. 2014. EarPut. Proceedings of the 26th AustralianComputer-Human Interaction Conference on Designing Futures the Future of
Design - OzCHI ’14 (New York, New York, USA, 2014), 300–307.
[8] Yamashita, K. et al. 2017. CheekInput. Proceedings of the 23rd ACM
Symposium on Virtual Reality Software and Technology - VRST ’17 (NewYork, New York, USA, 2017), 1–8.
[9] Zhang, C. et al. 2016. TapSkin. Proceedings of the 2016 ACM on Interactive
Surfaces and Spaces - ISS ’16 (New York, New York, USA, 2016), 13–22.

3