On the Tip of my Tongue – A Non-Invasive Pressure-BasedTongue Interface
Jingyuan Cheng, Ayano Okoso*, Kai Kunze*, Niels Henze**, Albrecht Schmidt**, Paul Lukowicz,
Koichi Kise*
DFKI Kaiserslautern, Germany*Osaka Prefecture University, Japan**Stuttgart University, Germany
jingyuan.cheng@dfki.de, {okoso,kunze}@m.cs.osakafu-u.ac.jp
ABSTRACT

Mobile and wearable devices became pervasive in daily life.
The dominant input techniques for mobile and wearable technology are touch and speech. Both approaches are not appropriate in all settings. Therefore, we propose a novel interface
that is controlled through the tongue. It is based on an array of textile pressure sensors attached to the user’s cheek.
It can be easily integrated into helmets or face masks in a
non-invasive way. In an initial study, we investigate gestures
for tongue-based interface. Six participants repeatedly performed five simple tongue gestures. We show that gestures
can be recognized with 98% accuracy. Based on feedback
from participants, we discuss potential use cases and provide
an outlook on further improvement of the system.
Author KeywordsTongue interface; pressure sensor; user interface; hands-freegestures; mobile HCI; wearable computing

Figure 1. Two ways to attach the tongue interface to the cheek. On theleft tape is used and on the right a flexible fabric that is also used in theconducted study.

ACM Classification Keywords

might have inherent limitations [9] and similarly Starner argues that ”speech interfaces are not the holy grail for wearables” [10]. The recognition rate of current system is still
restrained in noisy settings and social acceptability of using
speech-based interfaces in public seems limited.

I.5.4 Pattern Recognition Applications: Signal processing;
H.5.m. Information Interfaces and Presentation (e.g. HCI):
Miscellaneous
INTRODUCTION

With wearable and mobile devices on the rise, computing gets
more and more integrated into daily lives. A body of work
investigated output techniques for mobile applications. Input techniques to control such devices, however, still remaina major challenge. Traditional devices such as keyboard andmouse are of limited use for mobile users. With the ongoing success of smartphones, touch screens are the dominant
input technique for mobile interaction. As touch interfaces
are typically operated with the hand they do not allow handsfree interaction. Therefore, hands-free input modalities, mostprominently voice has gained momentum [15]. However,Shneiderman argues that speech used as an input techniquePrevious work on assistive devices showed that tongue-based
interfaces are a interesting opportunity for human-computerinteraction [11]. The amount of sensory receptor on thetongue’s surface is comparable to the number of sensory receptor on the hands’ surface. Similarly, the number of brain
cells that is dedicated to control tongue and mouth movement
are approximately the same as the number of cells for controlling our hands [2]. As the tongue is a very flexible muscle,
humans are able to perform highly delicate movements. We
therefore suspect that users are able perform minute, specialized gestures with their tongue, making the tongue a promising alternative for computer interfaces.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise,or republish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. Request permissions from permissions@acm.org.
AH ’14, March 07 - 09 2014, Kobe, Japan.
Copyright c 2014 ACM 978-1-4503-2761-9/14/03...$15.00.
http://dx.doi.org/10.1145/2582051.2582063

Previous work on tongue-based interfaces focused on assistive systems for users with special needs [?]. As these users
often rely on assistive technologies they are more willing touse invasive interfaces that, for example, require attaching
sensors to the tongue. As tongue-based interaction enables
hands-free use we believe that it also offers potential for otheruser groups. In particular, for situations where users cannot use their hands while interacting with the computer, beit while doing maintenance at machines [3], but also while
being in a very crowed metro, riding a motorbike or during
surgical operations. Tongue-based interaction requires, however, an interface that is non-invasive and easy to use.
In this paper we present a novel interface that is controlled
through the tongue (see Figure 1). An array of resistive textile sensors that can easily be integrated into helmets or facemasks is attached to the user’s cheek. In the following we firstprovide an overview about related work. This is followed by adescription the interface we developed to enable non-invasive
tongue-based interaction. Further, we describe the results of astudy that we conducted to explore the use of tongue gestures
for human-computer interaction. We close the paper with ourconclusions and an outlook to future work.
RELATED WORK

A body of work investigated approaches to provide information via the tongue using tactile feedback. Tang and Beebe,
for example, developed an electrotactile display for the roofof the mouth [12]. Similarly, Vuillerme et al. used a tongue
placed tactile output device generating electrotactile stimulation of the tongue to provide biofeedback [13].
Previous work that explored how to use the tongue for input,
focused on interfaces designed for people with special needs.
In particular, inductive or magnetoresistive sensing is used todetermine the pose of the tongue inside the user’s mouth [5,7]. These invasive sensing technologies require to place sensors on the user’s tongue or to place sensors inside the mouth.
Similarly, researchers used optical tracking or piezoelectric
film sensors inside the mouth to facilitate tongue-based input [8, 6]. Researchers even investigated the optimal placement of sensor on the tongue to increase tracking accuracy[14]. Such invasive interfaces can have an enormous potential for users that cannot use common input devices due tomotor impairments or other special needs. While these systems are, thus, well suited for special target groups in specificsituations, we assume that the sensing approaches are to invasive to be used by other user groups.
Invasive tongue interfaces require wearing sensors inside themouth. They therefore make it difficult to speak, chew ormake other use of mouth and tongue. Yousefi et al. investigated less-invasive tongue-based interaction using an externally tracked magnetic tongue stud [16]. Through a trial over
five sessions they found that all performance measures improved over time. Liu et al. presented another step towards
non-invasive tongue-based interaction [4]. They use an optical system to track tongue gestures while the tongue is outside
the user’s mouth. Although this is no longer invasive, social
acceptability might still be limited.
In summary, previous work showed the potential of tonguebased interfaces. Still, truly a non-invasive input technique isneeded to provide a socially acceptable interaction technique.
In the following we describe, to the best of our knowledge, thefirst tongue interface that does neither require augmenting thetongue nor the mouth with an artificial object. We present apressure-based tongue interface that can easily be attached tothe cheeks using a helmet or a mask. The design of the inter-Figure 2. Architecture of the hardware. On the right is the 8 x 8 pressurematrix connected to a Tmote-Sky. While we currently transmit the dataover USB, the board has also wireless communication capabilities.

face is similar to the textile pressure matrix sensor proposedby Zhou et.al. for smart clothing and smart table cloth [17]and is also related to the general trend towards mainstream
smart textile [1]. We, however, developed a textile pressure
matrix with a much a higher resolution to enable a novel application of smart textiles. In addition, we present an initialstudy that investigates tongue gestures using a non-invasive
tongue interface.
PRESSURE-BASED TONGUE INTERFACE

In contrast to previous work, we decided to detect pressure
applied with the tongue on the inside of the cheek using asensor array attached to the outside of the cheek. This has theadvantage of being non-invasive and gives us the possibility
to integrate the interface easily in helmets and face masks. Inthe following, we discuss the hardware design and how wecurrently attach the prototype to the cheeks.
Hardware Design

To determine the pressure generated by the tongue onto thecheek, we designed a small pressure sensing matrix. The basis for this matrix is a fabric material that has high resistivity in general, yet reduces its resistivity if vertical pressure
is applied. When applying two groups of parallel conductive
stripes onto the top and bottom side of the pressure-resistive
material, each crossing point becomes a pressure sensor. By
applying high voltage (setting IO to output ’1’) on one column and low voltage (setting IO to output ’0’) on the rest,

Figure 3. The hardware prototype used for the experiments. The pressure matrix in yellow, connected to the T-Mote Skye in the foreground.

a column of sensors is selected. The voltage is divided by
an external resister and reflects the change of the material’s
local resistance and thus the pressure. After digitalizing thiscolumn with a group of ADCs, this column is set to ’0’ andthe next column to ’1’.
We cut normal ESD bags for the pressure-resistive material,
where the density of mix-in carbon powder grows with pressure and the resistivity drops. We use the Tmote-Sky measuring platform, which provides 8 IO pins, 8 embedded 12bits ADCs and a USB-Serial interface for data communication, forming a 8 × 8 pressure matrix with a sampling rate of
13Hz. The system discretizes the 8 × 8 pressure values fromthe matrix between 0, for low to no pressure, to 255 for highpressure. The overall architecture is shown in Figure 2.
Attachment to the Face

We tested different ways to attach the tongue interface to thecheek. The easiest is to integrate it in a helmet, as the helmet
is steady and provides a hard surface to press against. We didsome test runs with a helmet and the signal is excellent, better
than the data recorded during the experiments. However, wealso envision integrating the tongue interface in masks andsofter fabric. The best results we got so far is using the softand elastic fabric shown in Figure 1 that we used in the study
described in the following.
EXPERIMENTTo determine if the system is able to recognize tongue gestures, we conducted a controlled lab study. 6 participants (2female, 4 male, age 20-35) took part in the study. Each participant wore the hardware prototype on the right cheek, we ask
the participants to press the tongue in the middle of the cheek,
and fixate the tongue interface on top of the pressed position
with a flexible fabric (as seen in Figure 1). After attaching theinterface we performed an initial calibration. We ensured thatinterface provides a low-noise signal and the pressure matrix
can be easily pressed by the tongue through the cheek.
After applying the interface, participants performed 5 gestures with their tongue: swipe up, swipe down, swipe left,swipe right and ”click” (just pressing the tongue against themiddle of the interface). The directions are performed fromthe view point of the user. We initially planned to include a”draw a circle” gesture, yet as 4 of the 6 participants were notable to perform this gesture with their tongue, we exclude thegesture from the analysis. Each participant performs each of
the 5 gesture 10 times, resulting in 300 gestures. In addition
to testing the device, we conducted semi-structured interview
with the participants to discuss the usefulness of the interface
and explore potential application scenarios.
Exemplary ”pressure maps” for the five gestures are shownin Figure 4. The x and y axis are the respective channels,
the diagram is laid out the way a participant would wear thedevice. This means the top of the diagram is towards ”the up
direction” for the user. The 8×8 pressure values are displayedusing a heat map. Blue is shown no pressure and red themaximum pressure. For each of the 8 x 8 pressure values
we calculate the mean and the variance over a 15 samples
long sliding window with and overlap of 10 samples. This

Figure 4. Exemplary pressure maps for the five gestures. From left
to right: First is the ”click” gesture, then ”swipe left”, ”swipe right”,”swipe up”, ”swipe down”. Red signifies a high pressure value, blue
a lower pressure value, the scale is discretized between 0 (displayed asdark blue) and 255 (displayed as dark red).

roughly corresponds to a 0.75 second sliding window with
0.5 seconds overlap. We used k-nearest neighbors algorithm
to recognize if the participant interacts with the device and torecognize the gestures.
RESULTSIn the following we first analyze the performance of the gesture recognition and provide an overview about participants’feedback gathered through post interviews.
Gesture Recognition

We first determined if the system is able recognize if the participants is interacting with the device by applying pressure to
the cheek or not. Using a k-nearest neighbors algorithm (k =3), we can correctly distinguish between usage and non-usage
with 99% accuracy.
Afterwards we again used a k-nearest neighbors algorithm (k= 5) to determine the actual gesture with an 80% frame byframe classification. Applying a majority decision on top ofeach interaction interval detected, we can correctly classify
98% of the 300 gestures. Only 6 gestures were not correctlyclassified. For 2 participants, the click gesture was not correctly classified. In addition, the ”right swipe” gesture was
misclassified as ”click” 4 times. This is also understandable,
as ”left swipe” was difficult to perform easily, as some participants told us.
Interviews

Regarding the interview, all 6 participants considered thistype of interface as useful, especially if one needs to operate acomputer hands-free when wearing gloves, having wet handsor being in a subway with too little space. Participants mainconcern was the bulkiness of the current hardware prototype.
They stated that they would not want to wear it in public. Ifthe size and shape is reduced to a face mask (usually worn inJapan to avoid infections), they could imagine wearing it.
Asked about potential application areas, most participants
stated that the interface might be good for operating navigation systems or music/video applications, while riding a
motorbike or bicycle. One participant found it interesting tocombine a head-mounted display (HMD) with the interface
and would like to accept or make video calls hands-free using
the tongue interface.
The most difficult gestures for the participants to perform are”circle” and ”left to right”, which could already be seen fromthe experimental data. 4 of the 6 participants were not evenable to perform the ”circle” motion correctly and stated thatit’s too difficult for them. For one participant ”up” was themost difficult. Asked for other potential tongue gestures, 3participants wanted to use both cheeks as interaction interface. They thought it would be useful, especially if alreadywearing a helmet or a mask. Potential gestures proposed byparticipants include alternating presses on the two cheeks.
One participant mentioned that the front of the mouth (under and above the lips) might be preferable for gestures, as
it’s easier reachable and more complex gestures like ”circle” might be possible. Two participants raised concerns thatwearing it on just one cheek and interacting with it more often
might make them ”tongue tired”. So far, we have not experienced any problems, as the force needed is very low.
CONCLUSION AND FUTURE WORKIn this paper, we presented our initial work towards implementing an affordable, resistive tongue interface for handsfree human computer interaction. We show a fully functional hardware prototype and evaluate its performance in alab study. In addition, we gather impressions and potential
improvements from the users for future tongue interface systems and application scenarios.
The high recognition rates are encouraging, yet in the controlled lab environment we still take control on how to attach the device. This is crucial to get a good recognition rate.
Problems can also be seen in the pressure plots in Figure 4.
We envision different approaches to ensure high recognition
rates even outside the lab: Users could receive guidance fromthe system when putting on the interface, using a less flexible
material could ensure that the interface can only mounted atthe ”right” cheek position, or an interface that spans a larger
area could be calibrated after putting it on.
As a next step, we plan to evaluate the interface during everyday life. In particular, we are interested in how often the gesture recognition is triggered unintentionally if a user wears it
during a day. Furthermore, we plan to improve the sampling
rate and the precision by adopting the approach presented byZhou et.al. [17]. Thereby we can also reduce the device’s
size and a make it less obtrusive. We believe that the interface is mainly interesting for users that already wear helmets
or masks during work, e.g. surgeons, motorcyclists. Thus, wewill focus on these scenarios for future hardware prototypes.
Acknowledgment

This work was partially supported by the collaborative project
SimpleSkin under contract with the European Commission(#323849) in the FP7 FET Open framework. The support isgratefully acknowledged.
REFERENCES

1. Cheng, J., Lukowicz, P., Henze, N., Schmidt, A., Amft,O., Salvatore, G. A., and Troster, G. Smart textiles:
From niche to mainstream. Pervasive Computing (2013).
2. Coren, S., Ward, L. M., and Enns, J. T. Sensation andperception. John Wiley & Sons, 2004.
3. Kunze, K., Wagner, F., Kartal, E., Kluge, E. M., andLukowicz, P. Does context matter?-a quantitativeevaluation in a real world maintenance scenario. In
Pervasive Computing. 2009.

4. Liu, L., Niu, S., Ren, J., and Zhang, J. Tongible: anon-contact tongue-based interaction technique. In
Conference on Computers and accessibility (2012).
5. Lund, M. E., Christiensen, H. V., Caltenco, H. A.,Lontis, E. R., Bentsen, B., and Andreasen Struijk, L.
Inductive tongue control of powered wheelchairs. In
Conference on Engineering in Medicine and Biology(2010).
6. Monczak, L., Shapiro, D., Borisenko, A., Draghici, O.,
and Bolic, M. Characterization of piezoelectric film
sensors for tongue-computer interface. In Workshop on
Medical Measurements and Applications (2011).
7. Park, H., Gosselin, B., Kiani, M., Lee, H.-M., Kim, J.,
Huo, X., and Ghovanloo, M. A wirelessmagnetoresistive sensing system for an intra-oral
tongue-computer interface. In Conference on Solid-State
Circuits (2012).
8. Saponas, T. S., Kelly, D., Parviz, B. A., and Tan, D. S.
Optically sensing tongue gestures for computer input. In
Symposium on User interface software and technology(2009).
9. Shneiderman, B. The limits of speech recognition.
Communications ACM (2000).
10. Starner, T. E. The role of speech input in wearable
computing. Pervasive Computing (2002).
11. Struijk, L. N. A. An inductive tongue computer interface
for control of computers and assistive devices.
Transactions on Biomedical Engineering (2006).
12. Tang, H., and Beebe, D. J. An ultra-flexible
electrotactile display for the roof mouth. In Conference
on Engineering in Medicine and Biology and Meeting
Biomedical Engineering Soc. (1999).
13. Vuillerme, N., Pinsault, N., Chenu, O., Fleury, A.,
Payan, Y., and Demongeot, J. A wireless embedded
tongue tactile biofeedback system for balance control.
Pervasive and Mobile Computing (2009).
14. Wang, Y. K., Nash, M. P., Pullan, A. J., Kieser, J. A., andRöhrle, O. Model-based identification of motion sensor
placement for tracking retraction and elongation tongue.
Biomechanics and modeling in mechanobiology (2013).
15. Witt, H., Nicolai, T., and Kenn, H. Designing a wearable
user interface for hands-free interaction in maintenanceapplications. In Conference on Pervasive Computing
and Communications (2006).
16. Yousefi, B., Huo, X., Veledar, E., and Ghovanloo, M.
Quantitative and comparative assessment of learning ina tongue-operated computer input device. Transactions
on Information Technology in Biomedicine (2011).
17. Zhou, B., Cheng, J., Sundholm, M., and Lukowicz, P.
From smart clothing to smart table cloth: Design andimplementation of a large scale, textile pressure matrix
sensor. In Conference on Architecture of Computing
Systems (2014).