LipIO: Enabling Lips as both Input and Output Surface
Arata Jingu

University of Chicago
aratajingu@uchicago.edu

Yudai Tanaka

University of Chicago
yudaitanaka@uchicago.edu

Pedro Lopes

University of Chicago
pedrolopes@uchicago.edu

Figure 1: LipIO enables the user’s lips to be used simultaneously as an input and output surface. It supports eyes- and hands-freeinteractions via (a) electrotactile stimulation as output and (b) capacitive touch as input. We demonstrate a range of practicalapplications, such as (c, d) I/O for controlling appliances (e.g., a smart door or an e-bike’s GPS and lights), (e) outputting thestate of an interface (e.g., feeling the output of a guitar tuner rather than seeing it), and (f) rendering a simple game entirely on
the lips (e.g., whack-a-mole).

ABSTRACTCCS CONCEPTS

Abstract. We engineered LipIO, a novel device enabling the lips tobe used simultaneously as an input and output surface. LipIO comprises two overlapping fexible electrode arrays: an outward-facing
array for capacitive touch and a lip-facing array for electrotactile
stimulation. While wearing LipIO, users feel the interface’s state via
lip stimulation and respond by touching their lip with their tongue
or opposing lip. More importantly, LipIO provides co-located tactile feedback that allows users to feel where in the lip they are
touching—this is key to enabling eyes- and hands-free interactions.
Our three studies verifed participants perceived electrotactile output on their lips and subsequently touched the target location with
their tongue with an average accuracy of 93%, while wearing LipIO with fve I/O electrodes with co-located feedback. Finally, wedemonstrate the potential of LipIO in four exemplary applications
that illustrate how it enables new types of eyes- and hands-freemicro-interactions.

• Human-centered computing → Haptic devices.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor proft or commercial advantage and that copies bear this notice and the full citationon the frst page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission
and/or a fee. Request permissions from permissions@acm.org.
CHI ’23, April 23–28, 2023, Hamburg, Germany© 2023 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9421-5/23/04. . . $15.00
https://doi.org/10.1145/3544548.3580775

KEYWORDS
Haptics, lips, tongue, on-skin interface
ACM Reference Format:
Arata Jingu, Yudai Tanaka, and Pedro Lopes. 2023. LipIO: Enabling Lips as
both Input and Output Surface. In Proceedings of the 2023 CHI Conference on
Human Factors in Computing Systems (CHI ’23), April 23–28, 2023, Hamburg,
Germany. ACM, New York, NY, USA, 14 pages. https://doi.org/10.1145/
3544548.3580775

1

INTRODUCTION

Our eyes and hands are the primary way many users interact with
their environment. As such, it is common for users to fnd themselves in many everyday situations where their eyes and/or hands
are occupied and not available to interact with interfaces. To tackle
this, researchers have focused a tremendous amount of efort onengineering eyes-free and/or hands-free interfaces that provide
users with both input and output (I/O) capabilities. Not only dothese interfaces provide users with an alternative method to control
an interface when their eyes and/or hands are occupied, but thesecan also provide a degree of accessibility to those that cannot interact via eyes and/or hands. One popular and mainstream example
are speech interfaces with both I/O channels in the same auditory modality [45]. More recently, researchers have been exploring
new approaches that push this eyes-free I/O beyond the auditorymodality. These eforts are useful since sound-based interfaces are

CHI ’23, April 23–28, 2023, Hamburg, Germany

susceptible to noise [37, 45, 46] and prevent users from hearing
important sounds in their surroundings. These recent approaches
include eyes-free gesture-I/O [23, 50] and eyes-free muscle-I/O[9, 36, 40]. While they realize eyes-free interactions, they rely on
the user’s interaction via their hands. As such, these are unsuitable
for any everyday situation where the users’ hands and eyes areoccupied with a primary task.
As an alternative to the more traditional modalities that canoften be overloaded in everyday situations (as is the case of a user’seyes, hands, or ears), researchers have explored mouth-based interfaces. The most advanced implementations of these take the shapeof intraoral-based devices—they reside inside the user’s mouth,
typically atop the user’s tongue or roof of the mouth. These havebeen used for tongue/teeth input [10, 14, 16, 30, 32, 64] but also for
output, e.g., electrical stimulation on the tongue [24] or electricalstimulation on the upper palate [58]. However, realizing both input
and output inside the oral cavity has proven elusive, and only a
few assistive devices have been demonstrated. Most inspiring to us
is the assistive intraoral device by Tang et al. [16], which combines
an electrotactile array held by a teeth retainer against the roof ofthe mouth (for output) with a 9-button keypad (for input) that is
operated by the user’s tongue. While this device was not usablein everyday tasks, as it is not compatible with eating/speaking(i.e., it is a thick retainer with cables coming out of the mouth), it
nonetheless demonstrated the potential of combining electrotactile
stimulation of the upper palate with tongue input.
We take inspiration from this concept but turn away from theinside of the mouth and, instead, explore converting the user’s
lips as both an input and output surface for eyes- and handsfree interactions. While previous lip-based interactive devices arepromising, most researchers used the lips as either an input [8, 22]or output surface [15, 20, 21, 48, 51, 56, 62]. Instead, we propose
closing the tactile input and output loop for lip-based interactions by co-locating the input and output directly at the user’s lips.
The two principles that we leverage in LipIO are (1) sensitivityof the lips—researchers found that the lips are as sensitive as ourfngers [54,59]—we leverage this tactile sensitivity for repurposing
the lips as an output surface via electrotactile stimulation; and, (2)dexterity of the tongue —the tongue (and also the lips) is an exceptionally dexterous body part, in many ways comparable to ourfngers [1, 2, 57]. With this in mind, we engineered LipIO (Figure1), an I/O device comprised of two fexible sandwiched electrode arrays: an outwards-facing array for capacitive touch and a lip-facing
array for stimulation. Thus, while wearing our device, users can feelthe state of a user interface by means of electrotactile stimulationof their lips (Figure 1a), and they can respond by touching the felt
locations using their tongue (Figure 1b) or the lower lip for coarser
input. Moreover, whenever the user touches their lips to input, they
feel co-located tactile feedback at the location they touched. This
concept, inspired by tactile-transparent gloves [49, 55] is key toenabling eyes- and hands-free interaction.
We conducted three user studies to inform, understand and improve the accuracy of our technical approach to realizing input andoutput via the tongue and the lips. Our main fnding was that participants were able to perceive output on their lips and subsequently
touch the same location with their tongue with an average accuracy

Arata Jingu et al.

of 93%, while wearing a LipIO device with fve I/O electrodes andco-located tactile feedback.
To demonstrate practical applications for our concept, we implemented four applications that depict diferent uses of LipIO in
everyday contexts: outputting the state of an interface via the lips
(i.e., our guitar tuner application), controlling appliances via input & output on the lips (i.e., our application that allows a user tocontrol a smart door, or our e-bike application that allows the user
to switch between controlling the lights or GPS), and even simple
games played entirely on the user’s lips (i.e., our whack-a-molegame).
Finally, we also present a discussion highlighting further interactive uses for LipIO that go beyond eyes/hands-free and explore,
for instance, LipIO as device for accessibility research, as a haptic
interface, or as a supplementary modality.

2

RELATED WORK

While there are several works that realized either eyes- or handsfree, our goal was to explore the extreme case in which both of theserequirements must be satisfed. As such, we focus on approaches
designed specifcally to support eyes- and hands-free interactions.

2.1

Eyes-free and hands-free I/O devices

Sound-based I/O. Sound is the most common modality to realizeboth eyes and hands-free I/O. Sound-based interactions that feature
both input and output have become common in smartphones andsmart speakers [45], which provide micro-interactions for setting
timers, playing music, etc. However, while this is useful in many
circumstances, it is not applicable to all contexts that users might
fnd themselves in. Specifcally, sound-based I/O interferes withexternal sounds, and is harder to use in public settings [37] or in a
multi-party conversation [45, 46]. As such, researchers have turned
to alternative I/O approaches, namely leveraging the user’s body(e.g., gestures, poses) for I/O.
Touch- & gesture-based I/O. One way that researchers haverealized hands- or eyes-free I/O is by leveraging some part of theuser’s body as the I/O interface, i.e., the user can both feel output
and can respond with input through a limb. For instance, Gesture
Output [50] explored this by actuating the user’s fngers via an
actuated smartphone screen; users responded via gestures on thesame screen, i.e., symmetric I/O. Similarly, ThroughHand [23] enabled users with visual impairments to play interactive games by
delivering tactile output on the palm (via a shape display) whileproviding touch input on the back of the hand. These devices allow
users not only to interact eyes-free but even with a high bandwidth(e.g., Gesture Output ofers a full A-Z alphabet). However, theseapproaches also require the user to hold on to a device. As such, researchers have also engineered wearable forms of this concept. For
example, Proprioceptive Interaction [36] leveraged muscle sensing
and electrical muscle stimulation to realize information input andoutput via the same limb. Moreover, MuscleIO [9] and BioSync [40]leveraged simultaneous muscle stimulation and sensing to create
notifcations or share information across two users. While all theseare promising in that they realize eyes-free I/O without the needfor sounds, they all require the user’s hands to operate, which makesthese approaches not hands-free. As such, with hands out of theLipIO: Enabling Lips as both Input and Output Surface

CHI ’23, April 23–28, 2023, Hamburg, Germany

equation, some researchers turned to the next limb in the humanbody that possesses signifcant dexterity to enable I/O interactions:the mouth.

2.2

Mouth-based interactive devices

The mouth and its many organs, such as the tongue and the lips,
have an incredible tactile acuity and dexterity, which makes it asuitable target to realize devices that can handle eyes- & hands-free
input and output. While technically most speech-based interfaces
could also be considered mouth-based I/O, we specifcally focus on
research leveraging non-speech mouth/tongue movements.
Input. Taking advantage of the dexterity of the tongue, researchers have explored tongue-based input. One typical approach is to attach a retainer-like device in the oral cavity thatsenses tongue movements. This includes capacitive touch sensing [16, 30, 32], magnetoresistive sensing [43] and optical sensing[14, 52]. Another approach is to use an external device to measure
tongue movements, such as cameras [34, 41], ultrasound imaging[31], magnetic sensing [18, 19], pressure sensing [6], doppler radars[12], or EMG to measure tongue movements [38, 39, 53, 65]. As analternative technical approach, ChewIt [10] uses a chewing gum-likedevice that is tongue operated.
Output. To take advantage of the high tactile sensitivity of
the lips, researchers have explored lip-based output. For example,
LipNotif delivers tactile notifcations to the lips using ultrasound[20, 21]. Similarly, Shen et al. leveraged this approach for VR feedback [56]. While ultrasonic-based lip output allows for precise
stimulation, it requires users to carry around large haptic displays
comprised of hundreds of small ultrasonic transducers. Alternative
lip actuators have also been explored to reproduce realistic virtual
sensations such as drinking [15], lip contacts [51], and wind on lips
[48, 62]. However, much like ultrasonic arrays, vibration motors,
servo motors, or fans used to implement these devices are also large
and impossible to be mounted to the user’s lips directly.
Mouth-based I/O. Unfortunately, while the combination of
gesture-based eyes-free input with eyes-free output has yielded anumber of devices (e.g., the aforementioned [9, 23, 36, 40, 50]), thesame has not happened to mouth-based devices. In fact, to the bestof our knowledge, there is only one mouth-based I/O device [16].
This is a retainer-like device composed of an intra-oral keyboard
for the tongue (input) and an intra-oral electrotactile stimulator for
the upper palate (output), which enables two-way tactile communication between the system and the user. While this work opened up
possibilities for mouth-based I/O interfaces, the device is still thick,placed inside the oral cavity, and connected to an external device via
cables, which interferes with tasks such as speaking. Furthermore,
it only proposed one notifcation application that uses only tactile
output. In comparison, LipIO advances mouth-based I/O by leveraging a smaller form factor directly on the lips, which is demonstrated
by a range of mobile & eyes- & hands-free applications.

2.3

Tactile acuity of the lips & dexterity of the
tongueLipIO builds on the high tactile acuity of the lips as the output
surface and the high dexterity of the tongue as the input.

Figure 2: Interaction paradigm in LipIO: lips as input andoutput surface to control a target application.
Lips’ tactile sensitivity. Researchers have found that lips areas or more sensitive than fngers in the grating orientation discrimination test [54, 59]. Similarly, lips display a high tactile acuity inelectrotactile stimulation [35].
Tongue’s dexterity. The tongue has a high tactile sensitivity
rivaling that of the fngers [59]. On the motor side, the tongue canform diferent shapes using ten diferent muscles [57]. The tongue
also displays a relatively fast reaction time of around 600ms [11].
None of this is surprising as the tongue plays essential roles inspeaking and eating [2]. In fact, the tongue’s brain "real estate" isone of the largest in the primary sensory and motor cortical areas
[44].

3

OUR APPROACH: LIPS AS AN INPUT AND
OUTPUT SURFACE

Figure 2 depicts the fundamental concept of LipIO: (1) a user controls an external interactive device by using their lips as an input
surface (via touch input on lips, via tongue or opposite lip); and,conversely, (2) the interactive device responds and informs theuser by using their lips as an output surface (via electro-tactile
stimulation of the lips).
As a result, LipIO implements a new example of symmetric I/Ointeraction [50], in which users input and output using a very
similar vocabulary on the same surface. Moreover, by expanding
the notion of symmetric I/O beyond hand gestures and to the lips,
our concept enables creating new types of interfaces, especially
useful in eyes- and hands-busy situations. Yet, this concept does
not need to be limited to eyes- or hands-free and can also be used asa supplement to existing modalities (e.g., LipIO + speech I/O, LipIO+ gestures, etc.), as an accessibility I/O interface, and so forth.
LipIO widgets. To enable interacting with applications via one’s
lips, we designed four simple interactive widgets, depicted in Figure
3, which were inspired by those found in the conventional GUI.
Note that when the user inputs via any of these widgets, they
concurrently receive feedback from LipIO in the form of electrotactile stimulation at the touched location, which allows users to feel
where they are touching. Our four interactive lip widgets are: (1)momentary push button is an input widget, created from a single
I/O electrode pair, which enables users to trigger actions when its
sensing electrode is touched; (2) smack is an input widget, akin to

CHI ’23, April 23–28, 2023, Hamburg, Germany

Figure 3: Main interaction components that enable users tointeract via their lips in LipIO.
an invocation gesture, created from all I/O electrode pairs, whichenables users to activate/deactivate/switch LipIO applications bysmacking their lips together (similar to saying “pa” in English); (3)toggle is an I/O widget, created from two adjacent I/O electrodepairs, which enables switching between two values (by sliding to
the left or right electrode) or reading its state (by touching it, theuser can feel which one is active, left or right); and, fnally, (4) slider
is an I/O widget, created from at least three adjacent I/O electrodepairs, which allows the user to input linear values (by sliding themultiple electrodes in to left or right and stopping at the desired
level) or read its state (by feeling which electrode is the last in thestimulation sequence).
Feedback modes. Any interactions with LipIO can beneft fromtwo distinct electro-tactile stimulation modes: (1) single-point
feedback, in which users feel one electrode at a time (e.g., buttonpress, toggle slide, slider swipe)—as we show in our applications,even just a single point of stimulation can yield expressive feedback as the temporal profle can be controlled, e.g., render patterns;and (2) multi-point feedback, in which users feel two or more
electrodes being stimulated concurrently (e.g., rendering simple
animations on the lips, vibrating all points to confrm a lip-smack
gesture, etc.). We achieve this technically by time-based multiplexing (i.e., switching rapidly between electrodes).
Leveraging “classic” touch-based interactions. If needed,
LipIO-based applications can also leverage two interactive techniques typically found in touch-based interfaces (depicted in Figure4): (a) diferentiating between short (<0.5s) and long (>2s) presses;and (b) allowing users to touch adjacent targets by touching theupper lip with a wider region of the tongue or with the lower lip.
Dual lip interface. Finally, although we limited all our examples
to one single LipIO device on the upper lip (based on fndings from
our Experiment 1, which revealed this is the most ergonomic touch
location for the tongue), specialized applications can even use both

Arata Jingu et al.

Figure 4: Additional interactive techniques added to LipIO:(a) short vs. long press; (b) allowing touching adjacent targets
by touching the upper lip with a wider region of the tongue
or with the lower lip; and (c) using both lips as I/O surfaces.
lips as I/O surfaces by wearing two of our devices, which increases
I/O to 10 points.

4

CONTRIBUTION, BENEFITS, AND
LIMITATIONS

Our key contribution is that we propose, engineer, and evaluate a
new I/O device that enables lip-based interfaces. It also provides a
new technical approach to realizing eyes-free and hands-free I/O,
especially for micro-interactions.
Our system has four benefts: (1) it provides eyes- and hands-freeinterface; (2) it is fexible and fts the curve of the lip; (3) it features
co-located tactile feedback, leading to a usable I/O accuracy (93%);and (4) our device does not drastically interfere with everyday tasks
such as speaking and eating.
Our system is limited in that: (1) as any electrotactile device, it
requires calibration (∼5 minutes); (2) it sometimes requires recalibrating capacitive sensors due to saliva, though this takes only
a couple of seconds; (3) it interferes with some lip-interactions,such as adding lipstick or lip balms; and (4) it requires the user toattach a device to a very visible location. As for the latter, Chenet al. showed the social acceptability of mouth microgestures including "slide tongue along top lip," a key interaction of LipIO, wasrated 5.42 out of 7 [5], which suggests our interaction can be already socially acceptable. Also, one can even adopt transparent
electrotactile actuators for inconspicuous LipIO in the future [25].

5

APPLICATIONS

To illustrate this novel concept, we implemented four applications:(1) an interface to unlock a smart door; (2) a guitar tuner; (3) ane-bike interface; and (4) a lip-based "whack-a-mole" game. Thesefour applications represent examples of applying LipIO to everyday
situations in which users might fnd their eyes and hands occupied
with primary tasks but simultaneously desire to control an interface.

LipIO: Enabling Lips as both Input and Output Surface

5.1

CHI ’23, April 23–28, 2023, Hamburg, Germany

UI metaphors on the lip: unlocking a smart
door while eyes- & hands-busy

In this example, we depict how LipIO’s tactile stimulation allows for
rendering simple animations that create useful interface metaphors.
Here, we depict a user who is listening to music while cleaning—
their hands, ears, and eyes are occupied with these primary tasks.
Figure 5 depicts the whole interaction: (a) a visitor rings the door;(b) LipIO renders the “ringing” via its tactile feedback (vibration at
3 Hz) at the center electrode—this vibration is a typical UI metaphor
for ringing; (c) in response, the user touches the ringing electrodeto (d) stop the bell ringing. Now, to open the door, (e) the user
swipes from left to right—a typical UI metaphor for sliding a door
open. Now, LipIO renders the state of the door, so the user feels as
their friend walks in: (g→i) the stimulation fans out from the centerelectrodes every 700ms—a metaphor for opening; and then, as theirfriend closes the door, (j→l) the opposite animation is rendered tothe electrodes. Finally, (m) locks the door by repeating the swipe
gesture in the opposite direction of their initial unlock.

5.2

Taking advantage of lip-based tactile output:a hands-free & eyes-free guitar tuner

Our second application demonstrates how LipIO takes advantage
of its electro-tactile output of the user’s lips to enable new types ofhands- & eyes-free interactions. In this case, our user is tuning theirguitar while freely moving on stage and keeping eye contact withthe audience—something not possible with existing tuners which
require eye contact. Our LipIO tuner borrows directly from typicaltuners, which depict visually in which direction the string’s frequency deviates from the expected note (up, down, or tuned)—ourimplementation follows this layout to leverage familiarity. Figure
6 depicts this interaction: (a) the user plays while looking at theaudience; the user invokes the tuner using the lip-smack: (b) to do
this, they join their lips together and feel co-located tactile feedback
to confrm this gesture started. Then, they (c) smack the lips (as if
saying the sound "pa"); (d) now, as the user tunes the string with
their hands, (e) LipIO stimulates the leftmost point in response, indicating this string is out of tune, too low; then, (f) LipIO respondsby indicating that the string is now slightly overpitched, which theuser corrects by lowering; (g) fnally, LipIO indicates that the stringis tuned; (h) the user repeats the lip-smack gesture to (i) dismiss
LipIO.

5.3

Multi-page interface: switching between two
applications in an e-bikeWhile we limited our array to fve channels to ensure an I/O
accuracy of over 90% (see our three User Studies), some microinteractions might still require more than fve options. One way
to implement these is by adopting a multi-page layout. To demonstrate this, in our third application, we depict how a toggle interface
switches between two UI pages, each features diferent UI elementsof the user’s e-bike: a GPS navigation page and a gear settings page.
Figure 7 depicts the interaction: (a) the user is biking while wearing
LipIO; (b) the selected UI page is the GPS, informing the user tokeep moving forward, which they feel by means of the stimulating
electrode (the center of the three navigation electrodes). Then, (c)Figure 5: Leveraging animations on lip-based output & input
surfaces, LipIO can use metaphors to represent the state of
the interactive device. Here, we use these metaphors to represent a doorbell, opening/closing of a door, and lock/unlock
of a door.

CHI ’23, April 23–28, 2023, Hamburg, Germany

Arata Jingu et al.

Figure 6: Interacting with a guitar tuner solely via LipIO.
the user switches to the gear application by touching and “fipping”
the toggle with their tongue; (d) now, in the gears page, they feelthat their e-bike is in the lowest setting (least motor assistance);
the user adjusts the gear by (e) touching the current gear and (f)sliding the tongue to the next gear; until, (g) the e-bike is now thehighest gear (maximum motor assistance). Finally, (h) they switch
to the GPS page, and (j) they feel the GPS indicating a left turn.

5.4

Lip-based game interface: playing"whack-a-mole" game

Our fourth application is, conceptually speaking, the most unique
as it utilizes the lips as I/O surfaces to render a gaming experience,
which to the best of our knowledge, has not been explored (theclosest analog to this are proprioceptive-body games using hand
gestures [4, 36]). Most game UIs require users to focus their eyes,
ears, and hands on the game screen. Instead, in this application ouruser is playing a "whack-a-mole" game while searching for theirfriend and holding luggage (i.e., their eyes and hands are busy).

Figure 7: LipIO toggle to switch between two application
pages: a GPS application and a gear shift application.
Figure 8 depicts the entire gameplay: (a) the user is waiting for afriend, holding an umbrella and luggage; (b, c) the user starts the"whack-a-mole" game using the lip-smack gesture; (d) the mole ismoving between electrodes at random time intervals and in random
order. The mole is in the second hole; (e) the mole escapes to thefrst hole; (f) if the user can "whack" the mole with their tongue
before it moves, they receive three consecutive feedback, indicating
success; (g) the mole escapes to the fourth hole; (h) if the user taps
an empty hole, nothing happens; (i) the user can dismiss the gameby repeating the lip-smack gesture, and (j) the game is dismissed.

LipIO: Enabling Lips as both Input and Output Surface

CHI ’23, April 23–28, 2023, Hamburg, Germany

Figure 9: Detail of electrode layers used to realize LipIO.

Figure 8: A "whack-a-mole" game played solely via LipIO.1

6

IMPLEMENTATION

To help readers replicate our design, we provide the necessary details. To accelerate replication, we provide all the frmware andschematics of our implementation2 . LipIO is comprised of three
key components: (1) a fexible sensing electrode array, (2) a fexible electrotactile actuation electrode array, and (3) its electronic
circuitry.

6.1

Flexible electrode array for lips

Printing electrode arrays. Since our user studies (Study#2 andStudy#3) provided participants with their own electrode arrays forthe sake of hygiene and COVID protocol, we sped up the fabrication time of these using conductive inkjet printing, which was
1 This fgure was designed using an image from Flaticon.com
2 https://lab.plopes.org/#LipIO

achieved using a commodity inkjet printer (EPSON PX-S160T) thatprinted a silver nanoparticle-based ink (Mitsubishi NBSIJ-MU01) on
a white opaque coated paper (Mitsubishi NB-RC-3GR120)—inspiredby Tactlets [13]. Instead, to achieve our fnal form factor, whichis partially transparent, we cut copper tape (LOVIMAG) using a
craft cutter (Cricut Explore Air 2)—inspired by DuoSkin [28]. We
chose this approach because we found our copper tape results tolast longer than our screen-printing results (similar to [61, 63]),
and we especially found it more robust to bending. This approachtakes a bit longer than conductive inkjet printing but is better forvisual clarity. Each electrode array is taped to a fexible fat cable via
z-axis conductive tape (3M) and connected to the electrical circuit.
We measured an end-to-end resistance of ∼0.5Ω from electrodes toeach of their path ends.
Stacking electrode arrays. Our two electrode arrays arestacked together, as depicted in Figure 9. To prevent the output
layer and the lip surface from afecting the capacitive sensors, we
inserted an isolating layer of plastic sheet (135�m) between the input layer and the output layer. We placed a temporary tattoo paper(Silhouette) over the sensor to prevent the tongue from directly
touching the sensor and further placed a laser-cut adhesive sheet(Silhouette, 140�m) on top of that to insulate the non-electrodeparts from the tongue and thus block incorrect touch recognition.
Figure 9 (b) shows the path design for the input and output electrodearray. Figure 9 (c) shows the whole device. The resulting stacked
and co-located array is 360�m thin and fexible to ft the curve of
the user’s mouth, as depicted in Figure 9 (d).
Attaching to the lips. Finally, to attach the electrodes to theuser, we add another laser-cut adhesive sheet under the output layer.
We peeled of the protective sheet right before attaching the device
to the lips, leaving only the adhesive. After using the device, theCHI ’23, April 23–28, 2023, Hamburg, Germanyadhesive strength can be restored by re-applying another adhesive
sheet of the same shape.
Limitations. There are always trade-ofs to consider when fabricating thin conductive-based devices. For instance, decreasing
electrode radius beyond what is shown can result in decreased
conductivity and subsequently decreased sensor/actuator response.
Similarly, while we also explored thinner trace widths, this also
resulted in decreased conductivity. As such, after considering these
trade-ofs, we settled on a 4 mm electrode—this allows for a maximum of ∼9 electrodes in one LipIO device.

6.2

Electronics & circuit design

Sensing. To realize our capacitive sensing, we integrated an NXPMPR121 (on-chip capacitive touch controller with 12 independent
channels), which we chose due to its high-dynamic range (measures changes in electrode capacitance ranging from 1 pF to 2000
pF)—since capacitive sensing on the lips was a novel technical territory, this aforded the highest dynamic range to ensure we candetect diferent types of touches, such as lip-to-lip or tongue-to-lip.
Moreover, the MPR121 provides an onboard hysteresis flter anddynamic baseline calibration, which allows us to minimize any false
positives caused by residual saliva left after tongue touches.
Actuation. For electro-tactile stimulation, our system feeds ofan external electrical stimulator. For our studies, we utilized the
medical-compliant Hasomed RehaStim as the source of electrical stimulation, but any other stimulator is interchangeable, suchas smaller devices including Biosync wearable stimulator [40] or[3, 27]. To control which electrode outputs the stimulation, we implemented an array of Sharp PC817XxNSZ1B photo-relays arranged
in a 1:N multiplexer confguration. These photo-relays are rated
for up to 80V at 50mA of current, which is 10x above the typicalcurrent values for tactile stimulation; in fact, in our studies, we
utilized only 0.5mA-10mA (adjusted to the participants’ comfort).
Moreover, our channel multiplexer afords a response time of 4�s,enabling us to switch between channels or even "simultaneously"
stimulating multiple channels by means of time-multiplexing.

6.3

Application-specifc implementations

Guitar tuner was implemented with Pure Data and runs on a laptop that communicates to LipIO via USB. It tracks the current noteusing an FFT (window of 2048 samples at 44.1kHz). The fundamental frequency is extracted and compared to known guitar string
notes. If the diference between the fundamental and the target noteis larger than one half-note away, LipIO stimulates electrodes #1 or#5 (depending on whether it is above or below the target); similarly,
below one half-step, it will stimulate #2 or #4 respectively; fnally,
if the note is less than a quartertone to target, it stimulates #3.
Bike application & Door application were implemented using
Processing and run on a laptop that manages the wireless communication. Moreover, for the bike application, this Processing application communicates to the Android phone (Wi-Fi), LipIO (USB),and to the e-bike interface, which we controlled using 5V relays
and an Arduino (USB).
Whack-a-mole game was implemented as a self-contained application directly in the microcontroller that runs LipIO. We added

Arata Jingu et al.

two game designs specifc to LipIO. First, considering that the reaction time of the tongue is around 600ms [11], we set the timeinterval of the mole’s movement randomly between 800ms and1500ms. Second, imitating the usual whack-a-mole game, this app
recognizes an input when the tongue is pressed to the lips, in contrast to the other applications that detect input when the tongue isreleased from the lips.

7

USER STUDIES

Overview. We conducted three user studies to inform, understandand improve the accuracy of our concept of using the lips as input
and output surfaces. (1) In our frst study, we assessed whichof the lips is easier to touch with one’s tongue; we found that,in about 90.0% of the trials, participants found the upper lip waseasier to touch with their tongue—this allowed us to understandthat the upper lip was, ergonomically, the best target for our investigations, which we used in our subsequent studies. (2) In our
second study, participants felt electrotactile stimuli in their lips
and attempted to touch the felt position with their tongue—thistask was purposefully performed with no-tactile feedback and eyesfree, which is relatively difcult as participants could only rely
on tongue proprioception and vague lip tactile feedback during
tongue touch. This was a necessary step because before enabling
the real-time and co-located tactile feedback (i.e., to feel where thesystem senses the tongue touches), we needed to understand thebasics of tongue-to-lip interactive touches, to which there was noprior work. The raw data of participants’ touches were used toassess which common touch sensing approach (maximum-value
or centroid) was most suited for tongue-to-lip touches. We foundthat a centroid-based approach was better at estimating where thetongue was touching on the lip. We then implemented and integrated this centroid-sensing method in our device to enable users
to feel co-located output at the position they are currently touching.
Finally, (3) in our third study, we investigated the trade-ofs andbenefts of our implementation, namely: number of electrodes andco-located feedback. We found that adding the co-located feedback
improved the participants’ accuracy from an initial average of 41.3%(without co-located feedback) to an accuracy of around 66.3%. Finally, we found that reducing the number of I/O electrodes from
9 to 5 further improved the participants’ average accuracy from
66.3% to an accuracy of 93.1%, which we deemed usable forbuilding our applications.
All our user studies were approved by our Institutional Review
Board (IRB21-1229).

7.1

Study#1: which lip is easier to touch withthe tongue?

Study objective. Prior to designing any of our devices, we conducted a preliminary study to assess which of the lips is easierto reach with one’s tongue—this determined a critical factor, i.e.,
which lip is more suitable for tongue touches, which informed our
subsequent design decisions, implementations, studies #2 & #3 andapplications.
Participants & apparatus. We recruited 8 participants without
any motor impairments on the lips or tongue (M = 24.6 years old,
SD = 2.6; three identifed as female and fve as male). We applied

LipIO: Enabling Lips as both Input and Output Surface

fve equally spaced 2mm x 2mm tape squares on the participants’
upper and lower lips (spacing was relative to each participant’s lip
length).
Trial design & procedure. Participants were asked to touch apair of positions (upper vs. lower) with their tongue (no mirrors,
task was performed eyes-free) and chose which, upper vs. lower lip,
was easiest to touch. Each participant conducted 15 trials (5 pairs
× 3 repetitions) in a randomized order for a total of trials across all
participants.
Results. We found that most trials reported that the upper lip
was easier to touch (average of 90.0%; SD = 9.4). This is in linewith the ergonomics of the tongue since the protrusion movementof the tongue outside of the oral cavity is minimal if the tongue
touches the upper lip. Conversely, the protrusion is longer and moredemanding for touching the lower lip. While this motivated ourdecision to study LipIO on the upper lip, none of our participants
voiced discomfort or inability to touch the lower lip, which canenable future versions of LipIO-type devices on both upper andlower lip.

7.2

Study#2: understanding tongue-to-lip touch(without tactile feedback)Study objective. Since there is no prior work in tongue-to-lip
touches and to develop a real-time interactive system that uses
capacitive touch to detect tongue-to-lip touches, we frst gathered
participants’ tongue touches on diferent lip targets. The goal of thisstudy was to understand how the user touches the lips with their
tongue in an eyes-free manner and to devise methods to achieve
high accuracy of tongue touch. Note that this task was purposefully
performed with no-tactile feedback and eyes-free, which is relatively
difcult as participants could only rely on tongue proprioception
and vague lip tactile feedback during tongue touch.
Apparatus. Participants wore our LipIO device while sitting
down, with their head supported by a chinrest—this ensured thatthe data collection was robust. A keypad was placed near the participants’ dominant hand to confrm their input. As previously described, we used 9 electrodes for this study, which is the maximum
number of electrodes that we can robustly fabricate in our array,with 4 mm circular electrodes, 4.7 mm apart (see Implementation
for details).
Trial design. In a single trial, participants felt tactile stimulationin one of the nine locations. Then, they were asked to, eyes-free(no mirrors in sight), touch the point they felt with their tongue(in any way they preferred) and press a button to confrm thelocation. Once they pressed this button, our apparatus recorded a
photograph of their face (with their tongue sticking out at theirchosen touch location) and recorded the capacitive sensor values for
all the sensing electrodes obtained from the touch sensor controller.
Study procedure. Participants performed 36 trials (9 points x 4
repetitions). Locations were presented in randomized order. This
totaled 288 trials across all participants (each trial is comprised of
9 data points from the touch sensors and one image of the participant’s tongue touching the sensor array, for a total of 2592 touch
values and 288 images).

CHI ’23, April 23–28, 2023, Hamburg, Germany

Figure 10: Comparison of the root mean square error between
centroid estimator and maximum estimator.
Calibration. Prior to the start of the trials, the intensity of theelectro-tactile stimulation was calibrated for all nine points to ensure pain-free operation. The intensity ranged from a minimum of
0.5mA up to the value that the participant deemed clearly noticeable
and comfortable, which is the typical calibration threshold fromelectro-tactile study designs (e.g., [26] or [24]). Moreover, sub-mA
adjustments were also conducted, when necessary, by adjusting thestimulation pulse-width from 50�s up to 300�s (in steps of 50�s).
After calibration, we registered an average intensity of 2.3mA (SD=0.9mA; median = 2 mA) with a pulse-width of 223�s (SD = 73�s;
median = 200�s).
Lip images. We used a 1080p camera that gathered images ofparticipants’ lips with their consent. Our camera feed was corrected
for distortions via a fsheye lens calibration software using a printed
checkerboard pattern.
Participants. We recruited eight new participants (M = 22.9
years old, SD = 2.2; fve identifed as male, three as female). Participants were compensated 30 USD and consented to the photographing of their faces.
Results. Figure 10 depicts our main fndings by measuring theroot mean square error (RMSE) for two possible touch estimators: inpink, a maximum-value estimator (i.e., an estimator that outputs as
the touched location the highest value sensor reported from all theelectrodes) and in blue, a centroid estimator (i.e., an estimator that
outputs as the touched location the centroid of all the sensor values).
We found that the error was lower for the centroid estimator(M = 0.95; SD = 0.15) when compared to the maximum estimator(M = 1.27; SD = 0.29). These error values were calculated using
the electrode index (1-9). To exemplify, a value of “1.0” means thatthe stimulating electrode and the touched electrode are of by one
electrode. Our results suggested that, similar to early touchscreens,a centroid-based approach performed better than maximum-value
for tongue-to-lip touches.
The photographs taken from participants’ lips provided some
additional evidence that the maximum-value estimator led to moreerror than the centroid estimator. Figure 11 depicts examples (oneper participant) in which the correct target (annotated in blue)
was aimed for. In these examples, it appears that the tip of the
tongue (annotated in pink) was the closest anatomical feature to
the target electrode. However, while the tip is touching the target
electrode, the surrounding tongue areas are also touching adjacent

CHI ’23, April 23–28, 2023, Hamburg, Germany

Arata Jingu et al.

Figure 12: Comparison of the performance of the three methods (a) by accuracy (b) by root mean square error.

7.3

Figure 11: Exemplary trials of participants touching suggestthat participants might aim with the tip of the tongue (photos
with participants’ consent; all participants depicted).
electrodes—this happens because our tongue is wider than theelectrodes (similar to the well-known "fat fnger" problem [60]).
This is where the centroid estimator outperforms the maximumvalue estimator: since the tongue lands fairly symmetrical during
lip touches, the tip tends to be found closest to the center of the
raw touch data, which the centroid estimates better.
Discussion. We found the RMSE to be around an average of 0.95
(SD = 0.15), which suggests participants tended to be of by about
one target in these trials—again, we remind the reader that thiswas purposefully a difcult task as participants received no tactilefeedback while targeting, i.e., they could only judge the location they
were touching based on tongue proprioception and vague tactile
feedback (as the tongue presses on the fexible tactile array, thetouch pressure is not focal but blurred over the whole array). Thisis precisely what we will address by adding our co-located tactile
feedback that renders which point the user’s tongue is currently
touching. Yet, before we could enable this real-time tactile feedback,
we needed to collect raw data of participants’ touches to understandwhat sensing approach to use, which as aforementioned was foundto be the centroid. Moreover, in these extreme trials (eyes-free
and almost tactile-free), we measured, on average, low accuracy of
41.3% (SD=7.7), which allows us to advance two possible strategies
to improve the accuracy in our subsequent study: (1) adding theco-located tactile feedback to indicate which location the tongue iscurrently touching; and, since the RMSE suggests that participantstended to be of by one target, (2) reducing the number of available
I/O locations by skipping every second location, i.e., decreasingfrom 9 points to 5 points.

Study#3: measuring & improving theaccuracy of real-time LipIO touches

Study Objective. Now, armed with our centroid-based estimator,
we could measure the performance of LipIO in real-time. Now we
could provide co-located tactile feedback anytime the lips weretouched. This study was designed to allow us to understand the impact of our two aforementioned strategies to improve the accuracy
found in Study 2: (1) what is the impact in accuracy provided by theco-located tactile feedback? and (2) what is the impact in accuracy
provided by reducing 9 touch locations to 5 touch locations?
Apparatus. We utilized the same apparatus as in our previous study, except this time participants received co-located tactile
feedback. This feedback was determined in real-time using thepreviously validated centroid estimator.
Trial design. We followed the same trial design from our Study
2 (i.e., participants felt a tactile stimulation in one of the possible
locations and were asked to touch the point they felt with their
tongue, confrming it with a button press).
Conditions. We used our 9-point electrode array from Study 2
and an additional 5-point electrode array to investigate the impactof the reduction of touch points.
Study procedure & calibration. We followed the same procedure & calibration from Study 2. After calibration, we registeredan average intensity of 3.1mA (SD = 1.8mA; median = 2 mA) witha pulse-width of 145�s (SD = 92�s; median = 100�s). These calibration values were similar to those of Study 2, with the medianintensity at the exact same level and the median pulse-width 100�slower—this suggests that individual calibrations were consistent.
Participants. To directly compare the improvement over thebaseline recorded in the previous study, we re-invited the same
participants from Study 2, and they were additionally compensatedwith 30 USD for their participation.
Results. 12 depicts our main fndings: First, we found an accuracy of 66.3% (SD = 14.9) when using touch feedback with the
9-point array, as depicted in 12 (a). Compared to the baseline of theprevious study (41.3%), in which participants did not receive any
tactile feedback, this suggests that touch feedback increased theperformance by 25.0%. Moreover, as depicted in 12 (b), the RMSE
also decreased from 0.95 (SD = 0.15) to 0.59 (SD = 0.17); yet, this

LipIO: Enabling Lips as both Input and Output Surface

CHI ’23, April 23–28, 2023, Hamburg, Germany

Figure 13: Three distinct types of envisioned practical applications for LipIO that go beyond eyes- and hands-free interactions.
is still comparatively high, which still suggests that participants
often are one electrode away from the target. As such, we turn toanalyze our 5-point array. We found that the RMSE also decreased
from 0.59 to 0.17 (SD = 0.20). More importantly, we found that the
reduction of electrodes further increased the performance by 26.8%,
resulting in a fnal average accuracy of 93.1% (SD = 11.2).

7.4

Study conclusions

Taken together, our results suggest that we reached an accuracyof around 93% for our most important touch-based interactions
with LipIO, i.e., tongue-to-lip touches. Moreover, we observed that
adding the co-located tactile feedback is important for this type ofdevice, as it increased the accuracy by ∼25%. Our results are particularly exciting in that our participants had no training in using
LipIO, except for a few calibration trials to adjust the intensity ofthe electrotactile sensations—as such, we can safely envision thatgiven additional hours of using this device, a prospective user could
perform even better and might not need the reduction from 9 to
5 electrodes—potentially other spacings and compromises can be
explored by researchers building on our fndings. Regardless, we
believe that 93% for untrained individuals provided validation to
enable us to explore a range of applications with our device.

8

FURTHER PRACTICAL APPLICATIONS

While we implemented four applications that demonstrated how
LipIO supports interactions while the user’s eyes & hands are occupied with other main tasks, we believe that these represent only oneof many interactive domains for LipIO. In this section, we envision
how future researchers might apply our concept to further practicalapplications, which we depict in Figure 13.

In particular, we highlight three use cases that go beyond eyesand hands-free interactions: (1) LipIO as a potential device foraccessibility research, (2) LipIO for enhancing realism; and (3) LipIO
as a supplementary modality.
1. Potential for accessibility. Like many other aforementioned
tongue/oral interfaces [17, 18, 29], LipIO might be applied to the research in accessibility. In this domain, LipIO presents two promisingadvantages: (1) Unlike existing intraoral interfaces, which typicallyrequire attaching magnetic trackers to the tip of the tongue (typically via adhesives or piercings), LipIO is applied using skin-safe
adhesives that are simpler to attach/remove. (2) The vast majority ofthe prior work on tongue/oral interfaces for accessibility is limited
to input-only; instead, LipIO presents users with an input surface
that can also render collocated output. We depict an envisionmentof this in Figure 13 (a), in which a user controls their motorized
wheelchair via LipIO. Furthermore, tactile encoding of informationis also possible. Figure 13 (b) depicts how LipIO can be extended,
for instance, to render braille to lips—this is inspired by [33, 42, 66],but, instead, depicts how users might be able to read braille via
their lips with LipIO. With regards to these envisioned applications,
it is important to note that LipIO will not, by itself, solve any inherent challenge faced by users with disabilities, not only because
it is an emergent interface that has not been yet explored in theaccessible domain, but, more importantly, because no technologicalsolution should “just solve” the unique experience of these individuals. Rather, we are inspired by the potential uses that LipIO might
have in accessibility research.
2. Adding realism to lip-based interactions. A diferent use
for LipIO is as a haptic interface to increase the realism of virtualinteractions. For instance, Figure 13 (c) depicts an envisioned use of

CHI ’23, April 23–28, 2023, Hamburg, Germany

LipIO to render a more realistic sense of tasting virtual ice cream—
this is inspired by prior work that applied electrical stimulation of
the tongue to render virtual basic tastes (e.g., sour, salty, bitter, andsweet sensations as in [47]). While the existing methods to deliver
these tongue stimulations require cumbersome electrodes directly
clipped to the tongue (e.g., [47]), LipIO can leverage its vantage
point to stimulate the tongue by momentarily switching the sensor,
via a multiplexer, to act as another electrotactile actuator whenever
the user licks the sensor.
3. Supplementing existing modalities. Finally, LipIO can also
work as a supplementary modality that enriches an existing interaction. For instance, Figure 13 (d) envisions LipIO as a supplementary
modality for interacting with a touchscreen; in this interaction,
sliding the tongue controls the contextual pop-up menu of options
to the user’s currently touched location. Similarly, Figure 13 (e) envisions LipIO as a supplementary modality to control the volume ofthis videogame experience without the need to invoke sub-menus
or use a remote controller. Finally, Figure 13 (f) depicts a diferent usage of LipIO as an additional input for a DJ application—asituation in which users always need more inputs since they are
typically undertaking several simultaneous tasks. In this envisioned
example, the DJ uses LipIO to control the amount of “echo” while
simultaneously adjusting the tempo of a secondary song they areabout to mix in.

9

FUTURE WORK

Finally, we highlight how future researchers might extend LipIO
by (1) expanding its fabrication or via (2) further studies.
Further exploration of device fabrication. It is possible toexpand LipIO by using fabrication techniques that might reduce
the hardware footprint. One such technique is to switch to substrates that allow for thinner printed electrodes (e.g., [63]). Anotherapproach is to reduce the device to a single layer and temporally
multiplex the electrodes as either inputs or outputs (e.g., [40]). Moreover, it is possible to explore directly applying conductive paintto the skin (e.g., [7]). Finally, further size reduction is possible byfabricating a customized printed circuit board with a wireless unit.
Future studies. As the frst step in this territory of closed-loop
lip interactions, our work focused on proposing, engineering, andevaluating the technical feasibility to facilitate future research. Oncethe device fabrication of devices inspired by LipIO matures, further
research is required to evaluate the usability of future versions(e.g., their comfort or cognitive load during use [5]) or even toevaluate social aspects, such as the social acceptability of future
instantiations.

10

CONCLUSION

We proposed, engineered, and evaluated LipIO—a novel eyes- &hands-free interactive device that enables the user’s lips to be usedsimultaneously as an input and output surface. LipIO is built bystacking two fexible electrode arrays: an outwards-facing array forcapacitive touch and a lip-facing array for stimulation. Thus, while
wearing LipIO, users can feel the state of a user interface by means ofelectrotactile stimulation of their lips, and they can respond by also
touching their lips using their tongue (or also lips for coarser input).
Moreover, to facilitate interactions, whenever the user touches their

Arata Jingu et al.

lips to input, they feel co-located tactile feedback at the location
they touched. Using LipIO, we demonstrated a range of eyes- andhands-free micro-interactions, in which users leverage their lips
and tongue to, for instance, unlock a smart door while cleaning,tune a guitar while freely moving on stage, switch between two
applications in their e-bike, or play whack-a-mole while holding
luggage with both hands.

ACKNOWLEDGMENTS
This material was supported by NSF grants 2047189. Any opinions,fndings, and conclusions or recommendations expressed in thismaterial are those of the author(s) and do not necessarily refectthe views of any funding agencies.

REFERENCES[1] Michael Barnett-Cowan, Matin Soeizi, and Joseph F. X. DeSouza. 2015. Visual
attention at the tip of the tongue. i-Perception 6, 1: 1–4. https://doi.org/10.1068/
i0697sas[2] Bruno Bordoni, Bruno Morabito, Roberto Mitrano, Marta Simonelli, and Anastasia
Toccafondi. The Anatomical Relationships of the Tongue with the Body System.
Cureus 10, 12: e3695. https://doi.org/10.7759/cureus.3695[3] Jas Brooks, Shan-Yuan Teng, Jingxuan Wen, Romain Nith, Jun Nishida, and Pedro
Lopes. 2021. Stereo-Smell via Electrical Trigeminal Stimulation. In Proceedings ofthe 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21), 1–13.
https://doi.org/10.1145/3411764.3445300[4] Richard Byrne, Joe Marshall, and Florian “Floyd” Mueller. 2016. Balance Ninja:
Towards the Design of Digital Vertigo Games via Galvanic Vestibular Stimulation.
In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction inPlay (CHI PLAY ’16), 159–170. https://doi.org/10.1145/2967934.2968080[5] Victor Chen, Xuhai Xu, Richard Li, Yuanchun Shi, Shwetak Patel, and Yuntao Wang. 2021. Understanding the Design Space of Mouth Microgestures.
In Designing Interactive Systems Conference 2021 (DIS ’21), 1068–1081. https://doi.org/10.1145/3461778.3462004[6] Jingyuan Cheng, Ayano Okoso, Kai Kunze, Niels Henze, Albrecht Schmidt, Paul
Lukowicz, and Koichi Kise. 2014. On the tip of my tongue: a non-invasive pressurebased tongue interface. In Proceedings of the 5th Augmented Human International
Conference (AH ’14), 1–4. https://doi.org/10.1145/2582051.2582063[7] Youngkyung Choi, Neung Ryu, Myung Jin Kim, Artem Dementyev, and Andrea
Bianchi. 2020. BodyPrinter: Fabricating Circuits Directly on the Skin at Arbitrary
Locations Using a Wearable Compact Plotter. In Proceedings of the 33rd Annual
ACM Symposium on User Interface Software and Technology, 554–564. https://doi.
org/10.1145/3379337.3415840[8] Piotr Dalka and Andrzej Czyżewski. 2010. Human-Computer Interface Based on
Visual Lip Movement and Gesture Recognition. International Journal of Computing Science and Mathematics 7: 124–139.
[9] Tim Duente, Justin Schulte, Max Pfeifer, and Michael Rohs. 2018. MuscleIO:
Muscle-Based Input and Output for Casual Notifcations. Proceedings of the ACM
on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 2: 1–21. https://doi.org/10.1145/3214267[10] Pablo Gallego Cascón, Denys J.C. Matthies, Sachith Muthukumarana, and Suranga
Nanayakkara. 2019. ChewIt. An Intraoral Interface for Discreet Interactions. In
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems,
1–13. https://doi.org/10.1145/3290605.3300556[11] M. Gentil and C. L. Tournier. 1998. Diferences in fne control of forces generated
by the tongue, lips and fngers in humans. Archives of Oral Biology 43, 7: 517–523.
https://doi.org/10.1016/s0003-9969(98)00042-9[12] Mayank Goel, Chen Zhao, Ruth Vinisha, and Shwetak N. Patel. 2015. Tonguein-Cheek: Using Wireless Signals to Enable Non-Intrusive and Flexible Facial
Gestures Detection. In Proceedings of the 33rd Annual ACM Conference on Human
Factors in Computing Systems, 255–258. https://doi.org/10.1145/2702123.2702591[13] Daniel Groeger, Martin Feick, Anusha Withana, and Jürgen Steimle. 2019. Tactlets:
Adding Tactile Feedback to 3D Objects Using Custom Printed Controls. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software andTechnology (UIST ’19), 923–936. https://doi.org/10.1145/3332165.3347937[14] Takuma Hashimoto, Suzanne Low, Koji Fujita, Risa Usumi, Hiroshi Yanagihara,
Chihiro Takahashi, Maki Sugimoto, and Yuta Sugiura. 2018. TongueInput: Input
Method by Tongue Gestures Using Optical Sensors Embedded in Mouthpiece. In
2018 57th Annual Conference of the Society of Instrument and Control Engineers ofJapan (SICE), 1219–1224. https://doi.org/10.23919/SICE.2018.8492690[15] Yuki Hashimoto, Naohisa Nagaya, Minoru Kojima, Satoru Miyajima, Junichiro
Ohtaki, Akio Yamamoto, Tomoyasu Mitani, and Masahiko Inami. 2006. Strawlike user interface: virtual experience of the sensation of drinking using a straw.

LipIO: Enabling Lips as both Input and Output Surface

In Proceedings of the 2006 ACM SIGCHI international conference on Advancesin computer entertainment technology (ACE ’06), 50-es. https://doi.org/10.1145/
1178823.1178882[16] Hui Tang and D.J. Beebe. 2006. An oral tactile interface for blind navigation. IEEE
Transactions on Neural Systems and Rehabilitation Engineering 14, 1: 116–123.
https://doi.org/10.1109/TNSRE.2005.862696[17] Xueliang Huo, Chihwen Cheng, and Maysam Ghovanloo. 2009. Evaluation of
the Tongue Drive System by Individuals with High-Level Spinal Cord Injury.
Conference proceedings: ... Annual International Conference of the IEEE Engineeringin Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society.
Annual Conference 2009: 555–558. https://doi.org/10.1109/IEMBS.2009.5334555[18] Xueliang Huo, Jia Wang, and Maysam Ghovanloo. 2007. A Magnetic Wireless
Tongue-Computer Interface. In 2007 3rd International IEEE/EMBS Conference on
Neural Engineering, 322–326. https://doi.org/10.1109/CNE.2007.369676[19] Xueliang Huo, Jia Wang, and Maysam Ghovanloo. 2007. A Wireless TongueComputer Interface Using Stereo Diferential Magnetic Field Measurement. In
2007 29th Annual International Conference of the IEEE Engineering in Medicine
and Biology Society, 5723–5726. https://doi.org/10.1109/IEMBS.2007.4353646[20] Arata Jingu, Masahiro Fujiwara, Yasutoshi Makino, and Hiroyuki Shinoda. 2021.
Tactile Perception Characteristics of Lips Stimulated by Airborne Ultrasound.
In 2021 IEEE World Haptics Conference (WHC), 607–612. https://doi.org/10.1109/
WHC49131.2021.9517161[21] Arata Jingu, Takaaki Kamigaki, Masahiro Fujiwara, Yasutoshi Makino, andHiroyuki Shinoda. 2021. LipNotif: Use of Lips as a Non-Contact Tactile Notifcation Interface Based on Ultrasonic Tactile Presentation. In The 34th Annual ACM Symposium on User Interface Software and Technology, 13–23. https://doi.org/10.1145/3472749.3474732[22] Marcelo Archajo José and Roseli de Deus Lopes. 2015. Human–Computer Interface Controlled by the Lip. IEEE Journal of Biomedical and Health Informatics 19,
1: 302–308. https://doi.org/10.1109/JBHI.2014.2305103[23] Jingun Jung, Sunmin Son, Sangyoon Lee, Yeonsu Kim, and Geehyuk Lee. 2021.
ThroughHand: 2D Tactile Interaction to Simultaneously Recognize and Touch
Multiple Objects. In Proceedings of the 2021 CHI Conference on Human Factors in
Computing Systems (CHI ’21), 1–13. https://doi.org/10.1145/3411764.3445530[24] K. A. Kaczmarek. 2011. The tongue display unit (TDU) for electrotactile spatiotemporal pattern presentation. Scientia Iranica 18, 6: 1476–1485. https://doi.
org/10.1016/j.scient.2011.08.020[25] Hiroyuki Kajimoto. 2012. Skeletouch: transparent electro-tactile display formobile surfaces. In SIGGRAPH Asia 2012 Emerging Technologies (SA ’12), 1–3.
https://doi.org/10.1145/2407707.2407728[26] Hiroyuki Kajimoto. 2016. Electro-tactile Display: Principle and Hardware. In
Pervasive Haptics: Science, Design, and Application, Hiroyuki Kajimoto, Satoshi
Saga and Masashi Konyo (eds.). Springer Japan, Tokyo, 79–96. https://doi.org/10.
1007/978-4-431-55772-2_5[27] Hiroyuki Kajimoto. 2021. Electro-Tactile Display Kit for Fingertip*. In 2021 IEEE
World Haptics Conference (WHC), 587–587. https://doi.org/10.1109/WHC49131.
2021.9517192[28] Hsin-Liu (Cindy) Kao, Christian Holz, Asta Roseway, Andres Calvo, and Chris
Schmandt. 2016. DuoSkin: rapidly prototyping on-skin user interfaces using skinfriendly materials. In Proceedings of the 2016 ACM International Symposium on
Wearable Computers (ISWC ’16), 16–23. https://doi.org/10.1145/2971763.2971777[29] Jeonghee Kim, Hangue Park, Joy Bruce, Erica Sutton, Diane Rowles, Deborah
Pucci, Jaimee Holbrook, Julia Minocha, Beatrice Nardone, Dennis West, Anne
Laumann, Eliot Roth, Mike Jones, Emir Veledar, and Maysam Ghovanloo. 2013.
The Tongue Enables Computer and Wheelchair Control for People with Spinal
Cord Injury. Science translational medicine 5, 213: 213ra166. https://doi.org/10.
1126/scitranslmed.3006296[30] Naoki Kimura, Tan Gemicioglu, Jonathan Womack, Richard Li, Yuhui Zhao,
Abdelkareem Bedri, Alex Olwal, Jun Rekimoto, and Thad Starner. 2021. Mobile,Hands-free, Silent Speech Texting Using SilentSpeller. In Extended Abstracts ofthe 2021 CHI Conference on Human Factors in Computing Systems, 1–5. https://doi.org/10.1145/3411763.3451552[31] Naoki Kimura, Michinari Kono, and Jun Rekimoto. 2019. SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks. In
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems,
1–11. https://doi.org/10.1145/3290605.3300376[32] Richard Li, Jason Wu, and Thad Starner. 2019. TongueBoard: An Oral Interface
for Subtle Input. In Proceedings of the 10th Augmented Human International
Conference 2019 (AH2019), 1–9. https://doi.org/10.1145/3311823.3311831[33] Weikang Lin, Dongsheng Zhang, Wang Wei Lee, Xuelong Li, Ying Hong, Qiqi
Pan, Ruirui Zhang, Guoxiang Peng, Hong Z. Tan, Zhengyou Zhang, Lei Wei, andZhengbao Yang. 2022. Super-resolution wearable electrotactile rendering system.
Science Advances 8, 36: eabp8738. https://doi.org/10.1126/sciadv.abp8738[34] Li Liu, Shuo Niu, Jingjing Ren, and Jingyuan Zhang. 2012. Tongible: a non-contact
tongue-based interaction technique. In Proceedings of the 14th international ACM
SIGACCESS conference on Computers and accessibility - ASSETS ’12, 233. https://doi.org/10.1145/2384916.2384969

CHI ’23, April 23–28, 2023, Hamburg, Germany[35] Wei Liu and Hui Tang. 2005. An initial study on lip perception of electrotactile
array stimulation. The Journal of Rehabilitation Research and Development 42, 5:705. https://doi.org/10.1682/JRRD.2005.02.0051[36] Pedro Lopes, Alexandra Ion, Willi Mueller, Daniel Hofmann, Patrik Jonell, andPatrick Baudisch. 2015. Proprioceptive Interaction. In Proceedings of the 33rd
Annual ACM Conference on Human Factors in Computing Systems (CHI ’15),
939–948. https://doi.org/10.1145/2702123.2702461[37] Ewa Luger and Abigail Sellen. 2016. “Like Having a Really Bad PA”: The Gulf
between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI
’16), 5286–5297. https://doi.org/10.1145/2858036.2858288[38] Takuro Nakao, Yun Suen Pai, Megumi Isogai, Hideaki Kimata, and Kai Kunze.
2018. Make-a-face: a hands-free, non-intrusive device for tongue/mouth/cheek
input using EMG. In ACM SIGGRAPH 2018 Posters (SIGGRAPH ’18), 1–2. https://doi.org/10.1145/3230744.3230784[39] Phuc Nguyen, Nam Bui, Anh Nguyen, Hoang Truong, Abhijit Suresh, Matt
Whitlock, Duy Pham, Thang Dinh, and Tam Vu. 2018. TYTH-Typing On Your
Teeth: Tongue-Teeth Localization for Human-Computer Interface. In Proceedingsof the 16th Annual International Conference on Mobile Systems, Applications, andServices, 269–282. https://doi.org/10.1145/3210240.3210322[40] Jun Nishida and Kenji Suzuki. 2017. bioSync: A Paired Wearable Device for
Blending Kinesthetic Experience. In Proceedings of the 2017 CHI Conference on
Human Factors in Computing Systems (CHI ’17), 3316–3327. https://doi.org/10.
1145/3025453.3025829[41] Shuo Niu, Li Liu, and D. Scott McCrickard. 2019. Tongue-able interfaces: Prototyping and evaluating camera based tongue gesture input system. Smart Health
11: 16–28. https://doi.org/10.1016/j.smhl.2018.03.001[42] Viktorija Paneva, Sofa Seinfeld, Michael Kraiczi, and Jörg Müller. 2020. HaptiRead:
Reading Braille as Mid-Air Haptic Information. In Proceedings of the 2020 ACM
Designing Interactive Systems Conference (DIS ’20), 13–20. https://doi.org/10.1145/
3357236.3395515[43] Hangue Park, Mehdi Kiani, Hyung-Min Lee, Jeonghee Kim, Jacob Block, Benoit
Gosselin, and Maysam Ghovanloo. 2012. A Wireless Magnetoresistive Sensing
System for an Intraoral Tongue-Computer Interface. IEEE transactions on biomedical circuits and systems 6, 6: 571–585. https://doi.org/10.1109/TBCAS.2012.2227962[44] Wilder Penfeld and Theodore Rasmussen. 1950. The cerebral cortex of man; aclinical study of localization of function. Macmillan, Oxford, England.
[45] Martin Porcheron, Joel E. Fischer, Stuart Reeves, and Sarah Sharples. 2018. Voice
Interfaces in Everyday Life. In Proceedings of the 2018 CHI Conference on Human
Factors in Computing Systems (CHI ’18), 1–12. https://doi.org/10.1145/3173574.
3174214[46] Martin Porcheron, Joel E. Fischer, and Sarah Sharples. 2017. “Do Animals Have
Accents?”: Talking with Agents in Multi-Party Conversation. In Proceedings ofthe 2017 ACM Conference on Computer Supported Cooperative Work and Social
Computing, 207–219. https://doi.org/10.1145/2998181.2998298[47] Nimesha Ranasinghe and Ellen Yi-Luen Do. 2016. Digital Lollipop: Studying
Electrical Stimulation on the Human Tongue to Simulate Taste Sensations. ACM
Transactions on Multimedia Computing, Communications, and Applications 13, 1:
5:1-5:22. https://doi.org/10.1145/2996462[48] Nimesha Ranasinghe, Pravar Jain, Shienny Karwita, David Tolley, and Ellen YiLuen Do. 2017. Ambiotherm: Enhancing Sense of Presence in Virtual Reality by
Simulating Real-World Environmental Conditions. In Proceedings of the 2017 CHI
Conference on Human Factors in Computing Systems, 1731–1742. https://doi.org/
10.1145/3025453.3025723[49] J.M. Romano, S.R. Gray, N.T. Jacobs, and K.J. Kuchenbecker. 2009. Toward tactilely transparent gloves: Collocated slip sensing and vibrotactile actuation. In
World Haptics 2009 - Third Joint EuroHaptics conference and Symposium on
Haptic Interfaces for Virtual Environment and Teleoperator Systems, 279–284.
https://doi.org/10.1109/WHC.2009.4810815[50] Anne Roudaut, Andreas Rau, Christoph Sterz, Max Plauth, Pedro Lopes, andPatrick Baudisch. 2013. Gesture output: eyes-free output using a force feedback
touch surface. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems (CHI ’13), 2547–2556. https://doi.org/10.1145/2470654.2481352[51] Hooman Aghaebrahimi Samani, Rahul Parsani, Lenis Tejada Rodriguez, Elham
Saadatian, Kumudu Harshadeva Dissanayake, and Adrian David Cheok. 2012.
Kissenger: design of a kiss transmission device. In Proceedings of the Designing
Interactive Systems Conference (DIS ’12), 48–57. https://doi.org/10.1145/2317956.
2317965[52] T. Scott Saponas, Daniel Kelly, Babak A. Parviz, and Desney S. Tan. 2009. Opticallysensing tongue gestures for computer input. In Proceedings of the 22nd annual
ACM symposium on User interface software and technology - UIST ’09, 177. https://doi.org/10.1145/1622176.1622209[53] M. Sasaki, K. Onishi, T. Arakawa, A. Nakayama, D. Stefanov, and M. Yamaguchi.
2013. Real-time estimation of tongue movement based on suprahyoid muscle
activity. In 2013 35th Annual International Conference of the IEEE Engineering in
Medicine and Biology Society (EMBC), 4605–4608. https://doi.org/10.1109/EMBC.
2013.6610573

CHI ’23, April 23–28, 2023, Hamburg, Germany[54] K. Sathian and A. Zangaladze. 1996. Tactile spatial acuity at the human fngertip
and lip: Bilateral symmetry and interdigit variability. Neurology 46, 5: 1464–1464.
https://doi.org/10.1212/WNL.46.5.1464[55] Sue Ann Seah, Marianna Obrist, Anne Roudaut, and Sriram Subramanian. 2015.
Need for Touch in Human Space Exploration: Towards the Design of a Morphing
Haptic Glove – ExoSkin. In Human-Computer Interaction – INTERACT 2015
(Lecture Notes in Computer Science), 18–36. https://doi.org/10.1007/978-3-31922723-8_3[56] Vivian Shen, Craig Shultz, and Chris Harrison. Mouth Haptics in VR using a
Headset Ultrasound Phased Array. In Proceedings of the 2022 CHI Conference on
Human Factors in Computing Systems (CHI ’22).
[57] Maureen Stone, Jonghye Woo, Junghoon Lee, Tera Poole, Amy Seagraves, Michael
Chung, Eric Kim, Emi Z. Murano, Jerry L. Prince, and Silvia S. Blemker. 2018.
Structure and variability in human tongue muscle anatomy. Computer methods inbiomechanics and biomedical engineering. Imaging & visualization 6, 5: 499–507.
https://doi.org/10.1080/21681163.2016.1162752[58] Hui Tang and D.J. Beebe. 2003. Design and microfabrication of a fexible oral
electrotactile display. Journal of Microelectromechanical Systems 12, 1: 29–36.
https://doi.org/10.1109/JMEMS.2002.807478[59] R. W. Van Boven and K. O. Johnson. 1994. The limit of tactile spatial resolution
in humans: grating orientation discrimination at the lip, tongue, and fnger.
Neurology 44, 12: 2361–2366. https://doi.org/10.1212/wnl.44.12.2361[60] Daniel Vogel and Patrick Baudisch. 2007. Shift: a technique for operating penbased interfaces using touch. In Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems, 657–666. https://doi.org/10.1145/1240624.1240727

Arata Jingu et al.

[61] Martin Weigel, Aditya Shekhar Nittala, Alex Olwal, and Jürgen Steimle. 2017.
SkinMarks: Enabling Interactions on Body Landmarks Using Conformal Skin
Electronics. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), 3095–3105. https://doi.org/10.1145/3025453.3025704[62] Alexander Wilberz, Dominik Leschtschow, Christina Trepkowski, Jens Maiero,
Ernst Kruijf, and Bernhard Riecke. 2020. FaceHaptics: Robot Arm based Versatile
Facial Haptics for Immersive Environments. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–14. https://doi.org/10.1145/
3313831.3376481[63] Anusha Withana, Daniel Groeger, and Jürgen Steimle. 2018. Tacttoo: A Thin andFeel-Through Tattoo for On-Skin Tactile Output. In Proceedings of the 31st Annual
ACM Symposium on User Interface Software and Technology (UIST ’18), 365–378.
https://doi.org/10.1145/3242587.3242645[64] Xuhai Xu, Chun Yu, Anind K. Dey, and Jennifer Mankof. 2019. Clench Interface:
Novel Biting Input Techniques. In Proceedings of the 2019 CHI Conference on
Human Factors in Computing Systems (CHI ’19), 1–12. https://doi.org/10.1145/
3290605.3300505[65] Qiao Zhang, Shyamnath Gollakota, Ben Taskar, and Raj P.N. Rao. 2014. Nonintrusive tongue machine interface. In Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems (CHI ’14), 2555–2558. https://doi.org/10.
1145/2556288.2556981[66] Ziliang Zhou, Yicheng Yang, and Honghai Liu. 2022. A Braille Reading System
Based on Electrotactile Display With Flexible Electrode Array. IEEE/CAA Journalof Automatica Sinica 9, 4: 735–737. https://doi.org/10.1109/JAS.2022.105476