A Smart Bracelet Supporting Tactile Communication and Interaction

2022 IEEE Haptics Symposium (HAPTICS) | 978-1-6654-2029-7/22/$31.00 ©2022 IEEE | DOI: 10.1109/HAPTICS52432.2022.9765616

Stejara Dinulescu1 , Neeli Tummala2 , Gregory Reardon1 , Bharat Dandu2 ,Dustin Goetz3 , Sven Topp4 , and Yon Visell1,2,3

Abstract— The sense of touch can convey semantic and emotional information in social or computer-mediated interactions.
Touch plays an essential role in communication with individuals
affected by multiple sensory loss, many of whom use modes oftouch communication that can be broadly described as tactile
sign languages. Few technologies exist today to support suchinteractions. Here, we present a smart bracelet for facilitatingtactile communication and interaction. The smart bracelet
captures and analyzes vibrations that are elicited in the skin
via touch gestures performed on the hand. We demonstrate
the utility of this system for supporting communication via the
Deafblind Manual alphabet, which is a tactile sign language.
This smart bracelet can classify signed letters with greater than
90% per-letter accuracy. These results show how existing modesof tactile communication can be integrated with information
technologies. This work may furnish new paradigms for humancomputer interaction via self- and interpersonal-touch contact.

I. I NTRODUCTION
The skin is a highly expressive medium used for perceiving and interacting with the world, including communication
through touch. Indeed, touch interaction can convey significant meaning, intent, and sentiment. The diverse repertoireof touch interactions encountered in everyday settings,
including interpersonal touch, contrasts starkly with thelimited communicative and expressive range of touch foundin most computing systems today. In some interpersonal
interactions, including those involving individuals withmultiple sensory impairments such as deafblindness, touch isthe principal medium of communication [1], [2], [3]. Several
languages, organizations, and educational systems exist to
support the needs of individuals with deafblindness, butthere exists a lack of accessibility for digital communication,
which is crucial to address due to its ubiquity. However,current interaction paradigms primarily lie in the audiovisual
domains. New technologies are needed that can enable
individuals with all ranges of sensory abilities to engage
with information technologies and digital resources.
Here, we present a smart bracelet for supporting tactile
communication and interaction (Fig. 1A). This system
captures vibration signals elicited in the skin by touch contact

gestures performed on the hand; these gestures could be initiated by the wearer or by another person. The captured vibrations are analyzed using signal processing and machine learning methods that can recognize, from a designated lexicon,
which gestures are performed (Fig. 1B1). We demonstrate theability of this system to capture and classify touch gesturesfrom the Deafblind Manual alphabet (Fig. 1C), a tactile sign
language (TSL) used by individuals in Australia who aredeafblind, in order to support digital tactile communication.
While several researchers have investigated output technologies for reproducing TSL [4], [5], [6], [7], the present
work is, to the best of the authors’ knowledge, the first input
device specifically directed at supporting digital TSL communication. Such a device could address important unmet
needs, such as those of transcribing information conveyed
via TSL communications in court proceedings, emergencies,
policy forums, scholarly meetings, or moments of personalor historical significance, especially where transcripts mightotherwise be unavailable. Our TSL input device enables theletter-by-letter translation of touch gestures performed on
the hand. One might, for example, use the palm of one’s
own hand to enter text, take notes, or send a text message
normally requiring a standard phone interface (Fig. 1B2).
Such a system could assist individuals with limited vision,
for whom usage of mobile devices can be challenging.
In other applications, smart bracelets like the one presented here could also support the training of TSL interpreters. When combined with an output device capableof reproducing TSL gestures, such a smart bracelet could
provide a means of TSL communication over the internet, including remote TSL interpretation, improving access to suchservices by many individuals. When combined with computer translation algorithms, such a smart bracelet could also
facilitate conversation between individuals who communicate
via different languages, including deafblind individuals who
may not use the same TSL (Fig. 1B3). Indeed, a diverse
variety of TSLs are used around the world [6].
A. Deafblindness and Tactile Communication

1 Stejara

Dinulescu, Gregory Reardon, and Yon Visell are with the
Department of Media Arts and Technology, University of California, Santa
Barbara, Santa Barbara, CA 93106 sdinulescu@ucsb.edu
2 Neeli Tummala, Bharat Dandu, and Yon Visell are with the Department
of Electrical and Computer Engineering, University of California, Santa
Barbara, Santa Barbara, CA 93106
3 Dustin Goetz and Yon Visell (by courtesy) are with the Departmentof Mechanical Engineering, University of California, Santa Barbara, Santa
Barbara, CA 93106
4 Sven Topp is with the School of Psychology, University of Sydney,
Camperdown NSW 2006, Sydney, Australia

978-1-6654-2029-7/22/$31.00 ©2022 IEEE

Research and development in automatic speech
recognition and language processing over the past 50
years has yielded algorithms and systems for capturing,
computationally understanding, and interacting withcomputers via speech. More recently, analogous methods
to speech recognition have been used to support computerbased communication by detecting hand poses during visualsign languages, such as American Sign Language [8]. To
2022 IEEE Haptics Symposium (HAPTICS)

Authorized licensed use limited to: Univ of Calif Santa Barbara. Downloaded on October 27,2022 at 21:06:29 UTC from IEEE Xplore. Restrictions apply.

Fig. 1. A) Wrist-worn acoustic sensing interface to encode and detect touch gestures on the hand. B) System overview and envisioned applications for
detection of touch gestures on the skin. B1) Touch signals signed on the palm are transmitted to a PC via serial communication with the microcontroller.
The data is processed and features are extracted to be used for classification. B2) Wearable interface enables self-signing on the palm of the hand to encode
semantic meaning for note-taking purposes, sending a remote text message via touch, or transcription/record-keeping in the tactile domain. B3) Wearable
interface translates interpersonal touch gestures as they are being signed. This could facilitate learning of TSL gestures, transcription of conversations in
the tactile domain, and translation of TSL in real-time when the wearer is uncertain of the semantic mappings of the alphabet. C) The 26 letters of the
Deafblind Manual alphabet.

date, little attention has been given to achieving similargoals in tactile communication, particularly TSLs.

support TSL communication by enabling transcription,
analysis, or reproduction of TSL interactions.

A recent report by the World Federation of the Deafblind
indicates that between 0.2-2% of people are impacted by
sensory impairment of both vision and hearing. Deafblind
individuals use a variety of languages for tactile communication, depending on their region, community, and individual
factors, such as the onset and severity of sensory impairment.
Many TSLs exist today supporting critical daily activities
and social interaction [9]; in addition, many of the TSLs areletter-based semantic touch communication methods. Here,
we demonstrate a system for supporting TSL communication, specifically utilizing the Deafblind Manual alphabet,which involves the performance of touch gestures, including
tapping, lightly pinching, and sliding on the surface of thehand. Each of the 26 gesture patterns corresponds to a letterin the English alphabet [6] (Fig. 1C).

B. Capturing Touch Gestures via Vibration Signatures

Other technologies for supporting communication byindividuals with visual and auditory sensory loss include
Braille interfaces, emerging methods for tactile graphic
displays, and more commonly available screen-based
accessibility features for those with low vision. However,such methods do not support the communication needs of
all individuals who are deafblind. More recently, researchers
have engineered systems to translate text or speech into
tactile patterns [4], [5], including TSL gestures [6], [9],[10]. However, the authors are not aware of any priorefforts to facilitate or develop a computational encoding oftactile input. A system such as the one presented here could
978-1-6654-2029-7/22/$31.00 ©2022 IEEE

Touch interactions with the skin elicit vibrations that travel
far from the point of contact, encoding information about thecontact location and nature of the gesture [11], [12], [13],[14], [15]. This physical process makes it feasible to collectinformation about tactile interactions using vibration sensors
that are positioned remotely from the contact location.
Human-computer interaction research has utilized such processes to provide means of computational input via contact
with the skin, as in the Skinput device [16]. Many other sensing methods can be used for capturing information from skincontact, including capacitive, electromyography, ultrasound,force, and other sensing techniques [17], [18], [19], [20].
However, the propensity of touch-elicited vibrations to travel
great distances in the skin and the efficiency with which suchsignals can be captured electronically via low-cost, readily
available sensors makes this approach particularly attractive
for interactive device engineering, especially in settings
where it is preferable to leave the hand unencumbered.
C. Summary of Contributions
We present a smart wrist wearable for capturing andrecognizing touch gestures that occur on the hand duringskin-to-skin contact. To inform the design of this smart
bracelet, including the number and placement of sensors,
we captured the signal profiles of a subset of letters of the
Deafblind Manual alphabet with high spatial resolution using
2022 IEEE Haptics Symposium (HAPTICS)

Authorized licensed use limited to: Univ of Calif Santa Barbara. Downloaded on October 27,2022 at 21:06:29 UTC from IEEE Xplore. Restrictions apply.

Fig. 2. A) Measured whole-hand mechanical responses for a subset of the Deafblind Manual alphabet. We found that the contact is not only reflected inskin responses at the touched location, but also that this energy propagated outwards to regions on the skin, on both sides of the hand, far from the pointof contact. B) We found that the tactile gestures produced consistent, repeatable patterns of RMS acceleration across the hand (mean absolute correlation> 0.95 within trials of the same gesture), while yielding very low correlations when comparing across trials of different gestures.

a sensor array encompassing the whole hand. By analyzing
this data, we identified that even letters that are highly
similar to one another in tactile sign contain unique signal
profiles, even when measured at locations removed fromthe gesture contact. Informed by this finding, we designed
a compact, wrist-worn device to capture these propagating
signals without hindering natural manual movements andtactile interactions. To demonstrate the utility of this system
for tactile communication, we illustrate the capacity of ourdevice to classify 26 different touch gestures—in this case,the letters of the Deafblind Manual alphabet. We find thatthe device can accurately discriminate between different
touched contact locations (e.g. digit II, digit V, or palm)
and between different types of touch contact (e.g. slides,
impulses, pinches, or squeezes). We show using simple,
supervised machine learning methods that such a device isable to identify these 26 different gestures with above 90%
accuracy, even when trained on a small dataset.
II. W HOLE -H AND S ENSING

Letters were chosen for high frequency of use (i.e. vowels)and letter similarity (i.e. M and N, D and E). Because thepalmar surface of the hand was also covered with sensors,
we slightly shifted the signed location of M and N to
avoid interfering with the sensors. The palm was held facing
upwards, with the forearm stabilized against the edge of atable at chest-level to minimize noise from spurious hand
movements. The data consisted of ten, 84-channel 3-axis
acceleration signals for each letter. The data was de-meaned
and bandpass filtered, then compressed to a single axis bycomputing the magnitude of the vector. We then computed
the RMS of the acceleration magnitude.
To determine the spatial distribution of acceleration energy
for each gesture class, we compute root mean square (RMS)
acceleration over all trials. We interpolated these data using
coordinates obtained from an anatomically plausible 3D handmodel using squared distance weighting. We analyzed thedifferences between letters using this data and also computed
the mean pairwise absolute correlation (Pearson’s r) between
the RMS acceleration of all trials and of all letters.

A. Materials and Methods
In a first experiment, we surveyed the whole-hand mechanical responses produced during the signing of a subset
of letters from the Deafblind Manual alphabet. We used two
42-channel, 3-axis accelerometer arrays (Model LI3DSH,
ST Microelectronics) [12]. We captured data at a rate of1300 Hz via a field-programmable gate array (FPGA) based
multichannel data acquisition board, with firmware on the
FPGA and software on a Windows PC. We affixed eachaccelerometer in the array to the skin via doubled-sided
adhesive, with one array attached to the dorsal side of thehand and the other to the palmar (Fig. 2B).
We captured the skin acceleration for 10 trials of 9 lettersof the Deafblind Manual alphabet (A, D, E, F, I, M, N, O, U),signed on the palmar side of the hand through interpersonal
touch (i.e one person signs on the hand of another person).
978-1-6654-2029-7/22/$31.00 ©2022 IEEE

B. Results
Each tactile gesture class yielded distributions of RMS acceleration that reflected the contact location and the nature of
the gesture (single touch - A, E, I, O, U and multi-touch - D,
F, M, N) (Fig. 2A). For example, the peak RMS accelerationof the letter E was localized to the point of contact. Multitouch gestures tended to deliver high RMS acceleration(relative to the peak RMS acceleration for that gesture) tolocations far from the point of contact, reflecting that thecontact location on the hand was much larger (2-3 times thesize of the contact for single touch gestures). Furthermore,
all gestures delivered energy to the wrist; the amount of
RMS acceleration delivered at the wrist was a functionof the distance from the contact location (e.g. the contact
location for E is much further from the wrist than for M/N).
2022 IEEE Haptics Symposium (HAPTICS)

Authorized licensed use limited to: Univ of Calif Santa Barbara. Downloaded on October 27,2022 at 21:06:29 UTC from IEEE Xplore. Restrictions apply.

Fig. 3. A) Wrist-wearable device using four accelerometers affixed to the skin to capture mechanical vibrations produced during tactile fingerspelling.
B) Measured accelerometer signals produced during a single trial of each letter of Deafblind Manual alphabet. Palm tap and finger tap gestures (B1 andB2, respectively) produced transients that rapidly decayed. Finger taps tended to deliver less energy to the wrist when compared to palm taps, reflecting
the difference in distance between the point of contact and the sensor. Sliding or pinching/grabbing gestures (B3 and B4, respectively) produced signalsthat tended to decay at a much slower rate on the skin, reflecting that the gesture occurs over a longer time scale than the tap gestures.

The tactile gestures elicited vibration patterns that wereconsistent across gesture instances (Fig. 2B), and werequalitatively distinct from the other gestures. Consistent withthe observations of the whole-hand patterns (Fig. 2A), somegestures produced similar distributions of RMS accelerationand thereby high correlations (i.e. M/N and D/E/F). For thesubset of data captured by the ten accelerometers located on
the wrist, averaged within-gesture mean absolute correlationsof the RMS acceleration (0.94) were substantially larger thanacross-gesture correlations (0.48). These results suggested
that acceleration information captured at the wrist could be
used to classify varied tactile gestures.
III. W RIST-W ORN I NTERFACE
Informed by the preceding findings, including the extent
to which similar information could be captured via sensors
at locations removed from the point of contact with the hand,
we designed a smart bracelet using a minimal arrangementof sensors located at the wrist. We evaluated the utility ofthis device for classifying touch contact gestures performedon the hand, in the form of TSL gestures.
A. Materials and Methods
1) Device Design: The device was composed of an ABS
shell enclosing two Adafruit Feather M0 WiFi microcontrollers. It employed a total of four 3-axis analog accelerometers (Analog Devices ADXL335) (Fig. 1A). Each sensor was
adhered to the skin. Pairs of sensors were positioned 25 mmapart on both the palmar and dorsal sides of the wrist (Fig.
3A), at locations determined from the preceding experiment.
978-1-6654-2029-7/22/$31.00 ©2022 IEEE

The casing for the electronics was designed to avoid mechanically disturbing the sensing elements. The data was captured
at a sampling frequency of 1250 Hz and at a resolution of
12 bits using the onboard ADC of the microcontroller.
2) Collection of TSL Data: We collected accelerometermeasurements during TSL touch contact in each of two
conditions: one in which the signer performed gestures on
their own hand (self-sign, or SS, depicted in Fig. 1B2)
and one in which the signer contacts another person’s hand(interpersonal-sign, or IS, depicted in Fig. 1B3). The palmar
surface of the hand, held outstretched and flat, was facing
upwards throughout data collection, with the dorsal sideof the forearm stabilized against the edge of a table tominimize disturbance from unintentional limb movements.
In both conditions, we recorded 40 trials of each of the 26
letters.
3) Signal Processing and Feature Extraction: The data
for each letter were segmented, lowpass filtered at 250
Hz, and compressed from 3 axes to 1 axis using principal
component analysis (PCA). After compression, the data foreach trial was normalized independently,
yc,nyc,n =(1)
max{c,n} |yc,n |where n is the sample number and c is the channel number.
This preserved the inter-channel differences between
accelerometers, while also reducing inter-trial variabilityof the same gesture and improving the consistency of thefeature estimates across gestures.
For every trial, we computed time-domain, frequency2022 IEEE Haptics Symposium (HAPTICS)

Authorized licensed use limited to: Univ of Calif Santa Barbara. Downloaded on October 27,2022 at 21:06:29 UTC from IEEE Xplore. Restrictions apply.

domain, and spectro-temporal features for all 4 channels ofeach analyzed segment, yielding 96 features per trial. The
time-domain features were peak-to-peak amplitude
P T Pc = | max yc,n − min yc,n |
n

n

(2)

and mean absolute deviationN

M ADc =

1 X
|yc,n − ȳc |
N n=1

(3)

where ȳc is the mean of signal yc,n . This yielded a total of
8 time-domain features per segment.
For the second set of features, we computed the Discrete
Fourier Transform (DFT) of each of the normalized signals,
Yc,k = F{yc,n }, where F is the DFT. From the frequencydomain representation of the signal, we computed thespectral centroid
PN
fk |Yc,k |SCc = Pk=1(4)
N
k=1 |Yc,k |where fk is the center frequency of bin k and |Yc,k | isthe magnitude of bin k for channel c. In addition, we
computed the center frequency of the DFT bin with thelargest magnitude for each channel c,
Fc = fk∗ s.t. k ∗ = arg max |Yc,k |k(5)

We computed the mean and standard deviation of the
DFT bins over a 50 Hz bandwidth. Let the bandwidthbi ∈ {[0 50], [50 100], [100 150], [150 200], [200 250]} fori = 1, ..., 5. We associate the set of bins Ki whose center
frequencies lie within bandwidth bi and compute:
1 Xµc,bi =
|Yc,k | ,
(6)
|Ki |
k∈Ki
s
1 X
σc,bi =(|Yc,k | − µc,bi )2
(7)
|Ki |
k∈Ki

where |Ki | is the size of set Ki . This produced 40 features(2 measures × 5 bandwidths × 4 channels), for a total of
48 frequency-domain features.
Finally, we composed a set of spectro-temporal features
to capture temporal variations in the spectral envelope ofthe signal. We computed the Short-Time Fourier Transform(STFT) of the signals (window size K=256 samples, 50%
overlap, Hanning window), yielding a matrix of Fourier
coefficients (M × K) for each channel c where each of the
M rows is the DFT of the signal over different times. We
computed the mean and standard deviation of the measureµc,bi (see Eq. 6) for each row of the Fourier matrix. Letµc,bi ,m be the mean of bandwidth bi for channel c for row
m. We compute
M

FT
µST
=
c,bi

1 X
µc,bi ,m ,
M m=1

v
u
M
u 1 X
ST F T
F T )2σc,bi = t(µc,bi ,m − µST
c,bi
M m=1
978-1-6654-2029-7/22/$31.00 ©2022 IEEE

(8)

(9)

This yielded another set of 40 features (2 measures × 5
bandwidths × 4 channels). All features (temporal, frequency,
and spectro-temporal) were concatenated to form a 96dimensional vector, used to train the machine learning modeland classify the 26 different tactile gestures.
4) TSL Classification: To distinguish the 26 letters of the
Deafblind Manual alphabet via our smart bracelet, we passcomputed features into three common supervised machinelearning classification models: a support vector machine witha linear kernel (SVM), logistic regression with PCA (LR),and a random forest model (RF). To minimize bias during
training and classification, we employed a 10-fold crossvalidation procedure with a 90-10 train-test split. During
each classification run, we computed the mean and varianceof each feature in the training data and used these sample
moments to whiten both the training and testing data. We
performed classification for each mechanical dataset separately (SS and IS conditions, 40 trials per letter) and on bothdatasets (combined condition, 80 trials per letter).
B. Results and Discussion
1) Mechanical Signal Profiles: We found that the measured signals (Fig. 3B) yielded noticeably different timedomain acceleration responses, which became more prominent when we grouped the gestures by palm taps (Fig. 3B1),
finger taps (Fig. 3B2), slides (Fig. 3B3), and pinches/grabs(Fig. 3B4). We found that tapping gestures yielded transient
signals that decayed rapidly, while slides and pinches/grabs
decayed at much slower rates in the skin. This likely reflects
the variation in contact conditions between the gestures, forwhich slides and pinches/grabs occur at much longer timescales. We found that finger taps delivered less energy to
the wrist when compared to palm taps, reflecting that theproperties of the skin, which is a highly damped medium,play a role in encoding the distance from the contact location
to the wrist, thereby differentiating the measured tactile
gestures. Finally, we found that sliding gestures, regardless ofcontact location, tended to deliver less energy to the wrist, butthat this energy was nonetheless measurable and informative.
2) TSL Classification: When utilizing data from all four
sensors, the SVM classifier consistently outperformed the
RF and LR classifiers in the SS and IS conditions, withaverage accuracies greater than 93% across cross-validation
folds; in the combined condition, the RF classifier slightly
outperformed the others at close to 90% average accuracy
across folds (Table I). High performance using SVM ispromising for real-time applications, as a trained SVM model
can easily be implemented on a micro-controller without theneed for additional computational power. In the following
analysis, we focus on the performance of the SVM model.
We further analyzed classification performance using different subsets of sensor channels (Fig. 3A) to determine
the minimal number of sensing element required for robust classification. The subsets considered include all fourchannels (α, β, γ, δ), two channels on the palmar side of
the wrist (α, β), two channels on the dorsal side of thewrist (γ, δ), one channel on the palmar side of the wrist
2022 IEEE Haptics Symposium (HAPTICS)

Authorized licensed use limited to: Univ of Calif Santa Barbara. Downloaded on October 27,2022 at 21:06:29 UTC from IEEE Xplore. Restrictions apply.

DatasetSS
IS
Combined

SVM
93.9
93.7
86.3

Classifier
RF
LR
88.7
90.6
92.2
91.0
89.4
76.1

TABLE IAVERAGE C LASSIFICATION ACCURACY (%) ACROSS DATASETS AND
M ODELS FOR 4-S ENSOR C ONFIGURATION

DatasetSS
IS
Combinedα, β, γ, δ
93.9
93.7
86.3

Sensor Configuration
α, β
γ, δ
α, γ
α
86.5
86.8
87.1
76.3
88.3
85.4
87.1
75.0
76.8
75.6
78.4
61.6

γ
73.8
75.4
64.3

TABLE IIAVERAGE C LASSIFICATION ACCURACY (%) ACROSSDATASETS AND S ENSOR C ONFIGURATIONS FOR SVM M ODEL(α), and one channel on the dorsal side of the wrist (γ).
Table II summarizes the average classification accuracies
across cross-validation folds for the SS, IS, and combined
datasets. The use of all four channels yields the highest
classification accuracies, but the employment of just two
sensors maintains robust performance (above 85% in the
single subject conditions), indicating that they are sufficientfor detecting various sets of tactile gestures. Mounting thesensors on the palmar or dorsal side of the wrist does not
significantly change classification accuracies. On the other
hand, classification accuracy drops significantly when using
only one sensor, indicating that inter-channel differences
encode relevant classification information that cannot be
discarded. Close inspection of the signals shown in Fig. 3Bshows differences in the phase and shape of signals between
channels, most evident in the sliding gestures.
We found that the most commonly misclassified letterswere L and N, which is expected due to the similarity of thetactile gestures; N is signed with two fingers in the centerof the palm whereas L is signed with one finger in thesame location (Fig. 4). Additionally, R and T are commonly
mislabeled, as they are both taps occuring near the same
palmar location: T with one finger and R with two fingers.
Other commonly misclassified letter pairs include G and X,
and W and Z. The former are gestures performed with thesigner’s whole hand, producing high acceleration signals at
the palm; the latter produce similar high energy signals, but
occur at different locations on the hand.
We also examined classification accuracy within each gesture grouping shown in Fig. 3B (i.e. palm taps, finger taps,
slides, and pinches/grabs) for the combined dataset in thefour channel configuration, averaged across cross-validation
folds. Sliding gestures are classified with the highest accuracy (92.8%), likely because they produce the most distinct
mechanical signals. Finger taps are also accurately classified(90.8% accuracy) and are generally only misclassified asother finger taps, likely because the captured signals often
possess similar magnitudes and temporal profiles. Similarly,
palm taps (81.1% accuracy) are accurately recognized andare typically only misclassified as other palm taps. This
978-1-6654-2029-7/22/$31.00 ©2022 IEEE

Fig. 4. Confusion matrix for the linear SVM Classifier on the combined
dataset, with per-letter accuracies along the diagonal. On the off-diagonal,
type I (false positives) read column-wise and type II (false negatives) readrow-wise. Commonly misclassified letter pairs can often be grouped into
the subsets introduced in Fig. 3. For example. {E,D} are both taps on digit
II, and {G, X, Z}, {L, N}, and {R, T} are all taps on the palm.

is expected when considering the similarity between manyof the palm taps; for example, L, M, N, and V are all
signed in the middle of the palm but with different finger
placements. Finally, pinch or grab gestures (83.8% accuracy)
are somewhat more likely to be misclassified as other typesof gestures due to their heterogeneous nature.
Regardless of the classifier, dataset, or sensor
configuration, we observe accuracies above 60%, which issignificantly higher than chance (3.9% for 26 classes). In
the majority of SS and IS conditions, accuracies are above
90% when using four sensors, regardless of classifier used.
This indicates that accuracies approaching 100% can likely
be achieved via standard techniques such as lexicon-basedmodels that are widely used in natural language processing
and automatic speech recognition. Further improvements
might also be achieved through the use of state-of-the-art
multi-layer neural network classifiers.
IV. C ONCLUSION
We present a smart bracelet system that transforms theskin into a touch gesture input interface. The bracelet canclassify touch gestures of the Deafblind Manual alphabet, via
integrated wrist-worn sensors, with significantly higher-thanchance accuracy. Our approach demonstrates the success ofminimal accelerometer-based sensing that leaves the hand
unencumbered for manual interactions and movements. Measurements from a whole-hand sensing array were used in an
initial exploration to identify a subset of four sensors that areable to capture enough salient information in a touch gesture,thus reducing cost and complexity. We demonstrated theutility of this device for supporting tactile communication via
TSL. We present analyses of an experiment in which we cap2022 IEEE Haptics Symposium (HAPTICS)

Authorized licensed use limited to: Univ of Calif Santa Barbara. Downloaded on October 27,2022 at 21:06:29 UTC from IEEE Xplore. Restrictions apply.

tured and accurately classified signal profiles for all 26 lettersof the Deafblind Manual alphabet, demonstrating promisinguses of such systems for supporting tactile communication.
The smart bracelet can be used for input via tactile
gestures performed on the hand of the user by the user, which
could be useful for note-taking, transcription, or textual
communication. Such a device could support interpersonal
tactile communication by individuals who prefer using
touch over other modalities, in addition to encoding touchin the digital domain. These findings may also enhance theutility of output devices for tactile communication. Utilizing
input and output devices in conjunction could support
communication between two or more individuals, including
people who may communicate via different TSLs.
While we have demonstrated the capability to accurately
classify the 26 gestures of the Deafblind Manual alphabet,more remains to generalize features for classification oflarger mechanical datasets consisting of various TSL signers,
receivers, and alphabets. Further investigation into individual
differences between subjects, such as hand size, signingspeeds, and emotion (i.e. happy, angry) conveyed through
TSL, will be experimentally investigated.
We further envision supporting real-time classification of
TSL, where signing occurs as fast as five letters per second[6]. In these cases, intentional or accidental movements of
the hand (such as finger motion from a previously performed
tactile gesture) must not interfere with signals captured fromthe sensors. Thus, further data capture and signal processing
methods will be explored, examples of which include capturing data from two devices worn on both the signer’s andreceiver’s wrist, or capturing false positive dataset examples(such as muscle contractions, finger twitches, etc.) to help
identify gestural noise versus salient gesture data. We will
additionally explore multi-layer classification and feature
optimization by additional meta-analysis of similarities anddifferences between tactile signals in a gesture set, as well as
natural language processing as previously applied in speechtechnologies. Such advancements are expected to improveaccuracy and robustness, aiding applications in natural communication. This system holds promise for supporting tactile
communication, including scenarios where it is paired witha haptic communication output device, to address communication needs of individuals who are deafblind.
ACKNOWLEDGMENT
The authors thank Basil Duvernoy, Jeraldine Milroy, andYitian Shao for their feedback. The authors acknowledge
support from the University of California, Office of the
President, National Science Foundation (NSF) FellowshipNo. 2139319 to Dinulescu and Goetz, and NSF award
1751348 to Visell. Any opinion, findings, and conclusions or
recommendations expressed in this material are those of theauthors and do not necessarily reflect the views of the NSF.
R EFERENCES

method of speech communication. The Journal of the AcousticalSociety of America, 77(1):247–257, 1985.
[2] C. M. Reed, L. A. Delhorne, and N. I. Durlach. Tactile receptionof fingerspelling and sign language. The Journal of the AcousticalSociety of America, 82(S1):S24–S24, 1987.
[3] Jack M. Loomis. On the tangibility of letters and braille. Perception& Psychophysics, 29(1):37–46, January 1981.
[4] Hugo Nicolau, Kyle Montague, Tiago Guerreiro, André Rodrigues,
and Vicki L Hanson. Holibraille: Multipoint vibrotactile feedback onmobile devices. In Proceedings of the 12th International Web for All
Conference, pages 1–4, 2015.
[5] Ulrike Gollner, Tom Bieling, and Gesche Joost. Mobile lorm glove:
introducing a communication device for deaf-blind people. In Proceedings of the sixth international conference on tangible, embedded
and embodied interaction, pages 127–130, 2012.
[6] Basil Duvernoy, Ildar Farkhatdinov, Sven Topp, and Vincent Hayward.
Electromagnetic actuator for tactile communication. In International
Conference on Human Haptic Sensing and Touch Enabled Computer
Applications, pages 14–24. Springer, 2018.
[7] Hong Z. Tan, Charlotte M. Reed, Yang Jiao, Zachary D. Perez,
E. Courtenay Wilson, Jaehong Jung, Juan S. Martinez, and Frederico M. Severgnini. Acquisition of 500 english words through a tactile
phonemic sleeve (taps). IEEE Transactions on Haptics, 13(4):745–
760, 2020.
[8] C. Vogler and D. Metaxas. Parallel hidden markov models foramerican sign language recognition. In Proceedings of the SeventhIEEE International Conference on Computer Vision, volume 1, pages
116–122 vol.1, 1999.
[9] Basil Duvernoy, Sven Topp, and Vincent Hayward. Hapticomm,a haptic communicator device for deafblind communication. In
International AsiaHaptics conference, pages 112–115. Springer, 2018.
[10] Arthur Theil, Lea Buchweitz, James Gay, Eva Lindell, Li Guo, NilsKrister Persson, and Oliver Korn. Tactile board: A multimodal
augmentative and alternative communication device for individuals
with deafblindness. In 19th International Conference on Mobile andUbiquitous Multimedia, MUM 2020, page 223–228, New York, NY,
USA, 2020. Association for Computing Machinery.
[11] Yitian Shao, Vincent Hayward, and Yon Visell. Spatial patterns of cutaneous vibration during whole-hand haptic interactions. Proceedingsof the National Academy of Sciences, 113(15):4188–4193, 2016.
[12] Yitian Shao, Hui Hu, and Yon Visell. A wearable tactile sensor arrayfor large area remote vibration sensing in the hand. arXiv preprint
arXiv:1908.08199, 2019.
[13] Yitian Shao, Vincent Hayward, and Yon Visell. Compression of
dynamic tactile information in the human hand. Science advances,6(16):eaaz1158, 2020.
[14] Xavier Libouton, Olivier Barbier, Yorick Berger, Leon Plaghki, andJean-Louis Thonnard. Tactile roughness discrimination of the finger
pad relies primarily on vibration sensitive afferents not necessarily
located in the hand. Behavioural brain research, 229(1):273–279,
2012.
[15] Katherine O Sofia and Lynette Jones. Mechanical and psychophysicalstudies of surface wave propagation during vibrotactile stimulation.
IEEE transactions on haptics, 6(3):320–329, 2013.
[16] Chris Harrison, Desney Tan, and Dan Morris. Skinput: appropriating
the skin as an interactive canvas. Communications of the ACM,
54(8):111–118, 2011.
[17] Adiyan Mujibiya, Xiang Cao, Desney S Tan, Dan Morris, Shwetak N
Patel, and Jun Rekimoto. The sound of touch: on-body touch andgesture sensing based on transdermal ultrasound propagation. In
Proceedings of the 2013 ACM international conference on Interactive
tabletops and surfaces, pages 189–198, 2013.
[18] Yang Zhang, Wolf Kienzle, Yanjun Ma, Shiu S Ng, Hrvoje Benko,
and Chris Harrison. Actitouch: Robust touch detection for on-skinar/vr interfaces. In Proceedings of the 32nd Annual ACM Symposium
on User Interface Software and Technology, pages 1151–1159, 2019.
[19] Subramanian Sundaram, Petr Kellnhofer, Yunzhu Li, Jun-Yan Zhu,
Antonio Torralba, and Wojciech Matusik. Learning the signatures of
the human grasp using a scalable tactile glove. Nature, 569(7758):698–
702, 2019.
[20] Riley Booth and Peter Goldsmith. A wrist-worn piezoelectric sensor
array for gesture input. Journal of Medical and Biological Engineering, 38(2):284–295, 2018.

[1] C. M. Reed, W. M. Rabinowitz, N. I. Durlach, L. D. Braida,
S. Conway-Fithian, and M. C. Schultz. Research on the tadoma

978-1-6654-2029-7/22/$31.00 ©2022 IEEE

2022 IEEE Haptics Symposium (HAPTICS)

Authorized licensed use limited to: Univ of Calif Santa Barbara. Downloaded on October 27,2022 at 21:06:29 UTC from IEEE Xplore. Restrictions apply.