# Sub-Auditory Speech Recognition

### Kim Binsted ICS Department, University of Hawaii  Charles Jorgensen Code IC, NASA Ames Research Center

### Abstract

*Sub-auditory speech recognition using electromyogram (EMG) sensors is
potentially useful for interfaces in noisy environments, for discreet or
secure communications, and for users with speech related disabilities.
Past research has shown that a scaled conjugate gradient neural network,
using dual tree wavelets for feature transformation, can categorize EMG
signals for small sets of individual words. Here we describe an attempt
to recognize the individual phonemes of the English language.
Recognition rates are significantly higher than chance; however, they
are not high enough to be useful without further improvements, perhaps
from using multiple neural networks on specialized recognition tasks,
better sensor placement, and/or moving on to di/triphone recognition as
is commonly used in audible speech processing.*

### Introduction

Speech recognition research and technology to date (e.g. Huang *et al*,
2001) has, quite naturally, focused on audible speech. However, there
are many applications that call for sub-auditory speech recognition. In
particular, sub-auditory speech recognition would be valuable in noisy
environments (e.g. crowded rooms, cockpits), in environments where sound
does not carry well or at all (e.g. underwater, near-vacuum), when
discreet or secure communications are necessary or desirable (e.g.
military applications, off-line comments during meetings), and for users
with speech-related disabilities (e.g. vocal cord damage).

### Relevance 

![](media/image1.wmf){width="1.5972222222222223in"
height="2.0833333333333335in"}NASA is interested in multi-modal
interfaces as a way of increasing communication robustness and reducing
information overload in human-human and human-agent systems. Imagine an
astronaut exploring the surface of Mars, in collaboration with other
human astronauts, a variety of robotic rovers, intelligent shipboard
systems, and a mixed human-agent team back on Earth. In a space suit,
both inputs and outputs are severely limited, and audible speech is by
far the most convenient communications channel. However, audible speech
interfaces cannot support many parallel interactions and, in the case of
human-agent communication, are not very robust in the presence of noise,
speaker stress, changes in gas mixture, etc. By adding sub-vocal speech
to the repertoire of space suit interface designers, we hope to increase
the robustness of audible speech by providing redundancy, and also to
provide an alternate means of communication when appropriate (e.g.
discreet communications) or necessary (e.g. a physiological problem
renders the audible speech interface unusable).

### Related Work

Little work appears to have been done on the usefulness of EMG sensors
alone in speech recognition. Auditory speech recognition augmentation
with EMG sensors, along the lines of that in our word experiments, was
performed by Chan (2001), who proposed supplementing voiced speech with
EMG in the context of aircraft pilot communication. Chan studied the
feasibility of augmenting auditory speech information with EMG signals
recorded from primary facial muscles using sensors imbedded in a pilot
oxygen mask. Chan used five surface signal sites during vocalized
pronunciation of the digits zero to nine using Ag-AgCl button electrodes
and an additional acoustic channel to segment the signals. This work
demonstrated the potential of using information from multi-source
aggregated surface sensors to improve performance of a conventional
speech recognition engine.

### Approach

Here, we attempt to apply lessons learned from signal processing in
general, and speech recognition in particular, to the problem of
sub-auditory speech recognition. Using EMG sensors placed strategically
in the throat area (see Figure 2), we hope to detect pre-speech EMG
signals, and recognize the intended speech, even when it is not produced
audibly. A key research question is: To what extent can audible speech
recognition technologies and principles be applied to the problem of
sub-auditory speech recognition?

![](media/image2.jpeg){width="2.7625in"
height="2.7263888888888888in"}*et al*, 2003) has shown considerable
promise. Using dual tree wavelets for feature transformation and a
scaled conjugate gradient net for categorization, Jorgensen *et al*
achieved 92% word recognition, over a set of six control words ("stop",
"go", "left", "right", "alpha" and "omega"). A raw EMG signal from this
experiment is shown in Figure 1. Similar results have been achieved for
the digits ("zero" through "nine"). However, for a practical speech
recognition interface, it is probably ![](media/image3.wmf)necessary to
recognize sub-word patterns, because of the large number of words
available to speakers.

Here, we establish a baseline for sub-auditory speech recognition by
applying the approach described above to the task of recognizing *all*
the phonemes used in the English language (see Table 1).

**Data Collection**

Two subjects, both female, were used. Two symmetrical EMG sensors (two
electrodes each) were placed on the throat of each subject. Each subject
was asked to sub-vocalize each phoneme, while thinking of the target
word for that phoneme. For example, while sub-vocalizing the phoneme
***ao***, the subject would focus on the central vowel sound of the word
"dog". On each subvocalization, the subject pressed a key, to record
signal timing.

![](media/image4.wmf){width="7.097916666666666in"
height="2.5756944444444443in"}***ax***. So, the consonant phoneme
***d*** would be pronounced as ***dax***, or "duh".

Each subject generated ten sets for each phoneme, each set consisting of
approximately twelve examples. Since each of the two signal channels is
processed independently, this leads to 12x10x2=240 signals per phoneme
per subject.

### Data Processing

![](media/image5.wmf)From the raw data, 1.5 second-long samples were
automatically extracted whenever the signal exceeded a threshold. Then,
each sample was reviewed both automatically and manually, so that poor
data (e.g. no subject key press indicated, or overlapping signals) could
be removed. In order to introduce some temporal noise (which seems to
help with training the network) into the sample set, we then shifted
each sample a small amount right and left, resulting in five samples for
each original sample, Then, a dual tree wavelet transform was applied
for feature extraction. Finally, a scaled conjugate gradient neural
network was trained (on 80% of the data) and tested (on 20% of the
data).

### Results

Initially, we used a single neural network to categorize all 40 phonemes
for one subject. The resulting success rate was 18% (2500 iterations) -
considerably better than chance, but not good enough to be useful for
speech recognition. Then, we tried to categorize only the vowel
phonemes.

![](media/image6.wmf)Results from vocalized speech recognition led us to
expect very poor results for the 18 vowels, and we were pleased to
achieve a 36% success rate, using 2500 iterations in training (see Table
3). Not surprisingly, those pairs that human listeners find difficult to
discern (e.g. ***ix***, "sick", vs. ***ih***, "hit"), were also often
confused by our system.

The 22 consonants had a higher relative success rate of 33% after 1500
iterations (see Table 2). It seems that *voicing* is not easily detected
by the system, in that confusable pairs (e.g. ***d***, voiced alveolar
plosive, and ***t***, voiceless alveolar plosive) are often
distinguished only by voicing. Also, the network seems to put
difficult-to-identify items into "garbage can" categories, and seemed to
have particular difficulty with the alveolar consonants.

The above results suggested that the phonological features used to
distinguish vocalized phonemes might also be relevant for subvocalized
phonemes, and that a productive approach might be to train specialized
networks to categorize the phonemes based on these features. These
networks could then be arranged into a decision tree, which could then
be used to categorize all 40 phonemes effectively.

The next step, then, was to see if specialized networks could recognize
the key phonological features. We have not yet completed this step, but
partial results are given in Table 4. In several cases, one or more
categories had very few samples, leading to unreliable results. This
problem can only be corrected by taking more samples.

These partial results suggest that a decision tree something like the
one in Figure 3 might be effective at categorizing the full phoneme set.
If the accuracy estimates at each node are correct, then this tree would
have an overall accuracy of 50-60%. Note that both the binary features
for the vowels and the combination of manner and place of articulation
for consonants are quite redundant, so that it should be possible to
lower error rates by looking for feature combinations.

### Further Work

The immediate next step is to complete the above analysis, and determine
whether or not the decision tree approach will reduce error rates. Also,
we should experiment with sensor placement -- perhaps some problem
feature distinctions can be eliminated or reduced if the sensors are
placed to detect key muscle movements.

We have, however, achieved our primary goal, which was to establish a
baseline for sub-auditory speech recognition. Also, it seems that
features of spoken speech that are relevant to auditory speech
recognition are also relevant to sub-auditory speech recognition (i.e.
using an EMG signal). This suggests that techniques which have proven
useful in processing spoken speech, such as diphone or triphone
recognition, would also be useful in processing sub-auditory speech. We
plan to explore further in this direction.

### Bibliography

A. Chan, K. Englehart, B. Hudgins, and D.F. Lovely (2001). "Myoelectric
Signals to Augment Speech Recognition," *Medical and Biological
Engineering & Computing*, pp. 500-506 vol *39(4).*

Huang, X., Acero, A., and Hon, H-W. (2001). *Spoken Language Processing:
A guide to theory, algorithm and system development*. New Jersey,
Prentice-Hall PTR.

Jorgensen, C., Lee, D., and Agabon, S. (2003). "Sub Auditory Speech
Recognition Based on EMG/EPG Signals." In Proceedings of International
Joint Conference on Neural Networks, Portland Oregon, July 2003.