System and method for classifying a speech signal

A system and method for classifying a speech signal within a likely speech signal class of a plurality of speech signal classes are provided. Stochastic models include a plurality of states having state transitions and output probabilities to generate state sequences which model evolutionary characteristics and durational variability of a speech signal. The method includes extracting a frame sequence, and determining a state sequence for each stochastic model with each state sequence having full state segmentation. Representative frames are determined to provide speech signal time normalization. A likely speech signal class is determined from a neural network having a plurality of inputs receiving the representative frames and a plurality of outputs corresponding to the plurality of speech signal classes. An output signal is generated based on the likely stochastic model.

TECHNICAL FIELD 
The present invention relates to systems and methods for classifying speech 
signals. 
BACKGROUND ART 
Speech is typically input to a speech recognition system using an analog 
transducer, such as a microphone, and converted to digital form. Signal 
pre-processing consists of computing a frame sequence of acoustic feature 
vectors by processing the speech samples in successive time intervals. In 
some systems, a clustering technique is used to convert these 
continuous-valued features to a sequence of discrete code words drawn from 
a code book of acoustic prototypes. Recognition of an unknown exemplar or 
speech utterance involves transforming the extracted frame sequence into 
an appropriate message. The recognition process is typically constrained 
by a set of acoustic models which correspond to the basic units of speech 
or speech signal classes employed in the recognizer, a lexicon which 
defines the vocabulary of the recognizer in terms of these basic units, 
and a language model which specifies allowable sequences of vocabulary 
items. The acoustic models, and in some cases the language model and 
lexicon, are learned from a set of representative training data or 
training exemplars. 
One recognition paradigm frequently employed in speech recognition is the 
neural network. A neural network is a parallel, distributed information 
processing structure consisting of processing elements interconnected via 
unidirectional signal channels called connections. Each processing element 
may possess a local memory and carry out localized information processing 
operations. Each processing element has many inputs and a single output 
that fans out into as many co-lateral connections as desired. The inputs 
to a processing element have a connection weight. The process of learning 
a given task by a neural network, such as recognizing a frame sequence to 
classify a speech signal, is the weight adaptation in which a connection 
weight changes as a non-linear function of the current connection weight, 
the internal excitation state of the neuron, and the current input to the 
neuron at that connection. The output of the neuron is a non-linear 
function of its internal excitation, such as the sigmoid function. 
Many neural net architectures can be trained for strong interclass 
discriminative properties. 
However, neural networks often lack the time normalization characteristics 
desired for speech signal processing. Because of speaker variability, 
different exemplars from the same speech signal class may vary in temporal 
scale. Time dilations and compressions among exemplars of the same class 
greatly reduce the reliability of the neural network due to the neural 
network's lack of time normalization characteristics. 
Time-delay neural network architectures, which are somewhat capable of time 
normalization, do exist. However, time-delay neural network architectures 
are very complex, and have not found wide acceptance in the art of speech 
recognition. Thus, using a time-delay neural network for speech 
recognition is not very practical. 
Another recognition paradigm frequently employed in speech recognition is 
the hidden Markov model. Hidden Markov modeling is a probabilistic pattern 
matching technique which is more robust than neural networks at modeling 
durational and acoustic variability among exemplars of a speech signal 
class. A hidden Markov model is a stochastic model which uses state 
transition and output probabilities to generate state sequences. Hidden 
Markov models represent speech as a sequence of states, which are assumed 
to model frames of speech with roughly stationary acoustic features. Each 
state is characterized by an output probability distribution which models 
acoustic variability in the spectral features or observations associated 
with that state. Transition probabilities between states model 
evolutionary characteristics and durational variabilities in the speech 
signal. The probabilities, or parameters, of a hidden Markov model are 
trained using frames extracted from a representative sample of speech 
data. Recognition of an unknown exemplar is based on the probability that 
the exemplar was generated by the hidden Markov model. 
One hidden Markov model based speech recognition technique involves 
determining an optimal state sequence through a hidden Markov model to 
represent an exemplar, using the Viterbi algorithm. The optimal state 
sequence is defined as the state sequence which maximizes the probability 
of the given exemplar in a particular hidden Markov model. During speech 
recognition, an optimal state sequence is determined for each of a 
plurality of hidden Markov models. Each hidden Markov model represents a 
particular speech signal class of the speech recognition system 
vocabulary. A likely hidden Markov model is selected from the plurality of 
hidden Markov models to determine the likely speech signal class. 
Training hidden Markov model based recognizers involves estimating the 
parameters for the word models used in the system. Parameters for the 
models are chosen based on a maximum likelihood criteria. That is, the 
parameters maximize the likelihood of the training data being produced by 
the model. This maximization is performed using the Baum-Welch algorithm, 
a re-estimation technique based on first aligning the training data with 
the current models, and then updating the parameters of the models based 
on this alignment. Because the hidden Markov models are trained on a 
class-by-class basis, interclass distinction may be rather poor. 
Attempts have been made to train all classes simultaneously based on 
maximum mutual information criteria. However, mathematical manipulations 
are complicated, algorithms are not very practical, and many assumptions 
must be made. Thus, training hidden Markov models for strong interclass 
distinction is not very practical. 
SUMMARY OF THE INVENTION 
It is, therefore, an object of the present invention is to provide a system 
and method for classifying a speech signal having robust time 
normalization characteristics and strong interclass distinction. 
In carrying out the above object and other objects and features of the 
present invention, a system and method for classifying a speech signal are 
provided. In a method of classifying a speech signal within a likely 
speech signal class, a plurality of speech signal classes correspond to a 
plurality of stochastic models. Each stochastic model includes a plurality 
of states having state transition and output probabilities to generate 
state sequences which model evolutionary characteristics and durational 
variabilities of the speech signal. The method comprises extracting a 
frame sequence from the speech signal, and determining a state sequence 
for each stochastic model. Each state sequence corresponds to the frame 
sequence and has full state segmentation. Each state of the sequence 
corresponds to at least one frame of the frame sequence. A likely 
stochastic model of the plurality of stochastic models is determined based 
on the state transition and output probabilities associated with the state 
sequences. 
Representative frames for the state sequence of the likely stochastic model 
are determined to provide speech signal time normalization. A likely 
speech signal class is determined from a neural network having a plurality 
of inputs receiving the representative frames, and a plurality of outputs 
corresponding to the plurality of speech signal classes. An output signal 
is generated based on the likely speech signal class, and preferably 
further based on the likely stochastic model. 
In a preferred embodiment, the stochastic models are first order hidden 
Markov models having left-to-right topology. Determining a state sequence 
for each stochastic model further comprises determining an optimal state 
sequence corresponding to the frame sequence; and when the optimal state 
sequence does not have full state segmentation, determining sub-optimal 
state sequences corresponding to the frame sequence, until a state 
sequence having full state segmentation is determined. 
Further, in a preferred embodiment, in addition to determining 
representative frames for each state of the state sequence of the likely 
stochastic model, additional representative frames are determined for at 
least one state based on the durational variability of the speech signal 
to preserve variabilities within the speech signal while normalizing 
dilations and compressions of the signal as a whole. 
Preferably, the neural network comprises a plurality of neural networks. 
Each neural network is configured based on a corresponding pair of the 
stochastic models for interclass distinction between a corresponding pair 
of speech signal classes. 
Further, in carrying out the present invention, a system for classifying a 
speech signal is provided. The system comprises stochastic model logic, 
frame logic, segmentation logic, model selection logic, time normalization 
logic, a neural network, and output logic. 
The advantages accruing to the present invention are numerous. For example, 
the stochastic models provide robust time normalization properties while 
the neural network provides strong interclass distinction. 
The above object and other objects, features, and advantages of the present 
invention are readily apparent from the following detailed description of 
the best mode for carrying out the invention when taken in connection with 
the accompanying drawings.

BEST MODE FOR CARRYING OUT THE INVENTION 
With reference to FIG. 1, a system of the present invention is generally 
indicated at 10. Frame logic 12 extracts spectral features from an input 
signal designated by I. A preliminary classifier 14 includes stochastic 
model logic 16 which represents a plurality of stochastic models. 
Segmentation and model selection logic are indicated at block 18. A 
dynamic programming algorithm is used to achieve full state segmentation 
for each stochastic model 16, and to select a likely stochastic model as a 
preliminary classifier. Time normalization logic 20 determines 
representative frames for the state sequence of the likely stochastic 
model to provide speech signal time normalization. A neural network 22 
determines a likely speech signal class. The neural network has a 
plurality of inputs receiving the representative frames, and a plurality 
of outputs corresponding to the plurality of speech signal classes. Output 
logic 24 generates an output signal, designated by O, indicative of the 
likely speech signal class or indicative of a "not classified" speech 
signal. 
The system 10 will now be described in detail. Feature extraction, at frame 
logic 12, involves computing sequences of numeric measurements, or feature 
vectors, which typically approximate the envelope of the speech spectrum. 
Spectral features can be extracted directly from the Discrete Fourier 
Transform (DFT) or computed using Linear Predictive Coding (LPC) 
techniques. Cepstral analysis can also be used to deconvolve the spectral 
envelope and the periodic voicing source. Each feature vector is computed 
from frame speech data defined by windowing samples of the signals. While 
a better spectral estimate can be obtained using more samples, the 
interval must be short enough so that the windowed signal is roughly 
stationary. 
For speech data, the number of samples is chosen such that the length of 
the interval covered by the window is approximately 25-30 milliseconds. 
The feature vectors are typically computed at a frame rate of 10-20 
milliseconds by shifting the window forward in time. Tapered windowing 
functions, such as the Hamming window, are used to reduce dependence of 
the spectral is estimate on the exact temporal position of the window. 
Spectral features are often augmented with a measure of the short time 
energy of the signal, as well as with measures of energy and spectral 
change over time. In one embodiment, 25.6 millisecond windows are used 
with a frame rate of about one-half of the window length to provide a 50% 
overlap of consecutive frames. 
Each stochastic model 16 includes a plurality of states having state 
transition and output probabilities to generate state sequences which 
model evolutionary characteristics and durational variabilities of the 
speech signal. One such stochastic model is the hidden Markov model. 
A hidden Markov model is a doubly stochastic process with an underlying 
Markov process that is not observable (the states are hidden), but can 
only be observed through another set of stochastic processes which are 
produced by the Markov process (the observations or outputs are 
probabilistic functions of the states). A sequence of observation 
O={O.sub.l, . . . , O.sub.T } is produced by a Markov state sequence 
Q={q.sub.l, . . . , q.sub.T } where each observation o.sub.t is from the 
set of X observation symbols V={v.sub.k ; 1.ltoreq.k.ltoreq.X} and each 
state q.sub.t is from the set of Y states S={S.sub.i ; 
1.ltoreq.i.ltoreq.Y}. Thus, a hidden Markov model can be characterized by: 
EQU .PI.={.pi..sub.i }, where .pi..sub.i =P(q.sub.1 =s.sub.i) is the initial 
state probability; 
EQU A={a.sub.ij }, where a.sub.ij =P(q.sub.t+l =s.sub.j .vertline.q.sub.t 
=s.sub.i) is the state transition probability; 
EQU .GAMMA.={.gamma..sub.j }, where .gamma..sub.j =P(q.sub.T =s.sub.j) is the 
last state probability; 
EQU B={b.sub.i (k)}, where b.sub.i (k)=P(o.sub.t =v.sub.k .vertline.q.sub.t =i) 
is the symbol probability; (1) 
and satisfies the probability constraints: 
##EQU1## 
Note here that the last state probability .GAMMA. is included in this 
definition. In a Markov state sequence, the last state probability models 
the different probable final states as the initial state probability does 
for the initial states. A hidden Markov model is denoted by a compact 
notation .lambda.={.pi.,A,.GAMMA.,B}. 
The training algorithm for a hidden Markov model is iterative. One salient 
characteristic of this algorithm is that if a particular state transition 
probability is initialized to zero during training, that probability will 
remain zero during the entire training procedure. As a result, a structure 
can be imposed on the state transition matrix. In left-to-right hidden 
Markov models, the observation sequence starts from state 1. From state 1, 
there are two probable transitions: self-transition to state 1 or 
transition to state 2; and so on. Preferably, embodiments of the present 
invention do not allow a skip in state. That is, all the elements in our 
initial state transition matrix except those on the main diagonal and the 
upper principal off-diagonal are preferably zero. Because the two 
probabilities in each row should sum up to one, both these probabilities 
are selected as 0.5 for initialization. 
The initial observation probabilities are selected as follows. Some of the 
training sequences are selected at random, and they are randomly segmented 
in a left-to-right fashion. From this segmentation, all the frames 
belonging to same state are used to compute multidimensional means, 
covariance matrices and mixture coefficients of the Gaussian sum 
observation probability in each state. This initialization procedure is 
repeated a number of times by exploiting the randomness in the initial 
observation probability computation. For each class, each initialization 
procedure leads to one model. The best model, in the sense of modified 
Viterbi scoring as will be described, is selected as the optimum model for 
that class. 
These estimated parameters are then used as the initial parameters for the 
Baum-Welch re-estimation algorithm. 
At segmentation and model selection logic 18, full state segmentation of 
each model 16, and model selection occur. With reference to FIG. 2a, a 
speech signal is seen as a succession of acoustic events and is generally 
indicated at 26. For example, the speech signal 26 is broken into eight 
successive events s.sub.1 through s.sub.8. Here, si is the background 
ocean noise; s.sub.2 and s.sub.3 are the events that depict the transition 
to frequency-rich mid portion of the speech signal; s.sub.4, s.sub.5 and 
s.sub.6 are the events depicting the various stages of the speech signal 
while s.sub.7 and s.sub.8 show the gradual demise of the speech signal. 
All frames of the sequence are generally indicated at 28. Frames that 
correspond to one event have the same state label, as shown. Each event 
roughly corresponds to one state of the hidden Markov model. To preserve 
the time progression of the events, left-to-right hidden Markov model 
topology, i.e. state "1" is followed by state "1" or state "2", state "2" 
is followed by state "2" or state "3", etc. is preferably employed. Thus, 
any speech signal is segmented into a fixed number of states (with full 
state segmentation) even though the speech signal could stretch over many 
numbers of frames. 
With reference to FIG. 2b, a time dilated speech signal is generally 
indicated at 32. Time dilated speech signal 32 contains the same 
information as speech signal 26 depicted in FIG. 2a. The frame sequence of 
speech signal 32 is generally indicated at 34. Both speech signals are 
recognized by the hidden Markov models due to the hidden Markov model's 
ability to model signal durational variability, that is, model a dilated 
or compressed signal. 
The full state segmentation is preferably achieved by a modified Viterbi 
algorithm. Achieving full state segmentation is essential for embodiments 
of the present invention. The straightforward application of the 
traditional Viterbi algorithm may not achieve full state segmentation in 
many situations. The Viterbi algorithm searches for the globally optimal 
state sequence. Often the globally second best or the globally third best 
etc. is the first sequence having full state segmentation. 
The objective of the modified Viterbi algorithm is to find these 
sub-optimal sequences. There are two approaches: parallel approach and 
serial approach. In the parallel approach, the trellis structure of the 
Viterbi algorithm is extended to a third dimension, the dimension of 
choice. Thus, all nodes in the second choice plane represent the globally 
second best cost, e.g., maximum likelihood probability, to reach that node 
and the transition from the previous layer. Similarly, all nodes in the 
third choice plane represent globally third best cost to reach that node 
and the transition from the previous layer; and so on. 
To track the globally optimal state sequence, all terminal nodes in the 
first plane (choice 1) are used. The node that has the highest probability 
(or the lowest cost) is selected as the terminal state, and is used for 
retracting the state sequence. To track the globally second best state 
sequence, the terminal node used to find the globally optimal state 
sequence is excluded. All terminal nodes in the first and second planes 
are used to track the globally second best state sequence. This idea is 
extended to track the globally third choice and so on until a full state 
segmentation is reached. 
The algorithm using the parallel approach is compactly written as: 
Step 0: Storage 
t=time index; 
.psi..sub.t (i,l), 
1.ltoreq.t.ltoreq.T,1.ltoreq.i.ltoreq.N,1.ltoreq.l.ltoreq.L=survivor 
terminating in state i at time t with choice l; 
.delta..sub.t (i,l), 
1.ltoreq.t.ltoreq.T,1.ltoreq.i.ltoreq.N,1.ltoreq.l.ltoreq.L=Maximum 
probability at state i at time t with choice l. 
Note that .psi..sub.t (i,l) is a three-dimensional array, and each element 
of this array stores a two-dimensional data. .delta..sub.t (i,l) is a 
three-dimensional array, and each element of this array stores a 
one-dimensional data. 
Step 1: Initialization 
.delta..sub.l (i,1)=.pi..sub.i b.sub.i (O.sub.1) for 1.ltoreq.i.ltoreq.N 
.delta..sub.1 (i,l)=0 for 1.ltoreq.i.ltoreq.N,2.ltoreq.l.ltoreq.L 
.psi..sub.l (i,l)=(0,0) for 1.ltoreq.i.ltoreq.N,1.ltoreq.l.ltoreq.L 
Step 2: Recursion 
For 2.ltoreq.t.ltoreq.T,1.ltoreq.j.ltoreq.N,1.ltoreq.l.ltoreq.L; compute 
.delta..sub.t (j,l) and .psi..sub.t (j,l). 
##EQU2## 
where (c-th) max.multidot.! denotes the c-th maximum. Step 3: Termination 
For 1.ltoreq.j.ltoreq.N,1.ltoreq.l.ltoreq.L 
##EQU3## 
Step 4: Back-tracking 
For t=T-1, T-2, . . . , 1; and P*(l), 1.ltoreq.l.ltoreq.L; 
EQU (i*.sub.t, l*.sub.t)=.PSI..sub.t+1 (i*.sub.t+1, l*.sub.t+l).(7) 
In the serial version of the modified Viterbi algorithm, all nodes traced 
by the globally optimal state sequence are updated for the second best 
cost and its associated transition. This is used to find the globally 
second best state sequence. The nodes in the second best state sequence 
are updated for the next best cost and its associated transition, and so 
on. 
The algorithm using the serial approach is compactly written as: 
Step 0: Storage 
t=time index 
c=iteration index 
.psi..sub.t (i,l) 
,1.ltoreq.t.ltoreq.T,1.ltoreq.i.ltoreq.N,1.ltoreq.l.ltoreq.L=survivor 
terminating in i.sub.t 
.delta..sub.t 
(i,l)1.ltoreq.t.ltoreq.T,1.ltoreq.i.ltoreq.N,1.ltoreq.l.ltoreq.L=survivor 
score in i.sub.t count (i,t),1.ltoreq.i.ltoreq.N,1.ltoreq.t.ltoreq.T=count 
of passes allowed at node i.sub.t 
Step 1: Initialization 
.delta..sub.1 (i,l)=.pi..sub.i b.sub.i (O.sub.l) for 1.ltoreq.i.ltoreq.N 
.delta..sub.l (i,l)=0 for 1.ltoreq.i.ltoreq.N,2.ltoreq.l.ltoreq.L 
##EQU4## 
Step 2: Pre-Recursion 
for 2.ltoreq.t.ltoreq.T,1.ltoreq.j.ltoreq.N 
##EQU5## 
Step 3: Backtracking 
##EQU6## 
where (c-th) max.multidot.! denotes the c-th maximum. for t=T-1, T-2, . . 
. , 1 
##EQU7## 
If I*={i*.sub.1 i*.sub.2. . . i*.sub.T }, the c-th optional state 
sequence, satisfies the given criteria or c exceeds limit, exit; 
otherwise, continue; 
Step 4: Forward-tracking 
for t=2,3, . . . ,T 
##EQU8## 
increase c by 1; repeat Step 3. 
It is to be appreciated that selection of the appropriate implementation of 
the modified Viterbi algorithm is dependent on the architecture employed 
in stochastic model logic 16, segmentation and model selection logic 18, 
and on the number of states in the hidden Markov models. Further, it is to 
be appreciated that other methods of achieving full state segmentation may 
be employed as desired; however, the modified Viterbi algorithm is the 
preferred method. 
After determination of a fully segmented state sequence for each of the 
models 16, a likely stochastic model is determined based on the state 
transition and output probabilities associated with the state sequences. 
For each stochastic model, the probability that the frame sequence was 
produced by that stochastic model is determined. Based on these 
probabilities, the stochastic model having the highest probability, with 
full state segmentation, is selected as the likely stochastic model. 
Time normalization logic 20 determines representative frames for the state 
sequence of the likely stochastic model to provide speech signal time 
normalization. That is, after full state segmentation, all of the 
successive frames that have the same state label, as shown in FIGS. 2a and 
2b, are represented by one average or representative frame. For example, 
all frames corresponding to state 1 will be replaced by a single average 
frame representing state 1. All of the frames corresponding to state 2 
will be replaced by a single average frame representing state 2; and so 
on. As a result, all speech signals, irrespective of temporal scale, are 
now represented by a fixed number of frames suitable for classification by 
the neural network 22. 
In other words, time normalization logic 20 removes any time dilation or 
compression from the speech signal. For example, the speech signals 
illustrated in FIGS. 2a and 2b will have identical representative frames 
after processing by time normalization logic 20. 
In a preferred embodiment, the states that have much higher durational 
probability relative to each other are assigned extra representative 
frames. Some states are more prevalent than others when an exemplar is 
labeled by its states using the modified Viterbi algorithm. Extra frames 
are proportionally assigned to states with higher duration occurrences. 
This allows time normalization logic 20 to remove any time dilation or 
compression of the speech signal as a whole, while preserving the 
evolutionary characteristics and durational variabilities within the 
speech signal. 
For example, in a hidden Markov model having seven states, six of the seven 
states may have ten frames per state after state labeling. The remaining 
state may have about twenty frames. Time normalization logic 20 assigns a 
representative frame to each state. Then, time normalization logic 20 
assigns an extra frame to the state that had the twenty frames. In this 
manner, the signal is time normalized while preserving the evolutionary 
characteristics and durational variability that is within the signal by 
assigning a total of eight representative frames: one frame for each of 
seven states plus one extra frame for the state having the higher 
durational probability. 
Similarly, six of the seven states may have about twenty frames while the 
remaining state has about forty frames. A signal such as this one would 
also result in a total of eight assigned frames: one frame for each of 
seven frames plus one extra frame for the state having the higher 
durational probability. Both of the previously described exemplary speech 
signals result in the same set of representative frames after processing 
by time normalization logic 20. 
Time normalization logic 20 will always have a fixed number of 
representative frames as its output. There will be one representative 
frame for each state of the state sequence of the likely stochastic model, 
and a fixed number of additional representative frames based on durational 
probabilities of the states. This provides a fixed number of inputs for 
the neural network 22. 
In one embodiment, the representative frame for each state has components 
which are the mean of corresponding components in all of the frames for 
that state. In another embodiment, the representative frame for each state 
has components which are the median of corresponding components of all of 
the frames for that state. Other techniques for determining the components 
of the representative frames are possible. 
It is to be appreciated that first order hidden Markov models are preferred 
to facilitate training of the models. Alternatively, if desired, higher 
order hidden Markov models having higher order state transition properties 
and/or higher order output probability distribution functions may be 
employed to model the speech signal classes. Further, it is to be 
appreciated that other stochastic models may be employed in embodiments of 
the present invention; however, hidden Markov models are preferred in the 
art of speech recognition. 
With reference to FIG. 3, a preferred embodiment of a neural network for 
systems and methods of the present invention is generally indicated at 40. 
The neural network 40 is a three-layer perceptron neural network. Neural 
network 40 has an input layer 42 which includes a plurality of neurons. 
Each neuron of the input layer 42 receives a component of a representative 
frame for the likely stochastic model. As illustrated, neuron 48 receives 
the first component of the first representative frame. Neuron 50 receives 
the d-th component of the first representative frame. That is, the 
representative frames are d dimensional feature vectors extracted by frame 
logic 12. Neuron 52 receives the first component of the second 
representative frame; and so on. All of the neurons of the first layer 42 
receive frame components in this manner, the d-th component of the M-th 
representative frame is received by neuron 54. 
Neural network 40 has a middle layer 44 including P neurons. Neural network 
40 further includes output layer 46 including N neurons. Each of the N 
neurons corresponds to a speech signal class. The neural network 
determines the likely speech signal class based on the internal 
excitations of the neurons in output layer 46. 
The learning law for the perceptron 40 is a simple error feedback. The 
network learns the associations between input and output patterns by being 
exposed to many lessons. The weights are adjusted until the desired target 
output is produced. This weight adaptation is referred to as error 
backpropagation learning law. 
In a preferred embodiment, the neural network 40 includes a plurality of 
neural networks, each neural network of the plurality of neural networks 
is configured based on a corresponding pair of the stochastic models. Each 
neural network is trained specifically for interclass distinction between 
a corresponding pair of the speech signal classes. Alternatively, one 
neural network may be employed to perform the interclass distinction. 
However, it is to be appreciated that the use of multiple neural networks 
simplifies training. It is to be appreciated that other neural network 
types may be employed in embodiments of the present invention; however, 
the multi-layered perceptron is preferred. 
The processing logic and associated neural network or networks may be 
implemented via software, firmware, hardware, microcode, or any 
combination of processing logic and neural network implementation methods 
known in the art of speech recognition. It is to be appreciated that the 
processing logic may be configured for optimum serial implementation of 
the modified Viterbi algorithm, optimum parallel implementation of the 
modified Viterbi algorithm, or any desired amount of optimization of the 
two implementations relative to one another. 
Referring to FIG. 4, a method of the present invention for classifying a 
speech signal will be described. At step 70, a sequence of frames is 
extracted from a speech exemplar. For each stochastic model, such as 
hidden Markov models, fully segmented state sequences are determined at 
step 72. At step 74, the likely stochastic model is determined. Because 
each stochastic model corresponds to a speech signal class, the likely 
stochastic model indicates a preliminary likely class. 
At step 76, a representative frame is determined for each state of the 
likely stochastic model. Preferably, additional representative frames are 
determined to model durational variability among the states as described 
previously, at step 78. 
At step 80, the likely speech signal class is determined from a neural 
network. In a preferred embodiment, the likely class is determined from a 
plurality of neural networks as described previously. 
All of the neural networks that distinguish the preliminary class from 
other classes are used to verify the integrity of the stochastic model 
preclassification. If all of the neural networks agree with the selected 
stochastic model, then the exemplar is considered "classified," and the 
likely speech signal class is the same as the preliminary speech signal 
class. If the neural networks and the stochastic model preclassifier 
disagree, or the neural networks disagree, the exemplar is considered "not 
classified," and there is not any likely speech signal class. 
Because the preliminary speech signal class may not always be correct, 
preferred embodiments of the present invention use more than one full 
state segmentation to select the stochastic model. That is, the best three 
(or any other number desired) full state segmentations are determined. If 
the three best full state segmentations are all from the same stochastic 
model, that stochastic model is selected. Otherwise, a stochastic model 
having two of the three full state segmentations may be selected. If all 
three full state segmentations correspond to different stochastic models, 
the exemplar may be immediately considered "not classified," or more full 
state segmentations may be obtained. 
Alternatively, representative frames from all three (or any other number 
desired) of the best full state segmentations may be separately introduced 
to a neural network or several neural networks. If any one of the three 
full state segmentations corresponds to a stochastic model which agrees 
with the outputs of the neural network or neural networks, that stochastic 
model corresponds to the likely speech signal class. 
It is to be appreciated that there are numerous ways to combine the results 
of the stochastic model classifier and neural network classifier to 
determine the likely speech signal class. The systems and methods of the 
present invention make it practical to combine the two paradigms to 
produce accurate speech signal classification by utilizing full state 
segmentation. 
For example, any number of fully state segmentations may be used to 
determine a plurality of possible stochastic models. For each full state 
segmentation, representative frames may then be introduced to the neural 
network or networks. The probabilities for each of the stochastic models 
corresponding to the fully segmented state sequences, and the outputs of 
the neural network or networks for each set of representative frames may 
be used to determine a likely speech signal class in a variety of ways 
which are appreciated by one of ordinary skill in the art. 
At step 82, an output is generated which is indicative of the likely speech 
signal class or is indicative of a speech signal being not classified. 
Further, additional outputs may be generated indicative of the degree of 
class distinction or integrity of classification, as appreciated by one of 
ordinary skill in the art. These additional outputs may be based on any of 
numerous combinations of stochastic model classifier and neural network 
classifier results. 
While the best mode for carrying out the invention has been described in 
detail, those familiar with the art to which this invention relates will 
recognize various alternative designs and embodiments for practicing the 
invention as defined by the following claims.