Method and apparatus for formant tracking using a residual model

A method of tracking formants defines a formant search space comprising sets of formants to be searched. Formants are identified for a first frame in the speech utterance by searching the entirety of the formant search space using the codebook, and for the remaining frames by searching the same space using both the codebook and the continuity constraint across adjacent frames. Under one embodiment, the formants are identified by mapping sets of formants into feature vectors and applying the feature vectors to a model. Formants are also identified by applying dynamic programming to search for the best sequence that optimally satisfies the continuity constraint required by the model.

BACKGROUND OF THE INVENTION

The present invention relates to speech recognition systems and in particular to speech recognition systems that exploit formants in speech.

In human speech, a great deal of information is contained in the first three resonant frequencies or formants of the speech signal. In particular, when a speaker is pronouncing a vowel, the frequencies and bandwidths of the formants indicate which vowel is being spoken.

To detect formants, systems of the prior art analyzed the spectral content of a frame of the speech signal. Since a formant can be at any frequency, the prior art has attempted to limit the search space before identifying a most likely formant value. Under some systems of the prior art, the search space of possible formants is reduced by identifying peaks in the spectral content of the frame. Typically, this is done by using linear predictive coding (LPC) which attempts to find a polynomial that represents the spectral content of a frame of the speech signal. Each of the roots of this polynomial represents a possible resonant frequency in the signal and thus a possible formant. Thus, using LPC, the search space is reduced to those frequencies that form roots of the LPC polynomial.

In other formant tracking systems of the prior art, the search space is reduced by comparing the spectral content of the frame to a set of spectral templates in which formants have been identified by an expert. The closest “n” templates are then selected and used to calculate the formants for the frame. Thus, these systems reduce the search space to those formants associated with the closest templates.

Although systems that reduce the search space operate efficiently, they are prone to errors because they can exclude the frequency of the actual formant when reducing the search space. In addition, because the search space is reduced based on the input signal, formants in different frames of the input signal are identified using different formant search spaces. This is less than ideal because it introduces another layer of possible errors into the search process.

Thus, a formant tracking system is needed that does not reduce the search space in such a way that the formants in different frames of the speech signal are identified using different formant search spaces.

SUMMARY OF THE INVENTION

A method of tracking formants defines a formant search space comprising sets of formants to be searched. Formants are identified for a first frame and a second frame by searching the entirety of the formant search space. Under one embodiment, the formants are identified by mapping sets of formants into feature vectors and applying the feature vectors to a model.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

FIG. 2is a graph of the frequency spectrum of a section of human speech. InFIG. 2, frequency is shown along horizontal axis200and the magnitude of the frequency components is shown along vertical axis202. The graph ofFIG. 2shows that human speech contains resonances or formants, such as first formant204, second formant206, third formant208, and fourth formant210. Each formant is described by its center frequency, F, and its bandwidth, B.

The present invention provides methods for identifying the formant frequencies and bandwidths in a speech signal.FIG. 3provides a general flow diagram for these methods.

In step300ofFIG. 3, a formant codebook is constructed by quantizing the possible formant frequencies and bandwidths to form a set of quantized values and then forming entries for different combinations of the quantized values. Thus, the resulting codebook contains entries that are vectors of formant frequencies and bandwidths. For example, if the codebook contains entries for three formants, the ith entry x[i] in the codebook would be a vector of [F1i, B1i, F2i, B2i, F3i, B3i] where F1i, F2i, and F3i, are the frequencies of the first, second, and third formants and B1i, B2i, and B3iare the bandwidths for the first, second, and third formants.

Under one embodiment, the formants and bandwidths are quantized according to the entries in Table 1 below, where Min(Hz) is the minimum value for the formant or bandwidth in Hertz, Max(Hz) is the maximum value in Hertz, and “Num. Quant.” is the number of quantization states. In most embodiments, the formant frequencies within the ranges are mapped to a mel-frequency scale and then uniformly quantized. For the bandwidths, the range between the minimum and maximum is divided by the number of quantization states to provide the separation between each of the quantization states. For example, for bandwidth B1in Table 1, the range of 260 Hz is evenly divided by the 5 quantization states such that each state is separated from the other states by 65 Hz. (i.e., 40, 105, 170, 235, 300).

The number of quantization states in Table 1 could yield a total of 1 million different sets of formants. However, because of the constraint F1<F2<F3, there are only 767,500 sets of formants in the formant search space defined by the codebook.

After the codebook has been formed, each entry x[i] in the codebook is mapped into a simulated feature vector F(x[i]) at step302. Elements used to perform this step under one embodiment of the present invention are shown inFIG. 4.

Under the embodiment ofFIG. 4, a simulated feature vector for an entry in a codebook400is formed first by generating a z-transfer function402based on the entry. This z-transfer function represents an all-pole model of a speech production system that is based only on the formants in the entry. Specifically, the z-transfer function is defined as:

H⁡(z)=G⁢∏k=1K⁢⁢1(1-zk⁢z-1)⁢(1-zk*⁢z-1)EQ.⁢1
where H(z) is the transfer function, G is a gain value which in most embodiments is set to 1, and

z=ⅇj2π⁢⁢fEQ.⁢2zk=ⅇ-π⁢⁢BkFs+j2π⁢FkFsEQ.⁢3
where f is a frequency, Fsis a sampling frequency used to sample input speech signals, Fkis the frequency of the kth formant of K formants in the entry, and Bkis the bandwidth of the kth formant.

The transfer function H(z) is then used to generate a simulated spectral distribution404representing the frequency content of a speech signal made up of only the formants in the entry. This is done by calculating the value of H(z) for a plurality of different frequencies f.

The spectral distribution is then applied to a set of filter banks406, which emphasize certain frequency bands in the spectral distribution as is well known in the art. The filtered distribution is then applied to a discrete cosine transform function408, which produces the feature vector410for the entry. The initial feature vector typically includes an element for the d.c. or 0 Hz contribution to the speech signal. To improve the performance of the system, this element is removed from the feature vector in most embodiments of the present invention.

The process described above is repeated for each entry in the codebook so that there is a separate simulated feature vector for each entry.

Once the simulated feature vectors F(x[i])410have been formed, they are used to train a residual model at step304. The residual model is a model of the differences between a set of observation training feature vectors and the simulated feature vectors. In terms of an equation:
r1=ot−F(x)  EQ. 4
where rtis the residual, otis the observed training feature vector at time t and F(x) is a simulated feature vector.

Under one embodiment, rtis modeled as a single Gaussian with mean μ and covariance Σ, where μ is a vector with a separate mean for each component of the feature vector and Σ is a diagonal covariance matrix with a separate value for each component of the feature vector.

To produce the observed training feature vectors used to train the residual model, a human speaker412generates an acoustic signal that is detected by a microphone416, which also detects additive noise414. Microphone416converts the acoustic signals into an analog electrical signal that is provided to an analog-to-digital (A/D) converter418. The analog signal is sampled by A/D converter418at the sampling frequency Fsand the resulting samples are converted into digital values. In one embodiment, A/D converter418samples the analog signal at 16 kHz with 16 bits per sample, thereby creating 32 kilobytes of speech data per second. The digital samples are provided to a frame constructor420, which groups the samples into frames. Under one embodiment, frame constructor420creates a new frame every 10 milliseconds that includes 25 milliseconds worth of data.

The frames of data are provided to a feature extractor422, which in one embodiment consists of a Fast Fourier Transform (FFT)424, filter bank426and Discrete Cosine Transform428. FFT424converts the time domain digital values into a set of frequency domain digital values representing the spectral content of the frame. The spectral content is then passed through filter bank426and discrete cosine transform428, which filter and transform the spectral content in a manner similar to the way in which filter bank406and discrete cosine transform408filter and transform the spectral distribution for the entries in the formant codebook. The output of discrete cosine transform428is a set of training feature vectors430representing the training speech signal. Note that the d.c. or 0 Hz component of the training vectors is removed if this element was removed from the simulated feature vectors.

Filter banks406and426and Discrete Cosine Transforms408and428may be replaced with other elements as long as the same processing is performed on the spectral distribution of the formant codebook entries and the spectral content of the frames of training speech. Thus, the present invention is not limited to any one particular type of feature vector.

The simulated feature vectors410and the training feature vectors430are used by a residual model trainer432to form a residual model434. Under one embodiment, residual model trainer432uses an Expectation Maximization (EM) algorithm to train the mean μ and covariance matrix Σ of the residual model. Using standard EM calculations and assuming that the hidden formant values are uniformly distributed, update equations for the mean and covariance are found to be:

μ^=1T⁢∑t=1T⁢∑i=1I⁢(ot-F⁡(x⁡[i]))·N⁡((ot-F⁡(x⁡[i]));μ′,∑′)∑i=1I⁢N⁡((ot-F⁡(x⁡[i]));μ′,∑′)⁢EQ.⁢5∑^⁢=1T⁢∑t=1T⁢∑i=1I⁢(ot-F⁡(x⁡[i])-μ^)2·N⁡((ot-F⁡(x⁡[i]));μ′,∑′)∑i=1I⁢N⁡((ot-F⁡(x⁡[i]));μ′,∑′)EQ.⁢6
where {circumflex over (μ)} is the updated mean, {circumflex over (Σ)} is the updated covariance value, N((ot−F(x└i┘)); μ′,Σ′) is the Gaussian residual model having a mean μ′ determined during a previous training iteration and a covariance matrix Σ′ determined during a previous training iteration, I is the number of entries in the codebook, and T is the number of frames in the training utterance. Note that Equations 5 and 6 are performed for each component of the mean vector and each cell of the covariance matrix. Thus, in Equations 5 and 6, the differences (ot−F(x└i┘)−{circumflex over (μ)}) and (ot−F(x└i┘)) are calculated on a component-by-component basis.

Residual model trainer432updates the mean and covariance multiple times, each time using the mean and covariance from the previous iteration to define the model used in Equations 5 and 6. After the mean and covariance reach stable values, they are stored as residual model434.

Once residual model434has been constructed it can be used in step306ofFIG. 3to identify formants in an input speech signal. A block diagram of a system for identifying formants is shown inFIG. 5.

InFIG. 5, a speech signal is generated by a speaker512. The speech signal and additive noise514are converted into a stream of feature vectors530by a microphone516, A/D converter518, frame constructor520, and feature extractor522, which consists of an FFT524, filter banks526, and a Discrete Cosine Transform528. Note that microphone516, A/D converter518, frame constructor520and feature extractor522operate in a similar manner to microphone416, A/D converter418, frame constructor420and feature extractor422ofFIG. 4. Note that if the d.c. component of the feature vectors has been removed from the simulated feature vectors, it is also removed from the input feature vectors produced by feature extractor522.

The stream of feature vectors530is provided to a formant tracker532together with residual model434and simulated feature vectors410. Formant tracker532uses one of several techniques under the present invention to identify a set of formants for each frame of the speech signal.

Under one set of techniques of the present invention, formant tracker532determines the formants for each frame independently such that the formants of a current frame are not dependent on the formants in other frames. Under one such technique, a maximum likelihood determination is made in which the formant entry x[i] in the codebook that maximizes the probability in the residual model is selected as the formant set for the frame. In terms of an equation:

The embodiment of Equation 7 is limited to finding formants that are in the quantized entries in the codebook. To avoid this limitation, a second embodiment of the present invention identifies the formants for a frame using a minimum mean squared error (MMSE) estimate that is given by:
{circumflex over (x)}MMSE=Σi=1Ix[i]N((ot−F(x[i])); μ,Σ)  EQ. 8
where equation 8 is evaluated for each component of the x[i] vector. Thus, each formant frequency and bandwidth in the final identified vector {circumflex over (x)}MMSEis a weighted sum of the formant frequencies and bandwidths in the entries in the codebook, where the weighting value is the probability generated by the residual model when using the codebook entry. Using equation 8, continuous values of the formant frequencies and bandwidths are possible.

In other embodiments of the present invention, formant tracker532utilizes continuity constraints when identifying formants. Under one such embodiment, the values of the formants at the current frame are dependent on the values of the formants at a previous frame such that:
xt=xt−1+wtEQ. 9
where xtis the set of formant frequencies and bandwidths at frame t, xt−1is the set of formant frequencies and bandwidths at previous frame t−1, and wtis a Gaussian with zero mean and a diagonal covariance Σwthat is set so that each value along the diagonal is proportional to a quantization error associated with a particular component in the vectors of the codebook. This quantization error is equal to the range of possible values for the element of the formant vector divided by the number of quantization states for that element. For example, the variance associated with the frequency of formant F1would be proportional to the ratio of the range of possible values for the frequency of formant F1(700 Hz) to the number of quantization states used to quantize the frequency of formant F1(20). Thus, the variance for the frequency of formant F1would be proportional to35.

Using this model, the probability of transitioning from a set of formants in a previous frame to a set of formants in a current frame, p(x[it]|x[it−1]), is found by applying the set of formants of the current frame to a Gaussian distribution with a mean equal to the set of formants for the previous frame and a covariance matrix equal to Σw.

The sequence of formants in a sequence of feature vectors can then be identified using a MAP estimate of:

Equation 10 can be estimated using a standard Viterbi search in which there is a separate node for each entry in the formant codebook at each frame. The search then involves moving forward through the frames, extending paths into the nodes in each new frame using equation 10. At each frame, low probability paths can be pruned, thereby reducing the number of active paths being considered. When the last frame is reached, a lattice of the top “n” paths has been produced. The most probable path is then selected and the sets of formants associated with the nodes along this path are identified as the formant sequence for the speech signal.

In a further embodiment of the present invention, the Viterbi search described above is extended to form a minimum mean square error (MMSE) estimate of the formants. Instead of selecting the most probable path at the last frame, each of the “n” best paths is used to form a MMSE estimate of the formants at each frame. For any given frame, the MMSE estimate is equal to the weighted sum of the formant nodes that have paths passing through them in the frame. The weighting value applied to a node is equal to the probability of the most likely path leading into that node times the probability of the most likely path exiting that node. In terms of an equation:

x^MMSE=∑v=1V⁢x⁡[v]⁢max⁢⁢p⁡(pathv⁢:⁢1→s)·max⁢⁢p⁡(pathv⁢:⁢s+1→T)∑v=1V⁢max⁢⁢p⁡(pathv⁢:⁢1→s)·max⁢⁢p⁡(pathv⁢:⁢s+1→T)⁢⁢whereEQ.⁢11max⁢⁢p⁡(pathv⁢:⁢1→s)=arg⁢⁢maxi1⁢…is⁢(∏t=1s⁢N⁡((ot-F⁡(x⁡[i]));μ,∑))·p⁡(x⁡[i1])·(∏t=2s⁢N⁡(x⁡[it];x⁡[it-1],∑w))EQ.⁢12max⁢⁢p⁡(pathv⁢:⁢s+1→T)=arg⁢⁢maxis⁢…iT⁢(∏t=s+1T⁢N⁡((ot-F⁡(x⁡[i]));μ,∑))·(∏t=s+1T⁢N⁡(x⁡[it];x⁡[it-1],∑w))EQ.⁢13
and V defines the set of nodes in the frame that are in paths identified by the Viterbi decoding.

Although four techniques for tracking formants using a residual model and a mapping from the formant space to the feature vector space have been described above, the present invention is not limited to these formant tracking techniques. Other techniques that utilize a residual model and/or a mapping from the formant space to the feature vector space are within the scope of the present invention.