Method and apparatus for robust speaker and speech recognition

A method of processing a speech signal comprises converting the speech signal to digital signals, converting the digital speech signal into short-time frames, applying a Fast Fourier Transform to each of the short-time frames to obtain an original spectrum, deriving a varied spectrum based on the original spectrum, applying discrete cosine transform to compute original cepstrum coefficients for the original spectrum and varied cepstrum coefficients for the varied spectrum and generating a set of frontend feature vectors for each of the short-time frames.

TECHNICAL FIELD

Example embodiments of the present disclosure relate generally to speaker and speech recognition technology, and more specifically to a method and an apparatus for robust speaker and speech recognition.

BACKGROUND

Contemporary speaker recognition and speech recognition systems typically employ Mel-frequency cepstrum coefficients as the feature representation of human speech. Mel-frequency cepstrum coefficients are usually derived by digitizing human speech and applying a shifting window to obtain short-term frames to satisfy the stationary signal assumption. For each of such frames, compute the FFT (Fast Fourier Transform) spectrum, calculate filter band energy output where the center frequencies of the bands are Mel-frequency distributed, and finally use Discrete Cosine Transform (DCT) to produce Mel-frequency cepstrum coefficients (MFCC). There is one vector of MFCC's for each frame.

MFCC's can be augmented by their first-order and second-order derivatives (expanded feature vectors) to enhance the recognition performance for speaker and speech recognition. Moreover, each MFCC can also be mean-removed in order to mitigate, e.g., channel distortion.

The above MFCC's and their expansion and/or normalization work best in a quiet environment where training and testing conditions match. For noisy environments, improvements have been achieved by incorporating some noise-robust algorithms, such as spectral subtraction.

Yet, no system works optimally both for quiet and noisy environments. For example, a noise-robust system generally yields degraded recognition accuracies when operating in a quiet condition when compared to a non-noise robust counterpart.

Thus, while advancements have been made in computer-assisted speaker/speech recognition during the past several decades, contemporary speaker/speech recognition systems are nevertheless subject to a variety of problems such as the capability of mitigating noise interference and of separating inter-speaker variability from channel distortion.

SUMMARY OF THE INVENTION

According to one exemplary embodiment of the present invention, a method of processing a speech signal comprises, digitizing the speech signal, converting the digital speech signal into short-time frames, applying a Fast Fourier Transform to each of the short-time frames to obtain an original spectrum, deriving a varied spectrum based on the original spectrum, applying discrete cosine transform to compute cepstrum coefficients for the original spectrum and cepstrum coefficients for the varied spectrum and generating a set of frontend feature vectors for each of the short-time frames.

According to one exemplary embodiment of the present invention, an apparatus of processing a speech signal comprises an analog-to-digital converter (A/D) configured to digitize the speech signal, a windowing unit configured to convert the digital speech signal into short-time frames, a spectrum analyzer unit configured to apply a Fast Fourier Transform to each of the short-time frames to obtain an original spectrum, a spectrum deriving unit configured to derive a varied spectrum based on the original spectrum and a discrete cosine transform unit configured to apply discrete cosine transform to compute cepstrum coefficients for the original spectrum and cepstrum coefficients for the varied spectrum.

According to one exemplary embodiment of the present invention, a computer program product comprises a non-transitory computer readable storage medium and computer program instructions stored therein. The computer program instructions comprise program instructions configured to digitize the speech signal by an analog-to-digital converter, convert the digital speech signal into short-time frames by a windowing unit, apply a Fast Fourier Transform to each of the short-time frames to obtain an original spectrum by a spectrum analyzer unit, derive a varied spectrum based on the original spectrum by a spectrum deriving unit and apply discrete cosine transform to compute cepstrum coefficients for the original spectrum and cepstrum coefficients for the varied spectrum, respectively, by a discrete cosine transform unit.

DETAILED DESCRIPTION OF THE EMBODIMENTS

FIG. 1illustrates a flow chart of calculating frontend feature vectors for speaker and speech recognition in accordance with an exemplary embodiment of the present invention (“example,” “exemplary” and like terms as used herein refer to “serving as an example, instance or illustration”).FIG. 2is a diagram of a signal processing system200in accordance with an exemplary embodiment of the present invention. With reference toFIGS. 1 and 2, speaker/speech recognition may comprise receiving speech signals from a speaker by a sound pickup device, for example, a microphone202(shown inFIG. 2) at step S110or a file input. The speech signal is then processed at subsequent steps. For example, at step S120, the speech signal may be converted to digital signal by an analog-to-digital converter204(shown inFIG. 2) to obtain digital speech signal.

A window function (such as Hamming window) may be applied by a windowing unit206(shown inFIG. 2) to the digital speech signals to obtain partially-overlapped frames (at S130) and conduct short-time spectral analysis at step S140. During the short-time spectral analysis, the digital speech signal may be divided into frames of, for example, about 10-30 ms over which the speech waveform is considered to be stationary. Each frame may be a short-time speech signal. In one embodiment, the microphone may pick up a one-second long audio signal. A window with size of 25 ms may be applied to obtain frames out of the one-second long audio signal. If we advance the window by an amount of 10 ms each time (i.e., with an overlap amount of 15 ms), 100 frames, or in other words, 100 short-time speech signals may be obtained. In this example, a frame may have 200 samples at a sampling rate of 8 kHz or alternatively, 400 samples using the window of the same length at a sampling rate of 16 kHz.

Each frame (namely, each short-time speech signal) may then be transformed to the frequency domain through, for example, Fast Fourier Transform by a spectrum analyzer unit208(shown inFIG. 2) to obtain a short time amplitude spectrum at step S140using linearly-spaced frequency bands. Assume that a short-time speech signal is x(t), where t is the time index. Further assume that the short time amplitude spectrum obtained by applying Fast Fourier Transform is |X(f)|, where f is the frequency index.FIG. 3illustrates the stylized Fast Fourier Transform amplitude spectrum |X(f)| of the short-time signal x(t), where Fsis the sampling frequency in Hz. Useful spectral information may be contained in the frequency range from 0 to Fs/2. The short time amplitude spectrum (i.e., |X(f)|) obtained above at step S140by the spectrum analyzer unit208may be determined as an original spectrum |X(0)(f)|.

In addition to using the original spectrum |X(0)(f)|, one embodiment proposes to derive one or more varied spectra from the original spectrum |X(0)(f)|, by taking into account of spectral subtraction, use of less noise-vulnerable spectral region, use of spectral region that carries more idiosyncratic discriminability, etc. In this embodiment, one or more varied spectra may be derived by a spectrum deriving unit212at step S150based on the original spectrum |X(0)(f)|. With reference toFIG. 2, one varied spectrum |X(1)(f)| and more than one varied spectrum |X(2)(f)| . . . |X(k)(f)| be derived by the spectrum deriving unit212based on the original spectrum |X(0)(f)|. In one embodiment, the variant may be that after applying spectrum subtraction to combat additive noise or that after removing or attenuating spectrum regions that are susceptible to acoustic environment changes but contain only little speaker information.

For the original spectrum and its derived varied spectra |X(0)(f)|, |X(1)(f)|, . . . , |X(k)(f)|, where k=0, 1, 2, . . . , K, a discrete cosine transform may be applied by a discrete cosine transform unit (e.g., a discrete cosine transform unit210) at step S160to each of the original spectrum |X(0)(f)| and to the one or more varied spectra |X(1)(f)| . . . |X(k)(f)| to produce corresponding original cepstrum coefficients and varied cepstrum coefficients) C(0)(q), C(1)(q) . . . C(k)(q), where k=0, 1, 2, . . . , K. Note that q is the index of cepstrum coefficients.

In some examples, to obtain cepstrum coefficients which approximate the human auditory systems' response more closely than the linearly-spaced frequency bands used in the original spectrum |X(0)(f)| and/or its variants |X(1)(f)| . . . |X(k)(f)|, powers of the original spectrum |X(0)(f)| and/or the varied spectra |X(1)(f)| . . . |X(k)(f)| may be mapped onto the Mel scale of frequency, followed by computing logarithms of the powers at each of the Mel frequency bands, resulting in nonlinear spaced frequency bands on the Mel scale. Then the discrete cosine transform may be applied to the log powers to produce Mel frequency cepstrum coefficients. Mel frequency spectrum is well known in the art and will not be explained in detail in this application.

After one or more sets of cepstrum coefficients (e.g., C(0)(q), C(1)(q) . . . C(k)(q)) are computed, weighting parameters β(0), β(1), . . . , β(k)that decide what set or sets of cepstrum coefficients receives more relatively heavier weights may be assigned to each cepstrum coefficient. For example, a weighting assignment unit214inFIG. 2may weight each cepstrum coefficient at step S170inFIG. 1. Weighting parameters may abide by equation: Σk=0Kβ(k)=1. In one example, when one varied cepstrum is derived, β(0), β(1)may be assigned to the original cepstrum coefficients C(0)(q) and the varied cepstrum coefficients C(1)(q), respectively. In this case, weighting parameters may abide by equation: β(0)+β(1)=1. These weighted cepstrum coefficients may then be summed by an adder216(shown inFIG. 2) at the corresponding quefrency indexes (the reciprocal of frequency, having the unit of time) at step S180to produce a set of superposed frontend feature vectors. Also with reference toFIG. 2, each of the original cepstrum coefficients C(0)(q) and varied cepstrum coefficients C(1)(q) . . . C(k)(q) may be weighted and assigned with weighting parameters β(0), β(1). . . β(K)respectively. The weighted cepstrum coefficients are then superposed together by the adder216to produce a set of superposed frontend feature vectors. The superposed frontend feature vectors of the short-time speech signal x(t) may be represented by

C~⁡(q)=∑k=0K⁢β(k)⁢C(k)⁡(q).
The superposed frontend feature vectors may then be mean-normalized by a mean normalization unit (not shown inFIG. 2) to obtain a set of mean-normalized frontend feature vectors which will be discussed in detail at step S202. The superposed frontend feature vectors and/or the mean-normalized frontend feature vectors may be augmented by their first-order derivatives. In another embodiment, second-order derivatives may be derived from the first-order derivatives by a frontend feature vector derivation unit (not shown inFIG. 2).

Cepstrum coefficients are common frontend feature vectors for speaker recognition and for speech recognition. To get better performance in noise environments, the cepstrum coefficients (e.g., original cepstrum coefficients and/or varied cepstrum coefficients) produced above may be further normalized in cepstrum domain using, for example, cepstrum mean normalization technique, to produce robust feature vectors. For speech recognition, cepstrum coefficient mean normalization of certain length (can be as long as the entire utterance) is effective in mitigating channel distortion, a distortion characteristic of convolution in the time domain (which pertains to addition in the cepstrum domain). For speaker recognition, however, cepstrum coefficient mean normalization is not as effective and efficient. The lack of success for speaker recognition is due to the fact that although cepstrum coefficient mean normalization may successfully eliminate or minimize channel distortion, it at the same time also reduces inter-speaker difference. An example is speaker's vocal tract length, which has played an important role in speaker recognition. During speech production, the vocal tract and other articulators convolute with the excitation source (the glottal source or a friction source). As a result, speaker recognition does not gain too much in accuracy or robustness by mean normalizing cepstrum coefficients.

As such, it is desirable to determine which cepstrum coefficients are insensitive to inter-speaker variability and separate cepstrum coefficients that are insensitive from those that are not. Insensitivity determination may be performed after weighting the original cepstrum coefficient and varied cepstrum coefficients at step S170and superposing the weighted cepstrum coefficients at step S180, as shown inFIG. 1. In another embodiment, the insensitivity determination may be performed prior to steps S170and S180depending on various applications.

Referring back toFIG. 1, if some of the original and varied cepstrum coefficients are determined to be sensitive to inter-speaker variability at step S190, no mean normalization is performed on these superposed cepstrum coefficients at step S200for the task of speaker recognition. In this manner, superposed frontend feature vectors may be produced at step S204.

On the other hand, if some of the original and varied cepstrum coefficients are determined to be insensitive to inter-speaker variability at step S190, the superposed cepstrum coefficients may be mean normalized at step S202, by a mean normalization unit (not shown inFIG. 2), to improve the robustness of the frontend feature vectors to channel distortions. In one embodiment, assume that there are Q+1 cepstrum coefficients, {tilde over (c)}(q), q=0, 1, 2, . . . , Q. The mean normalization may be embodied as a subtraction of the mean over a certain length, usually the entire utterance. In this case, the procedure is as follows:

where {tilde over (C)}n,qrepresents the qthcepstrum coefficient at frame n, mqis the mean of the qthcepstrum coefficient, and N is the number of frames over which the mean is calculated.

Note that qsstands for the collection of cepstrum coefficients whose mean may be preserved to maintain maximum inter-speaker difference, and that qcstands for the collection of cepstrum coefficients that may be mean-normalized to elevate the system's robustness against channel variation, thus producing mean-normalized frontend feature vectors for each short-time frame at step S204. In equations (2a) and (2b), Ĉn,qis the qthcepstrum coefficient at frame index n. From the above equations, it is seen that Ĉn,qequals to {tilde over (C)}n,qfor cepstrum indexes belonging to qs. In this case, cepstrum coefficients are selectively mean-normalized for speaker recognition.

In addition to the above subtraction approach, one may also utilize a high pass filter to rid of the mean component. Use of high-pass filter has the advantage of small latency because the process can commence as soon as the speaker starts speaking. The filter is typically implemented as a first-order difference equation in the discrete time domain:
Ĉn,q={tilde over (C)}n,q−a·{tilde over (C)}n-1,q, forqεqc, andn=1,2, . . . ,N.(3a)
Ĉn,q={tilde over (C)}n,q, forqεqs, andn=1,2, . . . ,N.(3b)
{tilde over (C)}0,q=0  (3c)

where a is a decimal number between 0 and 1 (exclusive). Often a may be set to 0.95. And equation (3c) is frequently referred to as the initialization condition.

At step206, if each short-time frame has obtained its associated superposed frontend feature vectors and/or mean-normalized frontend feature vectors, method may end at step208. Otherwise, method may proceed to step S140to perform the conversions, derivations and generations as described above.

FIG. 4shows a schematic block diagram of circuitry400, some or all of which may be included in, for example, A/D converter204, windowing unit206, spectrum analyzer unit208, discrete cosine transform unit210, spectrum deriving unit212, and/or frontend feature vector generation unit (not shown). In accordance with some example embodiments, circuitry400may include various means, such as one or more processors402, memories404, communications modules406, and/or input/output modules408.

As referred to herein, “module” includes hardware, software and/or firmware configured to perform one or more particular functions described herein. In this regard, the means of circuitry400as described herein may be embodied as, for example, circuitry, hardware elements (e.g., a suitably programmed processor, combinational logic circuit, and/or the like), a computer program product comprising computer-readable program instructions stored on a non-transitory computer-readable medium (e.g., memory404) that is executable by a suitably configured processing device (e.g., processor402), or some combination thereof.

Processor402may, for example, be embodied as various means including one or more microprocessors with accompanying digital signal processor(s), one or more processor(s) without an accompanying digital signal processor, one or more coprocessors, one or more multi-core processors, one or more controllers, processing circuitry, one or more computers, various other processing elements including integrated circuits such as, for example, an ASIC (application specific integrated circuit) or FPGA (field programmable gate array), or some combination thereof. Accordingly, although illustrated inFIG. 4as a single processor, in some embodiments, processor402may comprise a plurality of processors. The plurality of processors may be embodied on a single computing device or may be distributed across a plurality of computing devices collectively configured to function as circuitry400. The plurality of processors may be in operative communication with each other and may be collectively configured to perform one or more functionalities of circuitry400as described herein. In an example embodiment, processor402is configured to execute instructions stored in memory404or otherwise accessible to processor402. These instructions, when executed by processor402, may cause circuitry400to perform one or more of the functionalities of circuitry400as described herein.

Whether configured by hardware, firmware/software methods, or by a combination thereof, processor402may comprise an entity capable of performing operations according to embodiments of the present invention while configured accordingly. Thus, for example, when processor402is embodied as an ASIC, FPGA or the like, processor402may comprise specifically configured hardware for conducting one or more operations described herein. As another example, when processor402is embodied as an executor of instructions, such as may be stored in memory404, the instructions may specifically configure processor402to perform one or more algorithms and operations described herein.

Memory404may comprise, for example, volatile memory, non-volatile memory, or some combination thereof. Although illustrated inFIG. 4as a single memory, memory404may comprise a plurality of memory components. The plurality of memory components may be embodied on a single computing device or distributed across a plurality of computing devices. In various embodiments, memory404may comprise, for example, a hard disk, random access memory, cache memory, flash memory, a compact disc read only memory (CD-ROM), digital versatile disc read only memory (DVD-ROM), an optical disc, circuitry configured to store information, or some combination thereof. Memory404may be configured to store information, data, applications, instructions, or the like for enabling circuitry to carry out various functions in accordance with example embodiments discussed herein. For example, in at least some embodiments, memory404is configured to buffer input data for processing by processor402. Additionally or alternatively, in at least some embodiments, memory404may be configured to store program instructions for execution by processor402. Memory404may store information in the form of static and/or dynamic information. This stored information may be stored and/or used by circuitry400during the course of performing its functionalities.

Communications module406may be embodied as any device or means embodied in circuitry, hardware, a computer program product comprising computer readable program instructions stored on a computer readable medium (e.g., memory404) and executed by a processing device (e.g., processor402), or a combination thereof that is configured to receive and/or transmit data from/to another device, for example, another circuitry and/or the like. In some embodiments, communications module406(like other components discussed herein) can be at least partially embodied as or otherwise controlled by processor402. In this regard, communications module406may be in communication with processor402, such as via a bus. Communications module406may include, for example, an antenna, a transmitter, a receiver, a transceiver, network interface card and/or supporting hardware and/or firmware/software for enabling communications with another computing device. Communications module406may be configured to receive and/or transmit any data that may be stored by memory404using any protocol that may be used for communications between computing devices. Communications module406may additionally or alternatively be in communication with the memory404, input/output module408and/or any other component of circuitry400, such as via a bus.

Input/output module408may be in communication with processor402to receive audio signals and/or to produce an audible, visual and/or other output to a user. For example, for speaker recognition, an identity of a specific speaker may be provided to a user, or performing a certain action such as voiceprint for access control. For speech recognition, the output may comprise displaying transcript text on a screen or execution of a voice command. As such, input/output module408may include a microphone, a loudspeaker or earpiece, a display and/or other input/output mechanisms.

Any such computer program instructions and/or other type of code may be loaded onto a computer, processor or other programmable apparatus's circuitry to produce a machine, such that the computer, processor other programmable circuitry that executes the code may be the means for implementing various functions, including those described herein.

Embodiments of the present invention have been described above with reference to block diagrams and flowchart illustrations of methods, apparatuses, systems and computer program products. Each block of the circuit diagrams and process flowcharts, and combinations of blocks in the circuit diagrams and process flowcharts, respectively, can be implemented by various means including computer program instructions. These computer program instructions may be loaded onto a general purpose computer, special purpose computer, or other programmable data processing apparatus, such as processor402discussed above with reference toFIG. 4, to produce a machine, such that the computer program product includes the instructions which execute on the computer or other programmable data processing apparatus to create a means for implementing the functions specified in the flowchart block or blocks.

These computer program instructions may also be stored in a computer-readable storage medium (e.g., memory404) that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including computer-readable instructions for implementing the function discussed herein. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions that execute on the computer or other programmable apparatus provide steps for implementing the functions discussed herein.

Accordingly, the block diagrams and flowchart illustrations support combinations of means for performing the specified functions, combinations of steps for performing the specified functions and program instruction means for performing the specified functions. It will also be understood that each block/step of the circuit diagrams and process flowcharts, and combinations of blocks/steps in the circuit diagrams and process flowcharts, can be implemented by special purpose hardware-based computer systems that perform the specified functions or steps, or combinations of special purpose hardware and computer instructions.