Source: http://www.google.com/patents/US6519561?ie=ISO-8859-1&dq=%235,519,867
Timestamp: 2015-03-30 13:31:19
Document Index: 18900708

Matched Legal Cases: ['application no. 60', 'application no. 60', 'application No. 60', 'art 2', 'Application no. 60', 'application no. 60']

Patent US6519561 - Model adaptation of neural tree networks and other fused models for speaker ... - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign inAdvanced Patent SearchPatentsThe model adaptation system of the present invention is a speaker verification system that embodies the capability to adapt models learned during the enrollment component to track aging of a user's voice. The system has the advantage of only requiring a single enrollment for the user. The model adaptation...http://www.google.com/patents/US6519561?utm_source=gb-gplus-sharePatent US6519561 - Model adaptation of neural tree networks and other fused models for speaker verificationAdvanced Patent SearchPublication numberUS6519561 B1Publication typeGrantApplication numberUS 09/185,871Publication dateFeb 11, 2003Filing dateNov 3, 1998Priority dateNov 3, 1997Fee statusPaidAlso published asCN1302427A, EP1027700A1, EP1027700A4, WO1999023643A1Publication number09185871, 185871, US 6519561 B1, US 6519561B1, US-B1-6519561, US6519561 B1, US6519561B1InventorsKevin Farrell, William MistrettaOriginal AssigneeT-Netix, Inc.Export CitationBiBTeX, EndNote, RefManPatent Citations (4), Non-Patent Citations (6), Referenced by (29), Classifications (13), Legal Events (9) External Links: USPTO, USPTO Assignment, EspacenetModel adaptation of neural tree networks and other fused models for speaker verification
Conventional approaches to performing text-dependent speaker verification include statistical modeling, such as hidden Markov models (HMM), or template-based modeling, such as dynamic time warping (DTW) for modeling speech. For example, subword models, as described in A. E. Rosenberg, C. H. Lee ad F. K. Soong, �Subword Unit Talker Verification Using Hidden Markov Models�, Proceedings ICASSP, pages 269-272 (1990) and whole word models, as described in A. E. Rosenberg, C. H. Lee and S. Gokeen, �Connected Word Talker Recognition Using Whole Word Hidden Markov Models�, Proceedings ICASSP, pages 381-384 (1991) have been considered for speaker verification and speech recognition systems. HMM techniques have the limitation of generally requiring a large amount of data to sufficiently estimate the model parameters.
Other approaches include the use of Neural Tree Networks (NTN). The NTN is a hierarchical classifier that combines the properties of decision trees and neural networks, as described in A. Sankar and R. J. Mammone, �Growing and Pruning Neural Tree Networks�, IEEE Transactions on Computers, C-42:221-229, Mar. 1993. For speaker recognition, training data for the NTN consists of data for the desired speaker and data from other speakers. The NTN partitions feature space into regions that are assigned probabilities which reflect how likely a speaker is to have generated a feature vector that falls within the speaker's region.
A technique not requiring prior knowledge of the number of clusters is defined as �blind� clustering. This method is disclosed in U.S. patent application Ser. No. 08/827,562 entitled �Blind Clustering of Data With Application to Speech Processing Systems�, filed on Apr. 1, 1997, and its corresponding U.S. provisional application no. 60/014,537 entitled �Blind Speech Segmentation�, filed on Apr. 2, 1996, both of which are herein incorporated by reference. In blind clustering, the number of clusters is unknown when the clustering is initiated. In the aforementioned application, an estimate of the range of the minimum number of clusters and maximum number of clusters of a data sample is determined. A clustering data sample includes objects having a common homogeneity property. An optimality criterion is defined for the estimated number of clusters. The optimality criterion determines how optimal the fit is for the estimated number of clusters to the given clustering data samples. The optimal number of clusters in the data sample is determined from the optimality criterion. The speech sample is segmented based on the optimal boundary locations between segments and the optimal number of segments.
One critical aspect of any of the above-described speaker verification systems that can directly affect its success is robustness to intersession variability and aging. Intersession variability refers to the situation where a person's voice can experience subtle changes when using a verification system from one day to the next. A user can anticipate the best performance of a speaker verification system when performing a verification immediately after enrollment. However, over time the user may experience some difficulty when using the system. For substantial periods of time, such as several months to years, the effects of aging may also degrade system performance. Whereas the spectral variation of a speaker may be small when measured over a several week period, as time passes this variance will grow as described in S. Furui, �Comparison of Speaker Recognition Methods using Statistical Features and Dynamic Features�, IEEE Transactions on Acoustics, Speech, and Signal Processing, ASSP-29:342-350, pages 342-350, April 1981. For some users, the effects of aging may render the original voice model unusable.
With the model adaptation system and method of the present invention, re-enrollment sessions are not necessary. The adaptation process is completely transparent to the user. For example, a user may telephone into his or her �Private Branch Exchange� to gain access to an unrestricted outside line. As is customary with a speaker verification system, the user may be requested to state his or her password. With the adaptation system of the present invention, this one updated utterance can be used to adapt the speaker verification model. For example, every time a user is successfully verified, the test data may be considered as enrollment data, and the models trained and modeled using the steps following segmentation. If the password is accepted by the system, the adapted system uses the updated voice features to update the particular speaker recognition model almost instantaneously. Model adaptation effectively increases the number of enrollment samples and improves the accuracy of the system.
For example, the present invention provides an adaptation system and process that adapts neural network tree (NTN) modules. The NTN is a hierarchical classifier that combines the properties of decision trees and feed-forward Neural Networks. During initial enrollment, the neural tree network learns to distinguish regions of feature space that belong to the target speaker from those that are more likely to belong to an imposter. These regions of feature space correspond to �leaves� in the neural tree network that contain probabilities. The probabilities represent the likelihood of the target speaker having generated data that falls within that region of feature space. Speaker observations within each region are determined by the number of �target vectors� landing within the region. The probability at each leaf of the NTN is computed as the ratio of speaker observations to total observations encountered at that leaf during enrollment.
Although not necessary, the adaptive modeling approach used in the present invention is preferably based on subword modeling for the NTN and GMM models. The adaptation method occurs during verification. For adapting the DTW template, it is preferred that whole-word modeling be used. As part of verification, features are first extracted for an adaptation utterance according to any conventional feature extraction method. The features are then matched, or �warped�, onto a DTW template. This provides 1) a modified set of features that best matches the DTW template and 2) a distance, or �distortion�, value that can be used as a measurement for speaker authenticity. The modified set of features output by the DTW warping has been found to remedy the negative effects of noise or speech that precedes or follows a spoken password. At this point, the warped features are used to adapt the DTW template.
Next, the feature data is segmented into sub-words for input into the NTN and GMM models. While several types of segmentation schemes can be used with the present invention, including hierarchical and nonhierarchical speech segmentation schemes, it is preferred that the spectral features be applied to a blind segmentation algorithm, such as that disclosed in U.S. patent application Ser. No. 08/827,562 entitled �Blind Clustering of Data With Application to Speech Processing Systems�, filed on Apr. 1, 1997, and its corresponding U.S. provisional application no. 60/014,537 entitled �Blind Speech Segmentation�, filed on Apr. 2, 1996, both of which are herein incorporated by reference. During enrollment in the speaker verification system, the repetition in the speaker's voice is used by the blind segmentation module to estimate the number of subwords in the password, and to locate the optimal subword boundaries.
After preprocessing, feature extraction is performed on the processed speech in module 14. Spectral features are represented by speech feature vectors determined within each frame of the processed speech signal. In the feature vector module 14, spectral feature vectors can be obtained with conventional methods such as linear predictive (LP) analysis to determine LP cepstral coefficients, Fourier Transform Analysis and filter bank analysis. One method of feature extraction is disclosed in U.S. Pat. No. 5,522,012, entitled �Speaker Identification and Verification System,� issued on May 28, 1996 and incorporated herein by reference. A preferred method for obtaining spectral feature vectors is a 12th order LP analysis to determine 12 cepstral coefficients.
Next, the feature data is warped using a dynamic time warping template 16. This removes extraneous noise or speech that precedes or follows the spoken password. The warped feature data is used for the subsequent segmentation and model evaluation. Additionally, a score is computed and stored during this warping process. This score provides a similarity measure between the spoken utterance and DTW template that can be used as a speaker verification score. This score, referred to as �x�, represents a distance value ranging between 0 and infinity. The score can be mapped onto the scale of a probability by raising its negative to an exponential, i.e., exp(−x). At this point it can be combined with the scores of the NTN and GMM to provide a third score component towards the overall model score.
Next, the speech is preferably segmented into sub-words using a blind segmentation module 18. The preferred technique for subword generation is automatic blind speech segmentation, or �Blind Clustering�, such as that disclosed in U.S. patent application Ser. No. 08/827,562 entitled �Blind Clustering of Data With Application to Speech Processing Systems�, filed on Apr. 1, 1997, and its corresponding U.S. provisional application No. 60/014,537 entitled �Blind Speech Segmentation�, filed on Apr. 2, 1996, both of which are herein incorporated by reference and assigned to the assignees of the present invention. During enrollment in the speaker verification system, the automatic blind speech segmentation determines the number of subwords in the password and the location of optimal subword boundaries. Additionally, the subword durations are normalized by the total duration of the voice phrase and stored in a database for subsequent use during verification.
The scores of the NTN 22 and GMM 26 modules can be combined to obtain a composite score for the subword in block 30. In the preferred embodiment, the results of the dynamic time warping 16, neural tree network 22 and the Gaussian mixture models 26 are combined using a linear opinion pool, as described below. Other ways of combining the data, however, can be used with the present invention including a log opinion pool or a �voting� mechanism, wherein hard decisions from the DTW 16, NTN 22 and GMM 26 are considered in the voting process. Since these three modeling approaches tend to have errors that are uncorrelated, performance improvements can be obtained by combining the model outputs.
The NTN 22 learns to distinguish regions of feature space that belong to the target speaker from those that are more likely to belong to an impostor. These regions of feature space correspond to leaves in the NTN 22 that contain probabilities. These probabilities represent the likelihood of the target speaker having generated data that falls within that region of feature space, as described in K. R. Farrell, R. J. Mammone, and K. T. Assaleh, �Speaker Recognition using Neural Networks and Conventional Classifiers�, IEEE Trans. Speech and Audio Processing, 2(1), part 2 (1994). The functioning of NTN networks with respect to speaker recognition is also disclosed in U.S. patent application Ser. No. 08/159,397, filed Nov. 29, 1993, entitled �Rapidly Trainable Neural Tree Network�, U.S. patent application Ser. No. 08/479,012 entitled �Speaker Verification System,� U.S. patent application Ser. No. 08/827,562 entitled �Blind Clustering of Data With Application to Speech Processing Systems�, filed on Apr. 1, 1997, and its corresponding U.S. Provisional Application no. 60/014,537 entitled �Blind Speech Segmentation�, filed on Apr. 2, 1996, each of which is incorporated herein by reference in its entirety. The adaptation of the NTN 22 model is described in detail below.
As discussed above, a Gaussian mixture model GMM 26 is also used to model each of the subwords. In the GMM 26, a region of feature space for a target speaker is represented by a set of multivariate Gaussian distributions. In the preferred embodiment, the mean vector and covariance matrix of the subword segments are obtained as a by-product of the blind segmentation module 18 and are saved as part of the GMM modules 26, as described in U.S. patent application Ser. No. 08/827,562 entitled �Blind Clustering of Data With Application to Speech Processing Systems�, filed on Apr. 1, 1997, and its corresponding U.S. provisional application no. 60/014,537 entitled �Blind Speech Segmentation�, filed on Apr. 2, 1996, both of which are herein incorporated by reference. The GMM probability distribution function is expressed as p  ( x / φ ) = ∑ i = 1 c  P  ( w i )  p  ( x / μ 1 , σ 1 2 ) . Each of the C mixture components is defined by a mixture weight P(wi) and normal distribution function p(x/μ1, σ1 2), where μi is the mean vector and σi is the covariance matrix. In the preferred embodiment, the normal distribution is constrained to have a diagonal covariance matrix defined by the vector σ1 2. The PDF is used to produce the sub-word GMM score.
In the preferred embodiment, a linear opinion pool method is used to combine the output scores from the DTW 16, NTN 22 and GMM 26. The linear opinion pool method computes the final score as a weighted sum of the outputs for each model: p linear  ( x ) = ∑ i = 1 n  a 1  p 1  ( x ) . Once the variables in the above equation are known, a threshold value is output and stored in the database. The threshold value output is compared to a �final score� in the testing component to determine whether a test user's voice has so closely matched the model that it can be said that the two voices are from the same person.
The NTN adaptation method can be better understood with reference to FIG. 4. The original speaker target vectors are designated as �1� in the figure. Imposter vectors are designated by a �0�. The adaptation vectors based on the updated voice utterance are those within the dashed circles 70, 74. For the left-most leaf 71 in FIG. 4, the original probability is computed as 0.6, by dividing the number of original speaker target vectors (i.e., three) by the total number of vectors (i.e., five). After applying the updated speech utterance, the adapted probability is determined to be 0.67, by dividing the speaker target vectors (i.e., 4) by the total number of vectors (i.e., 6). Advantages can also be obtained by applying more weight to the new observations.
For the second and third training scenarios, the first three training repetitions are kept fixed, while the second set of three repetitions are varied using a resampling scheme. The resampling technique is based on a leave-M�out data partitioning where M=3. For each training, three new repetitions are used. This allows for three independent training sequences for the ten available true-speaker repetitions. The fixed training repetitions used for scenarios 2 and 3 are the same as those used in scenario 1. The first scenario provides a baseline system performance, the second shows the benefit of adding speaker information to the original training, while the third shows the benefit of adapting the model using the additional speaker information.
The model adaptation techniques of the present invention can be combined with fusion adaptation and threshold adaptation, as described in copending patent application Ser. No. 08/976,280, entitled �Voice Print System and Method,� filed on Nov. 21, 1997 by Sharma et al., herein incorporated by reference. All of the adaptation techniques may effect the number and probability of obtaining false-negative and false-positive results, so should be used with caution. These adaptive techniques may be used in combination with channel adaptation, or each other, either simultaneously or at different authorization occurrences.
Patent CitationsCited PatentFiling datePublication dateApplicantTitleUS5596679 *Oct 26, 1994Jan 21, 1997Motorola, Inc.Method and system for identifying spoken sounds in continuous speech by comparing classifier outputsUS5638486 *Oct 26, 1994Jun 10, 1997Motorola, Inc.Method and system for continuous speech recognition using voting techniquesUS5835890 *Apr 9, 1997Nov 10, 1998Nippon Telegraph And Telephone CorporationMethod for speaker adaptation of speech models recognition scheme using the method and recording medium having the speech recognition method recorded thereonUS5839103 *Jun 7, 1995Nov 17, 1998Rutgers, The State University Of New JerseySpeaker verification system using decision fusion logic* Cited by examinerNon-Patent CitationsReference1Bonifas J-L et al: "Text-Dependent Speaker Verification Using Dynamic Time Warping and Vector Quantization of LSF" European Conference on Speech Communication and Technology. (Eurospeech), ES, Madrid: Graficas Brens, vol. CONF 4, Sep. 18, 1995, pp. 359-362, XP000854724.2Del Alamo C M et al: "Incremental Speaker Adaptation with Minimum Error Discriminative Training for Speaker Identification" Proceedings ICSLP 96. Fourth International Conference on Spoken Language Processing (CAT. No. 96TH8206), Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96, Philadelphia, PA, USA, Oct. 3-6, 1996, pp. 1760-1763 vol. 3, XP002154694 1996, New York, NY, USA, IEEE, USA ISBN: 0-7803-3555-4.3Furui S: "Recent Advances in Speaker Recognition" Pattern Recognition Letters, NL, North-Holland Publ. Amsterdam, vol. 18, No. 9, Sep. 1, 1997, pp. 859-872, XP004102227 ISSN: 0167-8655.4 *Kevin R. Farrell and Richard J. Mammone, "Data Fusion Techniques for Speaker Recognition," chapter 12 in Modern Methods of Speech Processing, ed. by Ravi P. Ramachandran and Richard J. Mammone, Kluwer, p. 279-297, 1995.*5 *Tomoko Matsui, "Speaker Adaptation of Tied-Mixture-Based Phoneme Models for Text-Prompted Speaker Recognition," Proc. IEEE ICASSP 94, vol. 1, p. 125-128, Apr. 1994.*6 *William Mistretta and Kevin R. Farrell, "Model Adaptation Methods for Speaker Verification," Proc. 1998 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 1998), vol. 1, p. 113-116, May 1998.** Cited by examinerReferenced byCiting PatentFiling datePublication dateApplicantTitleUS6735563 *Jul 13, 2000May 11, 2004Qualcomm, Inc.Method and apparatus for constructing voice templates for a speaker-independent voice recognition systemUS7171360 *May 7, 2002Jan 30, 2007Koninklijke Philips Electronics N.V.Background learning of speaker voicesUS7386448Jun 24, 2004Jun 10, 2008T-Netix, Inc.Biometric voice authenticationUS7437289 *Aug 16, 2001Oct 14, 2008International Business Machines CorporationMethods and apparatus for the systematic adaptation of classification systems from sparse adaptation dataUS7509257 *Dec 24, 2002Mar 24, 2009Marvell International Ltd.Method and apparatus for adapting reference templatesUS7734025Apr 29, 2003Jun 8, 2010Grape Technology Group, Inc.Methods and systems for providing on-line bills for use in communications servicesUS7809561 *Mar 28, 2007Oct 5, 2010Kabushiki Kaisha ToshibaMethod and apparatus for verification of speaker authenticationUS7953216Oct 10, 2007May 31, 20113V Technologies IncorporatedSystems and methods for RFID-based access management of electronic devicesUS7974392 *Mar 2, 2010Jul 5, 2011Research In Motion LimitedSystem and method for personalized text-to-voice synthesisUS8041571 *Jan 5, 2007Oct 18, 2011International Business Machines CorporationApplication of speech and speaker recognition tools to fault detection in electrical circuitsUS8050919 *Jun 29, 2007Nov 1, 2011Microsoft CorporationSpeaker recognition via voice sample based on multiple nearest neighbor classifiersUS8099288Feb 12, 2007Jan 17, 2012Microsoft Corp.Text-dependent speaker verificationUS8166297Jul 2, 2008Apr 24, 2012Veritrix, Inc.Systems and methods for controlling access to encrypted data stored on a mobile deviceUS8185391Sep 27, 2011May 22, 2012Microsoft CorporationSpeaker recognition via voice sample based on multiple nearest neighbor classifiersUS8185646Oct 29, 2009May 22, 2012Veritrix, Inc.User authentication for social networksUS8347370Aug 18, 2011Jan 1, 2013Veritrix, Inc.Multi-channel multi-factor authenticationUS8369499May 10, 2010Feb 5, 2013Grape Technology Group, Inc.Methods and systems for providing on-line bills for use in communication servicesUS8386248 *Sep 22, 2006Feb 26, 2013Nuance Communications, Inc.Tuning reusable software components in a speech applicationUS8468358Nov 9, 2010Jun 18, 2013Veritrix, Inc.Methods for identifying the guarantor of an applicationUS8474014Aug 16, 2011Jun 25, 2013Veritrix, Inc.Methods for the secure use of one-time passwordsUS8516562Aug 18, 2011Aug 20, 2013Veritrix, Inc.Multi-channel multi-factor authenticationUS8536976Jun 11, 2008Sep 17, 2013Veritrix, Inc.Single-channel multi-factor authenticationUS8555066Mar 6, 2012Oct 8, 2013Veritrix, Inc.Systems and methods for controlling access to encrypted data stored on a mobile deviceUS8566093 *May 16, 2006Oct 22, 2013Loquendo S.P.A.Intersession variability compensation for automatic extraction of information from voiceUS8886663 *Sep 20, 2008Nov 11, 2014Securus Technologies, Inc.Multi-party conversation analyzer and loggerUS20060100871 *Oct 20, 2005May 11, 2006Samsung Electronics Co., Ltd.Speech recognition method, apparatus and navigation systemUS20110040561 *May 16, 2006Feb 17, 2011Claudio VairIntersession variability compensation for automatic extraction of information from voiceUS20110285504 *Nov 27, 2009Nov 24, 2011Sergio Grau PuertoBiometric identity verificationEP2216775A1 *Feb 5, 2009Aug 11, 2010Harman Becker Automotive Systems GmbHSpeaker recognition* Cited by examinerClassifications U.S. Classification704/232, 704/E17.014, 704/250, 704/E17.006, 704/E15.011International ClassificationG10L15/06, G10L17/00Cooperative ClassificationG10L15/07, G10L17/04, G10L17/20European ClassificationG10L17/20, G10L17/04, G10L15/07Legal EventsDateCodeEventDescriptionJul 16, 2014FPAYFee paymentYear of fee payment: 12Aug 10, 2010FPAYFee paymentYear of fee payment: 8Aug 24, 2006ASAssignmentOwner name: USB AG. STAMFORD BRANCH, CONNECTICUTFree format text: SECURITY AGREEMENT;ASSIGNOR:NUANCE COMMUNICATIONS, INC.;REEL/FRAME:018160/0909Effective date: 20060331Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:NUANCE COMMUNICATIONS, INC.;REEL/FRAME:018160/0909Owner name: USB AG. STAMFORD BRANCH,CONNECTICUTFree format text: SECURITY AGREEMENT;ASSIGNOR:NUANCE COMMUNICATIONS, INC.;US-ASSIGNMENT DATABASE UPDATED:20100216;REEL/FRAME:18160/909Free format text: SECURITY AGREEMENT;ASSIGNOR:NUANCE COMMUNICATIONS, INC.;REEL/FRAME:18160/909Aug 16, 2006FPAYFee paymentYear of fee payment: 4Aug 16, 2006SULPSurcharge for late paymentApr 7, 2006ASAssignmentOwner name: USB AG, STAMFORD BRANCH, CONNECTICUTFree format text: SECURITY AGREEMENT;ASSIGNOR:NUANCE COMMUNICATIONS, INC.;REEL/FRAME:017435/0199Effective date: 20060331Owner name: USB AG, STAMFORD BRANCH,CONNECTICUTFree format text: SECURITY AGREEMENT;ASSIGNOR:NUANCE COMMUNICATIONS, INC.;US-ASSIGNMENT DATABASE UPDATED:20100216;REEL/FRAME:17435/199Free format text: SECURITY AGREEMENT;ASSIGNOR:NUANCE COMMUNICATIONS, INC.;US-ASSIGNMENT DATABASE UPDATED:20100309;REEL/FRAME:17435/199Feb 27, 2004ASAssignmentOwner name: SPEECHWORKS INTERNATIONAL, INC., MASSACHUSETTSFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:T-NETIX, INC.;REEL/FRAME:015017/0634Effective date: 20020702Owner name: SPEECHWORKS INTERNATIONAL, INC. 695 ATLANTIC AVENUFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:T-NETIX, INC. /AR;REEL/FRAME:015017/0634Sep 27, 1999ASAssignmentOwner name: BANK ONE COLORADO, NA, AS AGENT, COLORADOFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:T-NETIX, INC.;GATEWAY TECHNOLOGIES, INC.;T-NETIX MONITORING, INC.;AND OTHERS;REEL/FRAME:010263/0435Effective date: 19990909Jan 25, 1999ASAssignmentOwner name: T-NETIX, INC., COLORADOFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:FARRELL, KEVIN;MISTRETTA, WILLIAM;REEL/FRAME:009745/0661Effective date: 19980911RotateOriginal ImageGoogle Home - Sitemap - USPTO Bulk Downloads - Privacy Policy - Terms of Service - About Google Patents - Send FeedbackData provided by IFI CLAIMS Patent Services