Reference pattern learning system

Abstract: In a reference pattern learning system, when a first parameter set constituting reference patterns of each category in speech recognition based on pattern matching with a reference pattern is to be determined from a plurality of learning utterance data, the first parameter set is determined so that a third evaluation function represented by a sum of a first evaluation function representing a matching degree between all learning utterances and corresponding reference patterns and a second evaluation function representing a matching degree between elements of the first parameter set is maximized.

Description: Background of the Invention The present invention relates to a reference pattern learning system in speech recognition based on pattern matching with a reference pattern wherein a plurality of parameters which characterize reference patterns of each category are determined on the basis of a plurality of learning utterance data. A Hidden Markov Model (to be referred to as an HMM hereinafter) is most popular as a system for recognizing a pattern represented as a feature vector time series of, e.g., speech signals. Details of the HMM are described in "Speech Recognition by Probability Model", Seiichi Nakagawa, the Institute of Electronics and Communication Engineers of Japan, 1988 (to be referred to as Reference 1 hereinafter). In the HMM, modeling is performed on the basis of an assumption wherein a feature vector time series is generated by a Markov probability process. An HMM reference pattern is represented by a plurality of states and transitions between these states. Each state outputs a feature vector in accordance with a predetermined probability density profile, and each transition between the states accompanies a predetermined transition probability. A likelyhood value representing a matching degree between an input pattern and a reference pattern is given by a probability at which a Markov probability model as a reference pattern generates an input pattern vector sequence. An interstate transition probability characterizing each reference pattern and parameters defining a probability density profile function can be determined by a "Baum-Welch algorithm" using a plurality of learning utterance data. The "Baum-Welch algorithm" as a statistical learning algorithm requires a large volume of learning data to determine model parameters. A new user must utter a lot of speech inputs, resulting in inconvenience and impractical applications. In order to reduce the load on a new user, there are available several speaker adaptive systems for adaptively applying a recognition apparatus to a new speaker by using a relatively small number of utterances by the new speaker. Details of the speaker adaptive system are described in "Speaker Adaptive Modeling in Speech Recognition", Sadahiro Furui, Journal of the Association of Television, Vol. 43, No. 9, 1989, pp. 929 - 934 (to be referred to as Reference 2 hereinafter). The most important point in the speaker adaptive modeling system is the way of estimating parameters of a model representing an acoustic event not contained in a small number of adaptive utterances by a new user and the way of adaptively modeling using these parameters. In each of the existing speaker adaptive modeling systems, a similarity between acoustic events is basically defined using a physical distance between feature vectors as a measure, parameters of a model representing acoustic events not appearing in the adaptive utterances are estimated on the basis of the similarity, and adaptive modeling is performed using these parameters. In the existing speaker adaptive modeling systems, by using reference patterns prepared in advance and adaptive utterance data of a new user, a similarity between acoustic events is basically defined using a physical distance between feature vectors as a measure, parameters of a model representing acoustic events not appearing in the adaptive utterances are estimated on the basis of the similarity, and adaptive modeling is performed using these parameters. In adaptive modeling on the basis of estimation in accordance with the above physical distance, recognition precision can be improved as compared with that prior to adaptive modeling. However, a recognition result is far from recognition performance by reference patterns of a specific speaker which are constituted by a sufficient amount of utterance data, as can be apparent from experiment results described in the above references. Summary of the Invention It is an object of the present invention to provide a reference pattern learning system capable of estimating a high-precision reference pattern very close to a reference pattern generated by a large amount of utterance data by a specific speaker, by using data associated with correlation between all acoustic events obtained from a large number of utterances of a large number of speakers in advance in addition to adaptive utterance data obtained by a small number of utterances by a new user. In order to achieve the above object of the present invention, there is provided reference pattern learning method and apparatus as set out in claims 1,3,5 and 7. Brief Description of the Drawings Fig. 1 is a flow chart for automatically generating a reference pattern from a small amount of utterance data by a new user in accordance with a reference pattern learning system according to an embodiment of the present invention; Fig. 2 is a flow chart for automatically determining parameters for defining a likelyhood function representing a matching degree between vector parameters constituting the reference pattern, by using a large amount of utterance data by a large number of speakers, in accordance with a method according to another embodiment of the present invention; and Fig. 3 is a flow chart for automatically generating a reference pattern from a small amount of utterance data by a new user in accordance with a reference pattern learning system according to still another embodiment of the present invention. Description of the Preferred Embodiments Before preferred embodiments of the present invention will be described in detail, the principle of the present invention, i.e., a reference pattern learning system utilizing knowledge about an unspecified speaker, will be described in detail below. Assume that a reference pattern of each target recognition category is represented by a set of D-dimensional vector parameters. Each reference pattern may have additional parameters. In the following description, a model including these parameters may be easily applied. However, since additional parameters complicate mathematical expressions, the additional parameters are excluded in the following description. Sets of vector parameters constituting all reference patterns are systematized, consecutive numbers are assigned to different vector parameters, and the i-th vector parameter is defined as µ 1 . In this case, a learning evaluation function L(µ 1 , µ 2 , ..., µ N ) associated with all the vector parameters (µ 1 , µ 2 ..., µ N ) is defined as follows: L (µ 1 ,...,µ N ) = L 1 (µ 1 ,...,µ N ) + λ L 2 (µ 1 ,...,µ N ) In the above equations, the variable w represents learning utterance data, a sum of values of the variable w is a sum of all learning utterances, and λ is an appropriate constant. L 1 (µ 1 , ..., µ N ) is a likelyhood value representing a matching degree between each of all the learning utterances and a corresponding reference pattern. This value corresponds to a logarithmic likelyhood value for causing each HMM reference pattern to generate a learning utterance when the HMM is used as a recognition scheme. When DP matching or the like is used, the likelyhood value corresponds to a total sum of distances between patterns. HMM and DP matching are described in Reference 1 in detail. L 2 (µ 1 , ..., µ N ) is a likelyhood value generally representing a matching degree between all the vector parameters. However, for the sake of descriptive simplicity, L 2 can express a total sum of all vector parameter pairs (i,j) of the likelyhood value lnQ(µ i , µ j , Λ (ij) ) representing a matching degree between the vector parameters µ i and µ j . The following method is applicable to simultaneous correlation of two or more points. In the above equation, Λ (ij) is a representative of a plurality of parameters defining the likelyhood value function lnQ. The function Q can be exemplified as a simultaneous probability density profile function of probability variables (µ i ,µ j ) and can be expressed as a multidimensional Gaussian distribution. In this case, Λ (ij) is a correlation matrix with an average vector of the Gaussian distribution. Information associated with correlation between all acoustic events obtained from a large number of utterances from a large number of speakers can be expressed as this parameter Λ (ij) . An example will be described below. A plurality of vector parameter pairs obtained by a reference pattern generation method (Baum-Welch method in HMM or clustering method in DP matching) corresponding to an employed recognition method from a large number of utterances from a large number of speakers are defined as (µ i (k), µ j (k) ). In these pairs, superscript suffix (k) differentiates a plurality of vector parameters from each other. For example, when reference patterns are created in units of speakers, the suffix indicates each specific speaker. In this case, the plurality of parameters Λ (ij) defining the function Q(µ i , µ j , Λ (ij) ) are estimated to maximize the function Q by using a large number of data (µ i (k) , µ j (k) ). When a multidimensional Gaussian distribution is employed as the function Q, a mathematical expression is given as follows: Q ( µ i ,µ j ,Λ ( ij ) ) = Gauss ( z ( ij ) - w ( ij ) , R ( ij ) ) Λ ( ij ) = { w ( ij ) , R ( ij ) } Gauss ( z-w,R ) = 1 2π M /2 R 1/2 exp[- 1 2 ( z-w ) t R -1 ( z-w )] In this case, estimation of the parameter Λ (ij) corresponds to estimation of a correlation matrix (2D rows x 2D columns) R (ij) and the 2D-dimensional average vector w (ij) of the simultaneous probability density profile function for the probability variables µ i and µ j . This estimated value is obtained by the following equations from the k (k = 1 ... K) vector parameter pairs (µ i (k) ,µ j (k) ) created from the large number of utterances of the large number of speakers: When the parameter Λ (ij) representing information associated with correlation between acoustic events from a large amount of utterance data of a large number of speakers is determined, in order to determine a vector parameter µ i constituting a new reference pattern from a small amount of adaptive utterance data uttered by a new user by using this information, the learning evaluation function defined by equation (1) is maximized for µ i . In this case, Λ (ij) determined as described above is used as the evaluation function parameter to reflect knowledge associated with an unspecified speaker on learning. When the multidimensional Gaussian distribution described above is employed as the function Q in consideration of the HMM in which a state vector output probability density distribution function serves as a single Gaussian distribution function as a recognition system, maximization of the evaluation function can be performed on the basis of the following steepest descent method. In this case, the vector parameter µ i characterizing the reference pattern is given as an average vector of the Gaussian distribution of state i in HMM. HMM symbols are defined as described below, as indicated in Reference 1: where P(w) is the HMM likelyhood function for a learning utterance w (word or sentence), α t (i) is the forward probability at time t in state i , β t (i) is the backward probability at time t in state i , a ij is the transition probability between states i and j , b i (x) is the probability at which state i outputs a vector x , and the parameter Σ i represents a covariance matrix of state i . The meanings of these symbols are described in Reference 1 in detail. In order to perform the steepest descent method for the evaluation function L in equation (1), derivative functions of L by µ i are required as follows: In the steepest descent method, all the vector parameters are sequentially updated from an appropriate initial value so as to be proportional to this derivative function. An updating expression of the vector parameter µ i(n) is given as follows: µ n i = µ n -1 i + εΣ i ∂ L ∂µ i where n is the repetition count, ε is a constant, and Σ i is the covariant matrix for matching the dimensions of the derivative function. The above description has been concerned with the first and second embodiments of the present invention, and a description will be made for other embodiments. In this case, a plurality of vector parameter pairs constituted by a large amount of utterance data of a large number of speakers is defined as (µ and i (k) ,µ and j (k) ). The superscript suffix (k) discriminates the plurality of vector parameters from each other. A reference vector parameter formed by a given method is defined as µ i (i = 1 ... N). The reference vector parameter can be exemplified as a representative speaker's reference pattern. In this case, a difference vector between the kth vector parameter µ i (k) and the reference vector parameter µ i is defined as sample data Δ i (k) of a correction vector Δ i : Δ k i = µ k i - µ i The learning evaluation function is defined by the reference vector parameter µ i and the correction vector Δ i for the new speaker's reference pattern as follows: The learning target is not the vector parameter itself but its correction vector, unlike the first and second embodiments. The parameter Λ (ij) representing information obtained from the large amount of utterance data of the large number of speakers represents knowledge associated with the matching degree between the correction vectors Δ i and Δ j of the vector parameter µ i . This parameter Λ (ij) can be determined by using the sample data Δ i (k) obtained by the large number of utterances of the large number of speakers in the same manner as in the method of the second embodiment. Learning of reference patterns of new users can be performed by adding correction vectors obtained by minimizing the evaluation function to the corresponding vector parameters. More specifically, the steepest descent method described with reference to the first and second embodiments can be used. In the above description, the HMM is used as a recognition scheme. The present invention is also applicable to another speech recognition scheme based on pattern matching between input patterns and reference patterns. Preferred embodiments will be described in detail with reference to the accompanying drawings. Fig. 1 is a flow chart of reference pattern learning by a reference pattern learning scheme according to the first embodiment of the present invention. In this embodiment, the single Gaussian distribution HMM is used as a recognition scheme in the same manner as in Reference 1, and a multidimensional Gaussian distribution is employed as a likelyhood function representing a matching degree between vector parameters constituting a reference pattern. This corresponds to the calculation described above, and variable representations and the like in this embodiment denote the same as those described above. Necessary parameters are initialized in step 101. These parameters are λ, ε, µ i , Σ i , and w (ij) , and R (ij) of all i and j values. Counters are initialized in steps 102 and 103. Operations in steps 104 to 108 are performed to calculate ∂L 1 /∂Lµ 1 in accordance with equation (13). In step 104, the initial value of the first correction amount δµ(1) of the vector parameter µ i is set to zero. In step 105, the counter associated with learning utterance is initialized. In step 106, a value for HMM is calculated by equation (14). In step 107, the counter is incremented by one, and the calculations are performed in step 106 until the maximum count W is reached. In the operations in steps 109 to 114, ∂L 2 /∂µ i is calculated in accordance with equation (15). In step 109, the initial value of the second correction value δµ(2) of the vector parameter µ i is set to zero. In step 110, the counter is initialized. The second correction values are accumulated in steps 111 and 112 in accordance with equation (15). In step 113, the counter is incremented by one. Calculations are performed in step 112 until the maximum value N of the vector parameter is reached. The vector parameters are corrected in accordance with equation (16) in step 115. In step 119, convergence of the sequential correction in accordance with the steepest descent method of the vector parameters is determined. If no convergence is detected, the flow returns to step 103, and calculations of the correction values are continued. The convergence is determined in accordance with whether the sequential correction count n exceeds a predetermined constant or whether the improvement value of the evaluation function L is less than a predetermined value, or in accordance with a combination thereof. Fig. 2 is a flow chart for determining parameters representing information associated with correlation between all acoustic events from a large number of utterances of a large number of speakers according to the second embodiment of the present invention. This flow chart represents that a multidimensional Gaussian distribution is employed as a likelyhood function representing a matching degree between vector parameters constituting a reference pattern. In this case, the parameter Λ (ij) = {w (ij) ,R (ij) } for maximizing the likelyhood value can be directly calculated in accordance with equations (9) and (10). Processing will be described with reference to the flow chart. In step 201, a plurality of reference patterns determined from the large amount of utterance data from the large number of speakers are initialized. In steps 202 to 219, the average value w (ij) and the variance R (ij) are calculated in accordance with equations (9) and (10). Fig. 3 is a flow chart of reference pattern learning according to a reference pattern learning scheme according to the third embodiment of the present invention. In this flow chart, a single Gaussian distribution HMM scheme is used as a recognition scheme, and a matching degree between correction values of vector parameters constituting a reference pattern is expressed by a multidimensional Gaussian distribution. This embodiment corresponds to the calculations described with reference to the principle of the present invention, and variables in this embodiment have the same meanings as those in the above description. Processing will be described with reference to the flow chart below. In step 301, necessary parameters are initialized. These parameters are λ, ε, and µ i, Δ i , Σ i , w (ij) , and R (ij) associated with all i and j values. µ i is a representative reference pattern obtained in advance from a large amount of utterance data from a large number of speakers. In steps 302 to 319, the vector parameter µ i is substituted with the correction amount Δ i in Fig. 1. In steps 320 to 323, the calculated correction vector Δ i is added to the vector parameter µ i representing the representative reference pattern, thereby transforming the reference pattern to an optimal reference pattern for a new user. The parameters w (ij) a and R (ij) defining likelyhood function representing a matching degree between the correction amounts of the vector parameters can be automatically determined by using a large amount of utterance data of a large number of speakers according to the fourth embodiment of the present invention. When a multidimensional Gaussian distribution is employed as a likelyhood function, µ is replaced with Δ in the method of the flow chart of Fig. 2 to calculate an optimal value of the parameter. According to the present invention, as has been described above, there is provided a reference pattern learning system wherein, in addition to a small amount of adaptive utterance data produced by new users, information associated with correlation between all acoustic events obtained by a large amount of utterances of a large number of speakers is used, high-precision reference pattern very close to the reference pattern generated by a large amount of utterance data of unspecified speakers can be obtained.

Claims: A reference pattern learning method wherein a first parameter set constituting reference patterns of each category of information to be represented by a certain reference pattern in speech recognition based on pattern matching with a reference pattern is to be determined from a plurality of learning utterance data, said method being characterized in that the first parameter set is determined so that a third evaluation function represented by a sum of a first evaluation function representing a matching degree between all learning utterances and corresponding reference patterns and a second evaluation function representing a matching degree between elements of the first parameter set is maximized. A method according to claim 1, wherein a second parameter set constituting the second evaluation function is determined so that a total sum of the second evaluation functions for a plurality of reference patterns prepared in advance is maximized. A reference pattern learning method wherein a first parameter set constituting reference patterns of each category of information to be represented by a certain reference pattern in speech recognition based on pattern matching with a reference pattern is to be determined from a plurality of learning utterance data, said method being characterized in that the first parameter set is determined so that a third evaluation function represented by a sum of a first evaluation function representing a matching degree between all learning utterances and corresponding reference patterns and a second evaluation function representing a matching degree between correction values from initial values of all elements of the first parameter set is maximized. A method according to claim 3, wherein a second parameter set constituting the second evaluation function is determined so that a total sum of the second evaluation functions for the correction amounts of all elements of a plurality of first parameter sets prepared in advance is maximized. A means for learning a reference pattern to be used in speech recognition based on pattern matching with a reference pattern wherein a first parameter set constituting reference patterns of each category of information to be represented by a certain reference pattern is to be determined from a plurality of learning utterance data, characterized in that said means comprises a means for determining said first parameter set so that a third evaluation function represented by a sum of a first evaluation function representing a matching degree between all learning utterances and corresponding reference patterns and a second evaluation function representing a matching degree between elements of the first parameter set is maximized. A means for learning a reference pattern according to claim 5, further comprising a means for determining a second parameter set constituting said second evaluation function so that a total sum of the second evaluation functions for a plurality of reference patterns prepared in advance is maximized. A means for learning a reference pattern to be used in speech recognition based on pattern matching with a reference pattern wherein a first parameter set constituting reference patterns of each category of information to be represented by a certain reference pattern is to be determined from a plurality of learning utterance data, characterized in that said means comprises a means for determining said first parameter set is so that a third evaluation function represented by a sum of a first evaluation function representing a matching degree between all learning utterances and corresponding reference patterns and a second evaluation function representing a matching degree between correction values from initial values of all elements of the first parameter set is maximized. A means for learning a reference pattern according to claim 7, further comprising a means for determining a second parameter set constituting the second evaluation function so that a total sum of the second evaluation functions for the correction amounts of all elements of a plurality of first parameter sets prepared in advance is maximized.

Inventors: ISO KEN-ICHI, ISO, KEN-ICHI, ISO, KEN-ICHI, C/O NEC CORPORATION
Applicants: NIPPON ELECTRIC CO, NEC CORPORATION
Application year: 1990
Main classification: G10L5
IPC codes: G10L11, G10L15