System and method for identifying semantic intent from acoustic information

In accordance with one embodiment of the present invention, unanticipated semantic intents are discovered in audio data in an unsupervised manner. For instance, the audio acoustics are clustered based on semantic intent and representative acoustics are chosen for each cluster. The human then need only listen to a small number of representative acoustics for each cluster (and possibly only one per cluster) in order to identify the unforeseen semantic intents.

BACKGROUND OF THE INVENTION

The present invention deals with identifying semantic intent in acoustic information. More specifically, the present invention deals with grouping acoustic information (such as acoustic information from call logs) into clusters, each representing a category of semantic intent.

Automatic voice response systems have gained increasing popularity in enhancing human-machine interaction. Conventional automatic voice response systems allow a user to call the system using a telephone and then navigate through a voice-responsive menu in order to receive desired information, or to be routed to a desired destination. For instance, in some such systems, a user may call to review an account summary of the user's account with a particular business. In that case, the user may navigate through an account summary menu, using voice commands, to obtain an account balance, for example.

In another such system, the user may dial the general telephone number of a company and navigate through a voice-responsive menu to reach a particular individual at the company, or to reach a department, such as “technical service”.

These types of systems have encountered a number of problems. In such systems, rules-based finite state or context free grammars (CFGs) are often used as a language model (LM) for simple, system-initiative dialog applications. This type of restricted strategy often leads to high recognition performance for in-grammar utterances, but completely fails when a user's response is not contained in the grammar.

There are at least two causes for such “out-of-grammar utterances”. First, the syntactic structure of the utterance may not be parsed consistently by the CFG. For instance, a user's response of “twentieth of July” may cause failure in a grammar which is structured to include a rule [month] [day]. Second, the user's utterance may reflect a semantic intent which was not anticipated by the author of the grammar. For instance, in a corporate voice dialer application, the grammar for the response to the opening prompt “Good morning, who would you like to contact?” may be designed to expect the user to provide a name. However, the user may instead respond by identifying a department such as “human resources.”

In sum, at the application design stage, it is difficult for an application developer to anticipate all the different ways in which a user may frame a request, which leads to the first problem. Similarly, it is difficult for an application developer to anticipate all the different semantic intents that the user may have, leading to the second problem.

Many attempts have been made to address the first problem (the difficulty in anticipating the different ways a user may frame a request) by building more robust language models. For example, hand-authored combinations of context free grammars (CFGs) with statistical language models has been attempted.

Prior attempts at solving the second problem (anticipating all the different semantic intents used by the user) typically require a large amount of transcribed and semantically annotated data from actual user calls. Of course, this tends to be extremely expensive to generate. For instance, in order to generate this type of semantically annotated data, the actual incoming calls must be recorded. Then, a human being must typically listen to all of these recordings in order to identify any semantic intents used by the caller, that were not yet expected or anticipated by the developer. However, a large company, which generates the call volumes necessary to obtain a useful quantity of data, may receive several thousand calls per day. Even if the human being only listens to the calls which failed in the interactive voice response unit (e.g., calls which ended in hang-ups) and if those calls only made up ten to twenty percent of the entire call volume, this would require the human to listen to hundreds of calls each day. This is extremely time consuming and expensive.

SUMMARY OF THE INVENTION

In accordance with one embodiment of the present invention, unanticipated semantic intents are discovered in audio data in an unsupervised manner. For instance, the audio acoustics are clustered based on semantic intent and representative acoustics are chosen for each cluster. The human then need only listen to a small number of representative acoustics for each cluster (and possibly only one per cluster) in order to identify the unforeseen semantic intents.

The acoustics are subjected to speech recognition. The clustering is then performed on the speech recognition results, as opposed to the acoustics themselves. The developer may be able to identify unknown semantic intent by reviewing the speech recognition results.

In one embodiment, the developer need not even listen to any of the acoustics to identify unanticipated semantic intents. Instead, the new semantic intents can automatically be determined by tracking whether the acoustic clusters were recognized in the speech recognition process using the application grammar or a background grammar. If they were recognized using rules from the application grammar, then the semantic intent already exists in the application grammar and is not new. However, if they were recognized using a background grammar, then the semantic intent is not represented in the application grammar and is identified as a new, or unanticipated, semantic intent.

In accordance with an embodiment, the clusters are analyzed, automatically, and possible additional rules or revisions to the application grammars or language models in the human-machine interface (such as the AVR system) are automatically suggested.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

A. Computing System Overview

The present invention relates to identifying unforeseen or unanticipated, semantic intents in acoustic data. However, before discussing the present invention in greater detail, one illustrative environment in which the present invention can be used will be discussed.

B. Acoustic Processing System Overview

FIG. 2illustrates an acoustic processing system200that can be used to cluster acoustics by semantic intent and to optionally suggest updates to a grammar in order to accommodate unanticipated semantic intents. System200includes a clustering system202which, itself, includes language model based clustering system204, ranking and filtering system206, and optional grammar updating system208.FIG. 2also shows that clustering system202is coupled to speech recognition system210and an automatic voice response (AVR) system (or application)212.

AVR system (or application)212is illustratively a human-machine interface that receives voice commands from a human being and attempts to take action based on those commands. In one illustrative embodiment, the voice commands are received by telephone. AVR system also illustratively logs (or stores) the acoustic data representative of the received audio commands. In one specific embodiment, AVR system212is an automatic attendant system deployed at a company to receive and direct calls.

Speech recognition system210is illustratively a conventional speech recognition system, and illustratively uses acoustic models that are the same as those used in clustering system202, described below. Speech recognition system210illustratively employs a large vocabulary such that it is a large, generalized vocabulary recognizer. Alternatively, speech recognition system210can include an in-domain (or context-specific) recognizer in conjunction with a large, generalized vocabulary recognizer.

Clustering system202clusters the stored acoustics, based on the speech recognition results. Each cluster is illustratively indicative of a semantic intent expressed by the acoustics in that cluster. System202can also, in one embodiment, suggest revisions to the application grammar in AVR system212.

FIG. 3is a flow diagram illustrating the overall operation of system200shown inFIG. 2. First, the acoustic information from desired call logs in AVR system212, is extracted. The call log information is represented by block214inFIG. 2and the extraction step is represented by block216inFIG. 3. The desired call log information to be extracted is represented by acoustic waveforms for which semantic intents are to be recovered. These call logs of interest can be identified in a wide variety of ways. For example, the call logs of interest can be the acoustic information corresponding to failed calls in AVR system212. It can also be a response to a particular prompt from all calls that failed in AVR system212. By failed, it is generally meant that the caller prematurely hung-up (e.g., hung-up prior to achieving a desired objective or reaching a desired dialog state in AVR system212). For instance, in one specific example, the call log information of interest can be the acoustic waveforms provided in response to a prompt “Good morning, who would you like to contact?” for all calls that ended with the caller hanging up prior to completing a task or speaking with a person. Of course, the call log information of interest212can also be all acoustics recorded over a given period of time, or a different subset of those acoustics, as desired.

Once the call log information of interest has been extracted, it is provided to speech recognition system210where speech recognition is performed on the extracted acoustics. The speech recognition results are indicated by block218inFIG. 2and the performance of speech recognition on the call log information214is indicated by block220inFIG. 3.

Speech recognition results218can take one of a variety of different forms. For instance, results218can be the one-best hypothesis recognized by speech recognition system210, the n-best hypotheses or a recognition lattice, all of which are known types of outputs from speech recognition systems. It is, of course, important that speech recognition system210cover words that are outside the application grammar used by AVR system212. This is to ensure that most words in the new or unanticipated semantic intents expressed in the extracted call logs are covered and can be recognized by speech recognition system210. However, it is not necessary that all words be within the grammar coverage of speech recognition system210, nor is it necessary to have all waveforms correctly recognized. Word level recognition can be used in the present clustering system, even if they are inaccurate recognition results, so long as acoustic waveforms with similar semantics have consistent recognition results. For instance, as long as acoustic waveforms representing the phrase “good morning” are recognized consistently as “get morning” these results can be used by clustering system202, even though they are incorrect.

Speech recognition results218are provided to clustering system202, and specifically to language model-based clustering system204. The detailed operation of language model-based clustering system204is described later with respect toFIGS. 4 and 5. Suffice it to say, for now, that the acoustic call log information of interest214which was extracted from AVR system212is clustered based on the semantic intent represented by speech recognition results218corresponding to that acoustic information, using a language model clustering approach implemented by system204. For instance, each acoustic waveform is represented by its recognition results218. A cluster is modeled by a per-cluster generative language model, which is a generative model of word sequences. The probability of an acoustic waveform, given a cluster, is basically the probability of its recognition results given the cluster language model. This is described in greater detail below with respect toFIGS. 4 and 5.

The semantically based clusters222are output by system204. The performance of language model-based clustering of acoustics based on speech recognition results218is indicated by block224inFIG. 3.

Clusters222are then ranked and filtered by system206. The clustering performed by clustering system204may result in a significant number of clusters. Therefore, it may be important to select certain of those clusters for presentation to an application developer, in order to save time and resources. This involves ranking the clusters in order of importance, filtering out unimportant or “garbage” clusters and representing a cluster in a simple and relatively self-descriptive way.

In accordance with one embodiment of the present invention, clusters222are ranked based on their frequency (i.e., based on the number of instances of utterances contained in a cluster). This information indicates how frequently a semantic intent occurs in the dataset.

Once the clusters222are ranked based on frequency, they are filtered. A cluster with a high frequency may not necessarily be relevant. For instance, there may be a relatively high number of calls that consist only of silence, noise, or other incoherent speech. These “garbage” utterances tend to be recognized as some certain function words or word sequences such as “a”, “oh”, “the”, for example. They are likely to be clustered together with a high cluster prior count. However, unlike utterances in a cluster with meaningful semantics, these garbage word sequences are seldom consistent with one another.

Therefore, in accordance with one embodiment of the present invention, a “consistency” measure is used to filter out garbage clusters. This metric can also be referred to as “compactness” as it is computed in an attempt to pick out those clusters with a large portion of constant instances, and to identify a “center” instance to represent the generative cluster. In one embodiment, a similarity measure is first defined between two utterances to be the number of word tokens they have in common, normalized by the total number of word tokens in both of their n-best decoding results. The “consistency” is then defined as the normalized, pair-wise similarity of all utterances in a cluster. The clusters with a consistency lower than a threshold value are considered “garbage” and are discarded. The threshold value can be empirically determined.

It will be recognized that there is a trade-off in setting the consistency threshold. If it set relatively high, then this enhances the likelihood that only relevant clusters will meet the consistency threshold, but the system may then discard some important or relevant clusters. If the threshold is set relatively low, then it is unlikely that the system will miss or filter out any relevant clusters, but it is more likely that it will include some garbage clusters.

Once ranking and filtering system206has ranked and filtered the clusters, it selects a central utterance to represent each remaining cluster. This utterance can be chosen to have a highest sum of similarities with all other utterances in the same cluster, or it can be chosen in other ways as well. This will likely turn out to be intuitively the most representative utterance in the cluster. The distance measure for “similarity” will illustratively be the same as that used to define consistency when filtering the clusters.

The selected clusters output by system206are represented by block226inFIG. 2, and the process of ranking and filtering clusters to obtain the selected clusters208is indicated by block228inFIG. 3.

In one illustrative embodiment, clustering system202is finished after this step and simply outputs the selected clusters226for developer review. This is indicated by block230inFIG. 3. In this embodiment (in which the selected clusters226are output), the present invention effectively helps an application developer to reduce the number of clusters which the developer is required to review, but the application developer still needs to decide, by inspecting the representative utterance(s) for each selected cluster226(or by listening to the corresponding acoustics if the speech recognition results are erroneous) whether the cluster has an unanticipated semantic intent or whether it has one that already exists in the application grammar used by AVR system212. Again, in this embodiment, once one of the selected clusters226is decided to have a new semantic intent, the application developer illustratively generates any necessary corrections to the word sequences in the cluster (in case the speech recognition results were erroneous as discussed above) and can use the cluster to learn or generate a new grammar rule based on the corrected word sequences. The new grammar rule can be generated automatically or manually. The updated grammar or rule can then be used to update the grammar used by AVR system212in order to accommodate the previously unanticipated semantic intent.

However, it will also be appreciated that the present clustering system202can include optional grammar updating system208which automatically generates a new grammar rule or updates the application grammar based on the selected clusters226. One suggested update to the application grammar can simply be the language model generated for the cluster. The top rule for the application grammar will then be given a weight which may illustratively be one minus the sum of the weights of all other newly discovered rules representing all other semantic intents. The new rule or grammar will thus be integrated into the existing application grammar and can be used by AVR system212. The updated application grammar is indicated by block232inFIG. 2, and the process of automatically updating the grammar used by the application is indicated by block234inFIG. 3.

A number of modifications can also be made to the embodiments described herein in order to assist the developer. For instance, where the selected clusters are output to the developer for review, the developer needs to decide which clusters are already represented by the application grammar and which are new (or were unanticipated). In order to do this, speech recognition system210may employ not only the large vocabulary recognizer, but may also employ the application grammar used by AVR system212. In that embodiment, if the speech recognition results218were generated using the large vocabulary grammar (or background grammar), but not the application grammar, they can be tagged as such and therefore easily identified as representing a new semantic intent (one not previously anticipated by the grammar used by AVR system212). However, if the speech recognition results218were generated by speech recognition system210using the application grammar used by AVR system212, then they can be tagged as such and easily identified as representing a semantic intent that is already covered by the application grammar.

If the results are tagged in this way, then the clusters can be identified as representing unanticipated semantic intent or previously covered semantic intent by simply counting the number of utterances in each cluster that have speech recognition results that were generated using the application grammar and those generated using the background grammar. If most of the utterances in a given cluster were generated using the background grammar, the developer may wish to determine that the cluster represents an unanticipated semantic intent. Alternatively, if most utterances corresponding to the cluster were generated using the application grammar, the developer may wish to determine that the semantic intent represented by that cluster is already covered by the application grammar. Of course, different schemes or thresholds can be used, as desired, in order to determine whether the cluster represents a new or existing semantic intent.

For instance, speech recognition results generated from the different grammars are not likely to be clustered together, since they likely do not have many lexicon items in common. Therefore, each cluster will likely have a pronounced majority of recognition results generated from one grammar, but not both. Therefore, the tag of the representative utterance may be sufficient to indicate whether the cluster represents known or unanticipated semantic intent.

Alternatively, instead of only clustering calls that ended in failure, the acoustic information for all calls to AVR system212can be used in accordance with the present invention, even if the calls succeeded. The acoustics corresponding to calls that failed can easily be tagged, as can the acoustic scores corresponding to calls that succeeded. The utterances represented by the acoustics tagged as corresponding to calls that succeeded can be assumed to contain semantic intent that is already covered by the application grammar. Those tagged as corresponding to calls that failed can be assumed to contain semantic intent that is not anticipated by the application grammar. It will of course be readily appreciated that this does not require the application grammar to be employed by the speech recognition system210, but it still allows the grammar updating system208to automatically determine whether a cluster represents unanticipated semantic intent or semantic intent that is already know by system212.

C. Clustering in More Detail

FIG. 4is a flow diagram which illustrates one illustrative embodiment of the particular language model-based clustering approach used by clustering system204, in greater detail. In one illustrative embodiment, a generative Markov model is used, where the acoustic feature sequence x of an utterance is generated from a word sequence w according to an acoustic model p(x|w), and a word sequence w is generated from a semantic intent (or cluster) c based on a per-cluster n-gram language model p(w|c). The complete likelihood of x,w and c then becomes:
p(x,w,c)=p(x|w)p(w|c)p(c),  Eq. 1

The present system illustratively trains models corresponding to semantic clusters so as to maximize the likelihood p(x). In one illustrative embodiment, a fixed acoustic model p(x|w) is used in clustering. This model is trained offline on a large set of telephony speech. Per-cluster uni-grams can be used to model p(w|c), where the sentence end probability is set to be equal among all clusters.

As previously mentioned, semantic intents are often expressed by very short utterances in telephony applications. Therefore, uni-grams can be chosen because it is believed that in such applications, a uni-gram language model corresponding to a semantic cluster has a perplexity that is not much higher than a bi-gram (or tri-gram) language model, but has a much lower computational complexity. Therefore, training in accordance with the present invention involves estimating the alphabet of the cluster c, the prior probability for semantic clusters p(c), and the language models p(w|c).

1. Model Initialization

Before discussing estimation of the language models in more detail, it should first be noted that model initialization can be important in unsupervised clustering. Therefore, the first step is to initialize models corresponding to the clusters. This is indicated by block300shown inFIG. 4. The process of initializing the clusters is indicated in greater detail by the flow diagram shown inFIG. 5, which will now be discussed, before the discussion proceeds with respect to the remaining blocks inFIG. 4.

In order to initialize the clusters, the language model based clustering system204first enumerates all vocabulary items in the speech recognition results218. This is indicated by block302inFIG. 5. For instance, assume that the speech recognition results included the utterances “operator”, “ACME operator”, and “the operator”. A cluster is thus initialized corresponding to each of the words “operator”, “ACME”, and “the”, since these are the lexical items (or words) contained in the speech recognition results218. Therefore, the number of clusters created is the same as the number of vocabulary items that have a count no less than a floor count (in the present example, the floor count is one) in the speech recognition results218, each cluster corresponding to one of the vocabulary items (or words) in the results. Creating a cluster for each enumerated vocabulary item is indicated by block304inFIG. 5.

The speech recognition results that contain these lexical items are then assigned to each of the clusters. For instance, since the speech recognition result “operator” contains the word “operator”, that utterance will be assigned only to the cluster created for the word “operator”. The utterance “ACME operator”, on the other hand, will be assigned to both the cluster created for the word “operator” and the cluster created for the word “AMCE”, since it contains both words. Similarly, the utterance “the operator” will be assigned both to the cluster created for the word “the” and the cluster created for the word “operator”.

The prior probability for each cluster p(c) corresponding to a word v is set to the normalized number of utterances containing v in that cluster. This is indicated by block306inFIG. 5. The instances of utterances are then assigned to the clusters. This is indicated by block308inFIG. 5. An n-gram language model p(w|c) for each cluster is then trained based on the word sequences wi*containing the word v (i.e., based on the instances assigned to that cluster). This is indicated by block310inFIG. 5.

2. Refining the Clusters

Once the clusters and language models are initialized as described with respect toFIG. 5, processing continues inFIG. 4where the clusters are refined and those having insufficient counts are removed. This is indicated by block312inFIG. 4.

Refining the clusters is performed by maximizing the likelihood of an acoustic dataset {xi}i=1Mconsisting of M waveforms xi. Since w and c are hidden, the EM algorithm can be applied to train the models. This can be done by reassigning each utterance xito a cluster by finding the posterior probability:

p⁡(c⁢❘⁢xi)=∑w⁢⁢p⁡(c)⁢p⁡(w⁢❘⁢c)⁢p⁡(xi⁢❘⁢w)∑c′⁢⁢∑w⁢⁢p⁡(c′)⁢p⁡(w⁢❘⁢c′)⁢p⁡(xi⁢❘⁢w)Eq.⁢2
where c a is specific cluster and c′ is a variable representing cluster i such that the sum over c′ means summing over all clusters.

Since the sum over the word sequence w at each iteration is impractical, offline recognition can be employed with a background language model (as opposed to recognition at each iteration using

∑c⁢p⁡(c)⁢p⁡(w⁢❘⁢c)).⁢Then,⁢p⁡(c⁢❘⁢xi)=p⁡(c)⁢p⁡(wi*⁢❘⁢c)∑c′⁢p⁡(c′)⁢p⁡(wi*⁢❘⁢c′)Eq.⁢3
where wi*is the recognition result for xi.

p⁡(wij⁢❘⁢xi)=p⁡(wij)⁢p⁡(xi⁢❘⁢wij)∑j′=1N⁢p⁡(wij′)⁢p⁡(xi⁢❘⁢wij′)Eq.⁢4
where p(wij) represents the background language model and p(xi|wij) represents the acoustic model.
Then,

p⁡(c,w⁢❘⁢xi)=p⁡(c)⁢p⁡(w⁢❘⁢c)⁢p⁡(xi⁢❘⁢w)∑c′,w′⁢p⁡(c′)⁢p⁡(w′⁢❘⁢c′)⁢p⁡(xi⁢❘⁢w′)Eq.⁢6
which can be approximated using recognition as:

p⁡(c,w⁢❘⁢xi)={p⁡(c⁢❘⁢xi)if⁢⁢w=wi*0if⁢⁢w≠wi*Eq.⁢7
or using an N-best list or lattices as:

p⁡(c,w⁢❘⁢xi)=⁢p⁡(wij⁢❘⁢xi)⁢p⁡(c⁢❘⁢wij)=⁢p⁡(wij⁢❘⁢xi)⁢p⁡(w)⁢p⁡(wij⁢❘⁢c)∑c′⁢p⁡(c′)⁢p⁡(wij⁢❘⁢c′)Eq.⁢8
when w is the N-best list, and otherwise:
p(c,w|xi)=0  Eq. 9
We now compute the following counts where #u(w) is defined as the number of times that the word token u occurs in the utterance w:

Computing these expected counts ψcand φc,ucorresponds to the E-step of the EM algorithm which provides sufficient statistics for the likelihood maximization. The M step thus simply includes normalizing φcto give the cluster prior probabilities p(c), and normalizing φc,uto give the class-conditional uni-gram probabilities.

p′⁡(c)=ψc∑c′⁢ψc′=ψcMEq.⁢12
Since p(w|c) is a uni-gram:

p⁡(w⁢❘⁢c)=∏k=1l⁡(w)⁢⁢pc⁡(w⁡(k))Eq.⁢13
where w(k) is the kthword in sentence w, l(w) is the length of sentence w, and pc(v) is the uni-gram probability of word v in class c. Finally:

In actual implementation, the language model used in recognition (i.e., p(w) in computing p(w|x)=p(x|w)p(w)) is decoupled from the per-cluster uni-gram language models (i.e., p(w|c)). In one specific embodiment, a task-independent large vocabulary background language model is used to compute p(w). This has the advantage that with the language model p(w) and the acoustic model p(x|w) fixed, the recognition is performed offline, only once. The obtained word sequence hypotheses and their acoustic scores are used directly in training the clusters.

In addition, in one specific embodiment, for computational efficiency, the word sequence hypothesis is restricted to a lattice, or N-best list, with p(w|xi) renormalized accordingly. In one aggressive embodiment, wherein an N-best list of length 1 is used, 1-best word sequence is obtained.

3. Model Refining with Viterbi Training

In addition, it should be noted that Viterbi training can be used instead of EM training to optimize cluster parameters. In other words, p(c|w) is renormalized to 0 or 1, depending on whether c is the best hypothesis given w.

More specifically, the following can be used:

For the 1-best case we then have:

For the embodiment in which an N-best list or lattice is used, there are two options. The first option is to choose one class as an overall choice for all hypotheses. To do this, let {tilde over (p)}(c|xi) place a probability of one on maximizer of p(c|xi) given in Equation 5 above. Then:

p⁡(c,w|xi)={p~⁡(c|xi)⁢p⁡(wij|c)∑j′=iN⁢p⁡(wij′|c)if⁢⁢w=wij0otherwiseEq.⁢17
The second option is to choose a class per hypothesis. This can be done by letting:

p⁡(c,w|xi)={p⁡(wij|xi)if⁢⁢w=wij⁢⁢and⁢⁢c⁢⁢maximizes⁢⁢p⁡(c)⁢p⁡(wij|c)0otherwiseEq.⁢18
The second option may be undesirably slow for all but very small N-best lists.

D. Merging of Clusters

At this point, some of the clusters may represent similar semantic intents. Therefore, it may be desirable to merge some of the clusters together based on a distance measure between two clusters (or between the representative language models of the two clusters), and to refine the merged clusters. This is indicated by blocks314and316inFIG. 4.

Techniques for merging and splitting clustered items have been studied in the field of text clustering. Many of these techniques are based on certain distance measures between two clusters. In one embodiment, the present invention uses a relatively low complexity distance measure based on the K-L divergence between the uni-gram distributions corresponding to two clusters. K-L divergence is explained in greater detail in T. M. Cover and J. A. Thomas,Elements of Information Theory, Wiley, (1991).

Assuming γc,uis the uni-gram probability of vocabulary item u in cluster c (γc,uis proportional to Φc,u) the distance is defined as an average of the asymmetrical K-L divergences,

D⁡(c1,c2)⁢=Δ⁢∑u⁢(γc1,u⁢log⁢γc2,uγc1,u+γc2,u⁢log⁢γc1,uγc2,u),Eq.⁢19
where u is summed over all vocabulary items appearing in clusters c1and c2, and any zero probabilities γc1,uor γc2,uare smoothed by a floor value. Two clusters c1and c2are merged if their D(c1,c2) is smaller than a threshold. Upon merging, p(w,c1,2|x)=p(w,c1|x)+p(w,c2|x) and the new model is re-estimated using these new posterior probabilities. A desired number of EM or Viterbi estimations are applied after all such pairs are merged.

In another embodiment, re-estimation can be applied after each pair is merged (the pair with the smallest divergence is to be merged first). But this can greatly increase computation and may thus be undesirable.

As an alternative to K-L divergence, the distance measure between two clusters and hence the measure to determine whether merging of two clusters should take place can be based on the EM auxiliary function. Specifically, the loss in the EM auxiliary function due to merging two clusters can be used as the distance measure between those clusters.

The EM auxiliary function is:

=∑c⁢ψc⁢log⁢⁢p′⁡(c)+∑c,v⁢ψcv⁢log⁢⁢pc′⁡(v)Eq.⁢21
using p′(c) and p′c(v) from the M-step described above:

=∑c⁢ψc⁢log⁢ψcM+∑c,v⁢ψcv⁢log⁢ψcv∑v′⁢ψcv′Eq.⁢22
If we're considering clusters c1and c2, then the unmerged auxiliary function is computed as:

The distance between c1and c2can be defined as the difference:

The loss of perplexity can also be used in determining the distance between two clusters. Perplexity is described in greater detail in the following papers: Young, Odell and Woodland,Tree-Based State Tying for High Accuracy Acoustic Modeling, ARPA, pages 307-312(March 1994); and Hwang and Huang,Shared-Distribution Hidden Markov Models for Speech Recognition, IEEE TSAP, volume 1, number 4, pages 414-420(1993).

In another embodiment, the clusters are merged until the auxiliary function changes by some predetermined amount rather than merging all clusters with a distance less than a threshold. The amount of change in the auxiliary function used to determine whether clusters are merged can be a predetermined percentage or other value empirically determined.

Further, the merging process can be repeated a plurality of times, interspersed with re-estimation. This is referred to as iterative merging.

Recall that once the similar clusters are merged, a representative utterance or label for each cluster is chosen. In one embodiment, this is based on the likelihood p(w|c) calculated for each utterance in each cluster. It should be noted that when the auxiliary function is used for merging and this likelihood is used for choosing a cluster representative, then merging, re-estimation and representative selection are all consistent (performed using the same criteria) and the implementation may thus be simpler.

It can thus be seen that the present invention provides significant advantages over prior systems. For instance, the present invention automatically clusters acoustics based on semantic intent. The present invention can also identify a representative acoustic record (or speech recognition record) representative of each cluster. Therefore, a developer need not listen to a large amount of data to identify unanticipated semantic intents in order to adapt application grammars to accommodate those semantic intents.

The present invention can also be used to suggest grammar rules or models to modify the application grammars either manually or automatically. The present invention also provides significant advantages in how it extracts data, clusters that data based on speech recognition results corresponding to that data, and trains representative models, representative of each cluster.