Adapting general-purpose speech recognition engine output for domain-specific natural language question answering

Abstract: Speech-based natural language question-answering interfaces to enterprise systems are gaining a lot of attention. General-purpose speech engines can be integrated with NLP systems to provide such interfaces. Usually, general-purpose speech engines are trained on large `general' corpus. However, when such engines are used for specific domains, they may not recognize domain-specific words well, and may produce erroneous output. Further, the accent and the environmental conditions in which the speaker speaks a sentence may induce the speech engine to inaccurately recognize certain words. The subsequent natural language question-answering does not produce the requisite results as the question does not accurately represent what the speaker intended. Thus, the speech engine's output may need to be adapted for a domain before further natural language processing is carried out. We present two mechanisms for such an adaptation, one based on evolutionary development and the other based on machine learning, and show how we can repair the speech-output to make the subsequent natural language question-answering better.

Speech-enabled natural-language question-answering interfaces to enterprise application systems, such as Incident-logging systems, Customer-support systems, Marketing-opportunities systems, Sales data systems etc., are designed to allow end-users to speak-out the problems/questions that they encounter and get automatic responses. The process of converting human spoken speech into text is performed by an Automatic Speech Recognition (ASR) engine. While functional examples of ASR with enterprise systems can be seen in day-to-day use, most of these work under constraints of a limited domain, and/or use of additional domain-specific cues to enhance the speech-to-text conversion process. Prior speech-and-natural language interfaces for such purposes have been rather restricted to either Interactive Voice Recognition (IVR) technology, or have focused on building a very specialized speech engine with domain specific terminology that recognizes key-words in that domain through an extensively customized language model, and trigger specific tasks in the enterprise application system. This makes the interface extremely specialized, rather cumbersome and non-adaptable for other domains. Further, every time a new enterprise application requires a speech and natural language interface, one has to redevelop the entire interface again.
An alternative to domain-specific speech recognition engines has been to re-purpose general-purpose speech recognition engines, such as Google Speech API, IBM Watson Speech to text API which can be used across domains with natural language question answering systems. Such general-purpose automatic speech engines (gp-ASR) are deep trained on very large general corpus using deep neural network (DNN) techniques. The deep learnt acoustic and language models enhance the performance of a ASR. However, this comes with its own limitations. For freely spoken natural language sentences, the typical recognition accuracy achievable even for state-of-the-art speech recognition systems have been observed to be about 60% to 90% in real-world environments BIBREF0 . The recognition is worse if we consider factors such as domain-specific words, environmental noise, variations in accent, poor ability to express on the part of the user, or inadequate speech and language resources from the domain to train such speech recognition systems. The subsequent natural language processing, such as that in a question answering system, of such erroneously and partially recognized text becomes rather problematic, as the domain terms may be inaccurately recognized or linguistic errors may creep into the sentence. It is, hence, important to improve the accuracy of the ASR output text.
In this paper, we focus on the issues of using a readily available gp-ASR and adapting its output for domain-specific natural language question answering BIBREF1 . We present two mechanisms for adaptation, namely
We present the results of these two adaptation and gauge the usefulness of each mechanism. The rest of the paper is organized as follows, in Section SECREF2 we briefly describe the work done in this area which motivates our contribution. The main contribution of our work is captured in Section SECREF3 and we show the performance of our approach through experiments in Section SECREF4 . We conclude in Section SECREF5 .
Most work on ASR error detection and correction has focused on using confidence measures, generally called the log-likelihood score, provided by the speech recognition engine; the text with lower confidence is assumed to be incorrect and subjected to correction. Such confidence based methods are useful only when we have access to the internals of a speech recognition engine built for a specific domain. As mentioned earlier, use of domain-specific engine requires one to rebuild the interface every time the domain is updated, or a new domain is introduced. As mentioned earlier, our focus is to avoid rebuilding the interface each time the domain changes by using an existing ASR. As such our method is specifically a post-ASR system. A post-ASR system provides greater flexibility in terms of absorbing domain variations and adapting the output of ASR in ways that are not possible during training a domain-specific ASR system BIBREF2 .
Note that an erroneous ASR output text will lead to an equally (or more) erroneous interpretation by the natural language question-answering system, resulting in a poor performance of the overall QA system
Machine learning classifiers have been used in the past for the purpose of combining features to calculate a confidence score for error detection. Non-linguistic and syntactic knowledge for detection of errors in ASR output, using a support vector machine to combine non-linguistic features was proposed in BIBREF3 and Naive Bayes classifier to combine confidence scores at a word and utterance level, and differential scores of the alternative hypotheses was used in BIBREF4 Both BIBREF3 and BIBREF4 rely on the availability of confidence scores output by the ASR engine. A syllable-based noisy channel model combined with higher level semantic knowledge for post recognition error correction, independent of the internal confidence measures of the ASR engine is described in BIBREF5 . In BIBREF6 the authors propose a method to correct errors in spoken dialogue systems. They consider several contexts to correct the speech recognition output including learning a threshold during training to decide when the correction must be carried out in the context of a dialogue system. They however use the confidence scores associated with the output text to do the correction or not. The correction is carried using syntactic-semantic and lexical models to decide whether a recognition result is correct.
In BIBREF7 the authors proposes a method to detect and correct ASR output based on Microsoft N-Gram dataset. They use a context-sensitive error correction algorithm for selecting the best candidate for correction using the Microsoft N-Gram dataset which contains real-world data and word sequences extracted from the web which can mimic a comprehensive dictionary of words having a large and all-inclusive vocabulary.
In BIBREF8 the authors assume the availability of pronunciation primitive characters as the output of the ASR engine and then use domain-specific named entities to establish the context, leading to the correction of the speech recognition output. The patent BIBREF9 proposes a manual correction of the ASR output transcripts by providing visual display suggesting the correctness of the text output by ASR. Similarly, BIBREF10 propose a re-ranking and classification strategy based on logistic regression model to estimate the probability for choosing word alternates to display to the user in their framework of a tap-to-correct interface.
Our proposed machine learning based system is along the lines of BIBREF5 but with differences: (a) while they use a single feature (syllable count) for training, we propose the use of multiple features for training the Naive Bayes classifier, and (b) we do not perform any manual alignment between the ASR and reference text – this is done using an edit distance based technique for sentence alignment. Except for BIBREF5 all reported work in this area make use of features from the internals of the ASR engine for ASR text output error detection.
We assume the use of a gp-ASR in the rest of the paper. Though we use examples of natural language sentences in the form of queries or questions, it should be noted that the description is applicable to any conversational natural language sentence.
In this paper we focus on question answering interfaces to enterprise systems, though our discussion is valid for any kind of natural language processing sentences that are not necessarily a query. For example, suppose we have a retail-sales management system domain, then end-users would be able to query the system through spoken natural language questions ( INLINEFORM0 ) such as INLINEFORM1 
A perfect ASR would take INLINEFORM0 as the input and produce ( INLINEFORM1 ), namely, INLINEFORM2 
We consider the situation where a ASR takes such a sentence ( INLINEFORM0 ) spoken by a person as input, and outputs an inaccurately recognized text ( INLINEFORM1 ) sentence. In our experiments, when the above question was spoken by a person and processed by a popular ASR engine such as Google Speech API, the output text sentence was ( INLINEFORM2 ) INLINEFORM3 
Namely INLINEFORM0 
 It should be noted that an inaccurate output by the ASR engine maybe the result of various factors such as background noise, accent of the person speaking the sentence, the speed at which he or she is speaking the sentence, domain-specific words that are not part of popular vocabulary etc. The subsequent natural language question answering system cannot answer the above output sentence from its retail sales data. Thus the question we tackle here is – how do we adapt or repair the sentence ( INLINEFORM0 ) back to the original sentence ( INLINEFORM1 ) as intended by the speaker. Namely INLINEFORM2 
 We present two mechanisms for adaptation or repair of the ASR output, namely INLINEFORM0 , in this paper: (a) an evolutionary development based artificial development mechanism, and (b) a machine-learning mechanism.
In the machine learning based mechanism of adaptation, we assume the availability of example pairs of INLINEFORM0 namely (ASR output, the actual transcription of the spoken sentence) for training. We further assume that such a machine-learnt model can help repair an unseen ASR output to its intended correct sentence. We address the following hypothesis
Using the information from past recorded errors and the corresponding correction, can we learn how to repair (and thus adapt to a new domain) the text after ASR?
Note that this is equivalent to, albiet loosely, learning the error model of a specific ASR. Since we have a small training set, we have used the Naive Bayes classifier that is known to perform well for small datasets with high bias and low variance. We have used the NLTK BIBREF11 Naive Bayes classifier in all our experiments.
Let INLINEFORM0 be the erroneous text (which is the ASR output), INLINEFORM1 the corresponding reference text (which is the textual representation of the spoken sentence) and INLINEFORM2 a feature extractor, such that DISPLAYFORM0 
where DISPLAYFORM0 
is a set of INLINEFORM0 features extracted from INLINEFORM1 . Suppose there are several pairs say ( INLINEFORM2 , INLINEFORM3 ) for INLINEFORM4 . Then we can derive INLINEFORM5 for each INLINEFORM6 using ( EQREF7 ). The probability that INLINEFORM7 belongs to the class INLINEFORM8 can be derived through the feature set INLINEFORM9 as follows. INLINEFORM10 
where INLINEFORM0 is the apriori probability of the class INLINEFORM1 and INLINEFORM2 is the probability of occurrence of the features INLINEFORM3 in the class INLINEFORM4 , and INLINEFORM5 is the overall probability of the occurrence of the feature set INLINEFORM6 . Making naive assumption of independence in the features INLINEFORM7 we get DISPLAYFORM0 
In our experiments, the domain specific reference text INLINEFORM0 was spoken by several people and the spoken speech was passed through a general purpose speech recognition engine (ASR) that produced a (possibly) erroneous hypothesis INLINEFORM1 . Each pair of reference and the ASR output (i.e. hypothesis) was then word aligned using edit distance, and the mismatching pairs of words were extracted as INLINEFORM2 pairs. For example, if we have the following spoken sentence: INLINEFORM3 
and the corresponding true transcription INLINEFORM0 
One of the corresponding ASR output INLINEFORM0 was INLINEFORM1 
In this case the INLINEFORM0 pairs are (dear, beer) and (have, has). As another example consider that INLINEFORM1 was spoken but INLINEFORM2 was recognized by the ASR. INLINEFORM3 INLINEFORM4 
Clearly, in this case the INLINEFORM0 pair is (than twenty, jewelry).
Let us assume two features, namely, INLINEFORM0 in ( EQREF7 ) is of dimension INLINEFORM1 . Let the two features be INLINEFORM2 . Then, for the INLINEFORM3 pair (than twenty, jewelry) we have INLINEFORM4 
since the number of words in than twenty is 2 and than twenty contains 3 syllables. INLINEFORM0 in this case would be the probability that the number of words in the input are two ( INLINEFORM1 ) when the correction is jewelry. A third example is: INLINEFORM2 INLINEFORM3 
Note that in this case the INLINEFORM0 pair is (peak sales, pixel).
Calculating thus the values of INLINEFORM0 for all reference corrections, INLINEFORM1 for all feature values, INLINEFORM2 for all the INLINEFORM3 features in INLINEFORM4 , we are in a position to calculate the RHS of ( EQREF9 ). When this trained classifier is given an erroneous text, features are extracted from this text and the repair works by replacing the erroneous word by a correction that maximizes ( EQREF9 ), INLINEFORM5 
Namely, the INLINEFORM0 for which INLINEFORM1 is maximum.
We present the results of our experiments with both the Evo-Devo and the Machine Learning mechanisms described earlier using the U.S. Census Bureau conducted Annual Retail Trade Survey of U.S. Retail and Food Services Firms for the period of 1992 to 2013 BIBREF12 .
We downloaded this survey data and hand crafted a total of 293 textual questions BIBREF13 which could answer the survey data. A set of 6 people (L2 English) generated 50 queries each with the only constraint that these queries should be able to answer the survey data. In all a set of 300 queries were crafted of which duplicate queries were removed to leave 293 queries in all. Of these, we chose 250 queries randomly and distributed among 5 Indian speakers, who were asked to read aloud the queries into a custom-built audio data collecting application. So, in all we had access to 250 audio queries spoken by 5 different Indian speakers; each speaking 50 queries.
Each of these 250 audio utterances were passed through 4 different ASR engines, namely, Google ASR (Ga), Kaldi with US acoustic models (Ku), Kaldi with Indian Acoustic models (Ki) and PocketSphinx ASR (Ps). In particular, that audio utterances were in wave format (.wav) with a sampling rate of 8 kHz and 16 bit. In case of Google ASR (Ga), each utterance was first converted into .flac format using the utility sound exchange (sox) commonly available on Unix machines. The .flac audio files were sent to the cloud based Google ASR (Ga) one by one in a batch mode and the text string returned by Ga was stored. In all 7 utterances did not get any text output, presumably Ga was unable to recognize the utterance. For all the other 243 utterances a text output was received.
In case of the other ASR engines, namely, Kaldi with US acoustic models (Ku), Kaldi with Indian Acoustic models (Ki) and PocketSphinx ASR (Ps) we first took the queries corresponding to the 250 utterances and built a statistical language model (SLM) and a lexicon using the scripts that are available with PocketSphinx BIBREF14 and Kaldi BIBREF15 . This language model and lexicon was used with the acoustic model that were readily available with Kaldi and Ps. In case of Ku we used the American English acoustic models, while in case of Ki we used the Indian English acoustic model. In case of Ps we used the Voxforge acoustic models BIBREF16 . Each utterance was passed through Kaldi ASR for two different acoustic models to get INLINEFORM0 corresponding to Ku and Ki. Similarly all the 250 audio utterance were passed through the Ps ASR to get the corresponding INLINEFORM1 for Ps. A sample utterance and the output of the four engines is shown in Figure FIGREF12 .
Figure FIGREF11 and Table TABREF14 capture the performance of the different speech recognition engines. The performance of the ASR engines varied, with Ki performing the best with 127 of the 250 utterances being correctly recognized while Ps returned only 44 correctly recognized utterances (see Table TABREF14 , Column 4 named "Correct") of 250 utterances. The accuracy of the ASR varied widely. For instance, in case of Ps there were as many as 97 instances of the 206 erroneously recognized utterances which had an accuracy of less than 70%.
Note that the accuracy is computed as the number of deletions, insertions, substitutions that are required to convert the ASR output to the textual reference (namely, INLINEFORM0 ) and is a common metric used in speech literature BIBREF17 .
For all our analysis, we used only those utterances that had an accuracy 70% but less that INLINEFORM0 , namely, 486 instances (see Table TABREF14 , Figure FIGREF13 ). An example showing the same utterance being recognized by four different ASR engines is shown in Figure FIGREF12 . Note that we used INLINEFORM1 corresponding to Ga, Ki and Ku in our analysis (accuracy INLINEFORM2 ) and not INLINEFORM3 corresponding to Ps which has an accuracy of INLINEFORM4 only. This is based on our observation that any ASR output that is lower that INLINEFORM5 accurate is so erroneous that it is not possible to adapt and steer it towards the expected output.
The ASR output ( INLINEFORM0 ) are then given as input in the Evo-Devo and Machine Learning mechanism of adaptation.
We ran our Evo-Devo mechanism with the 486 ASR sentences (see Table TABREF14 ) and measured the accuracy after each repair. On an average we have achieved about 5 to 10% improvements in the accuracy of the sentences. Fine-tuning the repair and fitness functions, namely Equation (), would probably yield much better performance accuracies. However, experimental results confirm that the proposed Evo-Devo mechanism is an approach that is able to adapt INLINEFORM0 to get closer to INLINEFORM1 . We present a snapshot of the experiments with Google ASR (Ga) and calculate accuracy with respect to the user spoken question as shown in Table TABREF16 .
Table TABREF16 clearly demonstrates the promise of the evo-devo mechanism for adaptation/repair. In our experiments we observed that the adaptation/repair of sub-parts in ASR-output ( INLINEFORM0 ) that most probably referred to domain terms occurred well and were easily repaired, thus contributing to increase in accuracy. For non-domain-specific linguistic terms the method requires one to build very good linguistic repair rules, without which the method could lead to a decrease in accuracy. One may need to fine-tune the repair, match and fitness functions for linguistic terms. However, we find the abstraction of evo-devo mechanism is very apt to use.
In the machine learning technique of adaptation, we considers INLINEFORM0 pairs as the predominant entity and tests the accuracy of classification of errors.
In our experiment, we used a total of 570 misrecognition errors (for example, (dear, beer) and (have, has) derived from INLINEFORM0 or (than twenty, jewelry) derived from INLINEFORM1 ) in the 486 sentences. We performed 10-fold cross validation, each fold containing 513 INLINEFORM2 pairs for training and 57 pairs for testing, Note that we assume the erroneous words in the ASR output being marked by a human oracle, in the training as well as the testing set. Suppose the following example ( INLINEFORM3 ) occurs in the training set: INLINEFORM4 INLINEFORM5 
The classifier is given the pair INLINEFORM0 (latest stills), cumulative sales} to the classifier. And if the following example occurs in the testing set ( INLINEFORM1 ), INLINEFORM2 INLINEFORM3 
the trained model or the classifier is provided INLINEFORM0 (wine) and successful repair would mean it correctly labels (adapts) it to remain the. The features used for classification were ( INLINEFORM1 in Equation ( EQREF8 ))
The combination of features INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 , INLINEFORM4 namely, (bag of consonants, bag of vowels, left context, number of words, right context) gave the best results with INLINEFORM5 % improvement in accuracy in classification over 10-fold validation.
The experimental results for both evo-devo and machine learning based approaches demonstrate that these techniques can be used to correct the erroneous output of ASR. This is what we set out to establish in this paper.
General-purpose ASR engines when used for enterprise domains may output erroneous text, especially when encountering domain-specific terms. One may have to adapt/repair the ASR output in order to do further natural language processing such as question-answering. We have presented two mechanisms for adaptation/repair of ASR-output with respect to a domain. The Evo-Devo mechanism provides a bio-inspired abstraction to help structure the adaptation and repair process. This is one of the main contribution of this paper. The machine learning mechanism provides a means of adaptation and repair by examining the feature-space of the ASR output. The results of the experiments show that both these mechanisms are promising and may need further development.
Nikhil, Chirag, Aditya have contributed in conducting some of the experiments. We acknowledge their contribution.

Figure and table captions:
Fig. 1 Fitness Function.
Table 1 Ontology Structure.
Fig. 2 T ′ accuracy (y-axis) for the 250 utterance (x-axis) for Ga, Ki, Ku and Ps.
Table 2 ASR engines and their output %accuracy
Fig. 4 All utterances that have and T ′ accuracy (y-axis) ≥ 70 and < 100 used in all our experiments.
Table 3 Evo-Devo experiments with Google ASR (Ga).

Questions about this paper:
Which of their proposed domain adaptation methods proves best overall?

Answers:
Machine learning approach