Introduction
Currently, voice-controlled smart devices are widely used in multiple areas to fulfill various tasks, e.g. playing music, acquiring weather information and booking tickets. The SLU system employs several modules to enable the understanding of the semantics of the input speeches. When there is an incoming speech, the ASR module picks it up and attempts to transcribe the speech. An ASR model could generate multiple interpretations for most speeches, which can be ranked by their associated confidence scores. Among the $n$-best hypotheses, the top-1 hypothesis is usually transformed to the NLU module for downstream tasks such as domain classification, intent classification and named entity recognition (slot tagging). Multi-domain NLU modules are usually designed hierarchically BIBREF0. For one incoming utterance, NLU modules will firstly classify the utterance as one of many possible domains and the further analysis on intent classification and slot tagging will be domain-specific.
In spite of impressive development on the current SLU pipeline, the interpretation of speech could still contain errors. Sometimes the top-1 recognition hypothesis of ASR module is ungrammatical or implausible and far from the ground-truth transcription BIBREF1, BIBREF2. Among those cases, we find one interpretation exact matching with or more similar to transcription can be included in the remaining hypotheses ($2^{nd}- n^{th}$).
To illustrate the value of the $2^{nd}- n^{th}$ hypotheses, we count the frequency of exact matching and more similar (smaller edit distance compared to the 1st hypothesis) to transcription for different positions of the $n$-best hypotheses list. Table TABREF1 exhibits the results. For the explored dataset, we only collect the top 5 interpretations for each utterance ($n = 5$). Notably, when the correct recognition exists among the 5 best hypotheses, 50% of the time (sum of the first row's percentages) it occurs among the $2^{nd}-5^{th}$ positions. Moreover, as shown by the second row in Table TABREF1, compared to the top recognition hypothesis, the other hypotheses can sometimes be more similar to the transcription.
Over the past few years, we have observed the success of reranking the $n$-best hypotheses BIBREF1, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10 before feeding the best interpretation to the NLU module. These approaches propose the reranking framework by involving morphological, lexical or syntactic features BIBREF8, BIBREF9, BIBREF10, speech recognition features like confidence score BIBREF1, BIBREF4, and other features like number of tokens, rank position BIBREF1. They are effective to select the best from the hypotheses list and reduce the word error rate (WER) BIBREF11 of speech recognition.
Those reranking models could benefit the first two cases in Table TABREF2 when there is an utterance matching with transcription. However, in other cases like the third row, it is hard to integrate the fragmented information in multiple hypotheses.
This paper proposes various methods integrating $n$-best hypotheses to tackle the problem. To the best of our knowledge, this is the first study that attempts to collectively exploit the $n$-best speech interpretations in the SLU system. This paper serves as the basis of our $n$-best-hypotheses-based SLU system, focusing on the methods of integration for the hypotheses. Since further improvements of the integration framework require considerable setup and descriptions, where jointly optimized tasks (e.g. transcription reconstruction) trained with multiple ways (multitask BIBREF12, multistage learning BIBREF13) and more features (confidence score, rank position, etc.) are involved, we leave those to a subsequent article.
This paper is organized as follows. Section SECREF2 introduces the Baseline, Oracle and Direct models. Section SECREF3 describes proposed ways to integrate $n$-best hypotheses during training. The experimental setup and results are described in Section SECREF4. Section SECREF5 contains conclusions and future work.
Baseline, Oracle and Direct Models ::: Baseline and Oracle
The preliminary architecture is shown in Fig. FIGREF4. For a given transcribed utterance, it is firstly encoded with Byte Pair Encoding (BPE) BIBREF14, a compression algorithm splitting words to fundamental subword units (pairs of bytes or BPs) and reducing the embedded vocabulary size. Then we use a BiLSTM BIBREF15 encoder and the output state of the BiLSTM is regarded as a vector representation for this utterance. Finally, a fully connected Feed-forward Neural Network (FNN) followed by a softmax layer, labeled as a multilayer perceptron (MLP) module, is used to perform the domain/intent classification task based on the vector.
For convenience, we simplify the whole process in Fig.FIGREF4 as a mapping $BM$ (Baseline Mapping) from the input utterance $S$ to an estimated tag's probability $p(\tilde{t})$, where $p(\tilde{t}) \leftarrow BM(S)$. The $Baseline$ is trained on transcription and evaluated on ASR 1st best hypothesis ($S=\text{ASR}\ 1^{st}\ \text{best})$. The $Oracle$ is trained on transcription and evaluated on transcription ($S = \text{Transcription}$). We name it Oracle simply because we assume that hypotheses are noisy versions of transcription.
Baseline, Oracle and Direct Models ::: Direct Models
Besides the Baseline and Oracle, where only ASR 1-best hypothesis is considered, we also perform experiments to utilize ASR $n$-best hypotheses during evaluation. The models evaluating with $n$-bests and a BM (pre-trained on transcription) are called Direct Models (in Fig. FIGREF7):
Majority Vote. We apply the BM model on each hypothesis independently and combine the predictions by picking the majority predicted label, i.e. Music.
Sort by Score. After parallel evaluation on all hypotheses, sort the prediction by the corresponding confidence score and choose the one with the highest score, i.e. Video.
Rerank (Oracle). Since the current rerank models (e.g., BIBREF1, BIBREF3, BIBREF4) attempt to select the hypothesis most similar to transcription, we propose the Rerank (Oracle), which picks the hypothesis with the smallest edit distance to transcription (assume it is the $a$-th best) during evaluation and uses its corresponding prediction.
Integration of N-BEST Hypotheses
All the above mentioned models apply the BM trained on one interpretation (transcription). Their abilities to take advantage of multiple interpretations are actually not trained. As a further step, we propose multiple ways to integrate the $n$-best hypotheses during training. The explored methods can be divided into two groups as shown in Fig. FIGREF11. Let $H_1, H_2,..., H_n $ denote all the hypotheses from ASR and $bp_{H_k, i} \in BPs$ denotes the $i$-th pair of bytes (BP) in the $k^{th}$ best hypothesis. The model parameters associated with the two possible ways both contain: embedding $e_{bp}$ for pairs of bytes, BiLSTM parameters $\theta $ and MLP parameters $W, b$.
Integration of N-BEST Hypotheses ::: Hypothesized Text Concatenation
The basic integration method (Combined Sentence) concatenates the $n$-best hypothesized text. We separate hypotheses with a special delimiter ($<$SEP$>$). We assume BPE totally produces $m$ BPs (delimiters are not split during encoding). Suppose the $n^{th}$ hypothesis has $j$ pairs. The entire model can be formulated as:
In Eqn. DISPLAY_FORM13, the connected hypotheses and separators are encoded via BiLSTM to a sequence of hidden state vectors. Each hidden state vector, e.g. $h_1$, is the concatenation of forward $h_{1f}$ and backward $h_{1b}$ states. The concatenation of the last state of the forward and backward LSTM forms the output vector of BiLSTM (concatenation denoted as $[,]$). Then, in Eqn. DISPLAY_FORM14, the MLP module defines the probability of a specific tag (domain or intent) $\tilde{t}$ as the normalized activation ($\sigma $) output after linear transformation of the output vector.
Integration of N-BEST Hypotheses ::: Hypothesis Embedding Concatenation
The concatenation of hypothesized text leverages the $n$-best list by transferring information among hypotheses in an embedding framework, BiLSTM. However, since all the layers have access to both the preceding and subsequent information, the embedding among $n$-bests will influence each other, which confuses the embedding and makes the whole framework sensitive to the noise in hypotheses.
As the second group of integration approaches, we develop models, PoolingAvg/Max, on the concatenation of hypothesis embedding, which isolate the embedding process among hypotheses and summarize the features by a pooling layer. For each hypothesis (e.g., $i^{th}$ best in Eqn. DISPLAY_FORM16 with $j$ pairs of bytes), we could get a sequence of hidden states from BiLSTM and obtain its final output state by concatenating the first and last hidden state ($h_{output_i}$ in Eqn. DISPLAY_FORM17). Then, we stack all the output states vertically as shown in Eqn. SECREF15. Note that in the real data, we will not always have a fixed size of hypotheses list. For a list with $r$ ($<n$) interpretations, we get the embedding for each of them and pad with the embedding of the first best hypothesis until a fixed size $n$. When $r\ge n$, we only stack the top $n$ embeddings. We employ $h_{output_1}$ for padding to enhance the influence of the top 1 hypothesis, which is more reliable. Finally, one unified representation could be achieved via Pooling (Max/Avg pooling with $n$ by 1 sliding window and stride 1) on the concatenation and one score could be produced per possible tag for the given task.
Experiment ::: Dataset
We conduct our experiments on $\sim $ 8.7M annotated anonymised user utterances. They are annotated and derived from requests across 23 domains.
Experiment ::: Performance on Entire Test Set
Table TABREF24 shows the relative error reduction (RErr) of Baseline, Oracle and our proposed models on the entire test set ($\sim $ 300K utterances) for multi-class domain classification. We can see among all the direct methods, predicting based on the hypothesis most similar to the transcription (Rerank (Oracle)) is the best.
As for the other models attempting to integrate the $n$-bests during training, PoolingAvg gets the highest relative improvement, 14.29%. It as well turns out that all the integration methods outperform direct models drastically. This shows that having access to $n$-best hypotheses during training is crucial for the quality of the predicted semantics.
Experiment ::: Performance Comparison among Various Subsets
To further detect the reason for improvements, we split the test set into two parts based on whether ASR first best agrees with transcription and evaluate separately. Comparing Table TABREF26 and Table TABREF27, obviously the benefits of using multiple hypotheses are mainly gained when ASR 1st best disagrees with the transcription. When ASR 1st best agrees with transcription, the proposed integration models can also keep the performance. Under that condition, we can still improve a little (3.56%) because, by introducing multiple ASR hypotheses, we could have more information and when the transcription/ASR 1st best does not appear in the training set's transcriptions, its $n$-bests list may have similar hypotheses included in the training set's $n$-bests. Then, our integration model trained on $n$-best hypotheses as well has clue to predict. The series of comparisons reveal that our approaches integrating the hypotheses are robust to the ASR errors and whenever the ASR model makes mistakes, we can outperform more significantly.
Experiment ::: Improvements on Different Domains and Different Numbers of Hypotheses
Among all the 23 domains, we choose 8 popular domains for further comparisons between the Baseline and the best model of Table TABREF24, PoolingAvg. Fig. FIGREF29 exhibits the results. We could find the PoolingAvg consistently improves the accuracy for all 8 domains.
In the previous experiments, the number of utilized hypotheses for each utterance during evaluation is five, which means we use the top 5 interpretations when the size of ASR recognition list is not smaller than 5 and use all the interpretations otherwise. Changing the number of hypotheses while evaluation, Fig. FIGREF30 shows a monotonic increase with the access to more hypotheses for the PoolingAvg and PoolingMax (Sort by Score is shown because it is the best achievable direct model while the Rerank (Oracle) is not realistic). The growth becomes gentle after four hypotheses are leveraged.
Experiment ::: Intent Classification
Since another downstream task, intent classification, is similar to domain classification, we just show the best model in domain classification, PoolingAvg, on domain-specific intent classification for three popular domains due to space limit. As Table TABREF32 shows, the margins of using multiple hypotheses with PoolingAvg are significant as well.
Conclusions and Future Work
This paper improves the SLU system robustness to ASR errors by integrating $n$-best hypotheses in different ways, e.g. the aggregation of predictions from hypotheses or the concatenation of hypothesis text or embedding. We can achieve significant classification accuracy improvements over production-quality baselines on domain and intent classifications, 14% to 25% relative gains. The improvement is more significant for a subset of testing data where ASR first best is different from transcription. We also observe that with more hypotheses utilized, the performance can be further improved. In the future, we aim to employ additional features (e.g. confidence scores for hypotheses or tokens) to integrate $n$-bests more efficiently, where we can train a function $f$ to obtain a weight for each hypothesis embedding before pooling. Another direction is using deep learning framework to embed the word lattice BIBREF16 or confusion network BIBREF17, BIBREF18, which can provide a compact representation of multiple hypotheses and more information like times, in the SLU system.
Acknowledgements
We would like to thank Junghoo (John) Cho for proofreading.

Question: Which ASR system(s) is used in this work?
Answer: Oracle

Question: What are the series of simple models?
Answer: perform experiments to utilize ASR $n$-best hypotheses during evaluation

Question: Over which datasets/corpora is this work evaluated?
Answer: $\sim $ 8.7M annotated anonymised user utterances

on $\sim $ 8.7M annotated anonymised user utterances