Towards NLU Model Robustness to ASR Errors at Scale
Yaroslav Nechaev

Weitong Ruan

Imre Kiss

work@remper.ru
Amazon Alexa
Cambridge, MA, USA

weiton@amazon.com
Amazon Alexa
Cambridge, MA, USAikiss@amazon.com
Amazon Alexa
Cambridge, MA, USA

ABSTRACTIn a large-scale Spoken Language Understanding system, Natural
Language Understanding (NLU) models are typically decoupled, i.e,
trained and updated independently, from the upstream Automatic
Speech Recognition (ASR) system that provides textual hypotheses
for the user’s voice signal as input to NLU. Such ASR hypotheses
often contain errors causing severe performance degradation as thedownstream NLU models are trained on clean human-annotatedtranscripts. Furthermore, as the ASR model updates, the error distribution drifts making it even harder for NLU models to recover andmaking manual annotation of erroneous ASR hypotheses impractical.
In this paper, we investigate data-efficient techniques applicable
to a wide variety of NLU models employed in large-scale production
environments to make them robust to ASR errors. We measure theeffectiveness of such techniques as both the ASR error distribution
and usage patterns change over time.

CCS CONCEPTS• Computing methodologies → Speech recognition; Supervised
learning; Neural networks.

KEYWORDS
datasets, natural language understanding, data augmentation, adversarial training, data-efficient learning
ACM Reference Format:
Yaroslav Nechaev, Weitong Ruan, and Imre Kiss. 2021. Towards NLU Model
Robustness to ASR Errors at Scale. In 2nd International Workshop on DataEfficient Machine Learning @KDD2021, 14–18 Aug, 2021, Virtual Conference. ACM, New York, NY, USA, 6 pages. https://doi.org/10.0000/0000000.
0000000

1

INTRODUCTION

Spoken Language Understanding (SLU) systems quickly became astaple part of people’s day to day activities. Cars, personal computers, phones and other consumer devices are equipped with personal
assistants responding to voice commands. Behind each of these products there is an SLU system, most widely used of which, to name afew, are Siri, Google Assistant, Alexa and Cortana. Implementing
SLU with comparable functionality requires significant effort from
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
DeMaL2021@KDD2021, 14–18 Aug, 2021, Virtual Conference© 2021 Association for Computing Machinery.
ACM ISBN 978-1-4503-XXXX-X/18/06. . . $15.00
https://doi.org/10.0000/0000000.0000000

many engineers and scientists. To enable efficient participation of somany people in the development process and support fast model iteration, such SLU systems are typically decoupled into a multitude ofinteracting components, each solving its separate task. Thus, recent
push in the academic community [8, 12, 17] towards powerful endto-end SLU approaches may not be practical to adopt for complex
SLU systems.
Two major parts of a typical SLU system are the Automatic
Speech Recognition (ASR) and the Natural Language Understanding (NLU) components. ASR is tasked with performing necessary
processing to convert user’s speech into text. Natural Language
Understanding (NLU) component [14] works on the output of the
ASR system predicting a domain (Domain Classification task orDC), intent (Intent Classification – IC) and performing slot filling(Named Entity Recognition or NER). For example, for utterance"what is the temperature in Barcelona" a typical pipeline could
output domain:weather, intent:get_weather and the following NER annotation:
what is the temperature in Barcelona.
O
requestO location
Due to this separation, ASR errors propagate to downstream NLU
components at times causing severe performance degradation [14].
If NLU component is trained and evaluated solely on clean humanannotated transcripts then it will have little to no chance to recoverfrom such errors. Additionally, NLU models trained to capture semantic similarity between words can have a hard time dealing withphonetic ambiguity, which is the main cause of ASR errors. Thus,
measures have to be implemented to make NLU systems robust to
upstream errors. Recently this problem has attracted considerable
interest from researchers proposing a wide variety of approaches tomitigate this drop in performance [6, 14, 21].
Additional challenges arise when trying to solve the issue of ASR
errors in a large-scale SLU system. Firstly, the ASR component isperiodically updated with the new releases causing a drift in both theerror distribution and the distribution of any auxiliary signals, suchas confidence scores. Thus, a system trained to be robust to errors
produced by a particular release of the ASR system can fail to adjust
to the newer versions. Secondly, the NLU itself can be decoupledwith multiple models of varying complexity being deployed at thesame time, requiring a potential robustness solution to be applicable
to as many model families as possible with little or no modification.
In third, the usage patterns change in response to new trends and newfeatures which, compounded with periodic ASR releases, further
exacerbates the input distribution drift. Finally, real-world SLU
systems can have strict constraints on both the inference and thetraining times to maintain reasonable operational costs.
In this paper, having these challenges in mind, we investigate three
simple techniques to improve downstream model robustness to ASR

DeMaL2021@KDD2021, 14–18 Aug, 2021, Virtual Conference

errors: Data Augmentation, Adversarial Training and a ConfidenceAware Layer. We design these approaches to be as data-efficient,
lightweight and model-agnostic as possible to be applicable to awide selection of NLU models while introducing as little overhead
to production latency and operational costs as possible. We showcase
this flexibility by applying them to a number of domain classification(DC) models of varying complexity.
Then, we measure the performance of the proposed approaches
in presence of ASR errors. To this end, we employ different snapshots of our internal test and training sets covering roughly a one
year period. This allows us to measure the effectiveness of thesetechniques as the input distribution starts to drift due to new ASR
model releases and changes in usage patterns. We employ the following evaluation protocol which we suggest to be adopted bothby industry and academia in future to ensure model robustness to
ASR errors in presence of various distribution drifts. We measure theproposed and the baseline NLU model performance using both thehuman-annotated transcripts and the ASR hypotheses as input. A successful model is expected to show both (i) significant improvement
as measured on ASR hypotheses and (ii) absence of degradation as
evaluated on transcripts. The former gives us a measure of model
robustness to ASR errors produced by a specific ASR system, while
the latter serves as a proxy measure to the ability of the algorithm togeneralize and potentially adapt to distribution drifts.
The rest of the paper is organized as follows. In Section 2 we summarize related work, while Section 3 describes proposed robustness
approaches. Section 4 defines models, datasets and metrics used
to measure robustness performance. Section 5 provides in-depth
evaluation of the proposed approaches. Section 6 concludes thepaper.

2

RELATED WORK

Various previous works tackled the problem of model robustness
to ASR Errors. A number of data augmentation techniques were
proposed exploiting ASR hypotheses [3, 11]. Further improvements
proposed to leverage word confusion information stored in the Word
Confusion Network (WCN) [5, 7, 18, 19]. In [7], the authors proposed a LatticeRNN approach which builds an embedding from thelattice generated by the ASR. However, these approaches are dataintensive and time-consuming since, as the ASR model improves
over time, previously annotated ASR one-best hypotheses become
stale and cannot be used for training. To avoid this problem, Simonnet et al. [15] proposed an approach to simulate ASR errors fromtranscriptions and showed that the noising process helps to improve
the robustness of the SLU system to ASR errors especially in caseof insufficient training data.
A number of studies aimed at improving the representation learning instead. Following the success of learning continuous word
representation, e.g. word2vec, in [13, 14] the authors came up witha confusion2vec [2], where the output embeddings contain not only
semantic and syntactic relations of words in human language but
also acoustic relationship between words. Similarly, Chung andGlass [2] designed a speech2vec architecture based on a RNN
Encoder-Decoder framework and a skipgram or continuous bagof-words training methodology. The resulting embedding has thecapacity of capturing speech signal not in plain text. Huang andYaroslav Nechaev, Weitong Ruan, and Imre Kiss

Chen [6] proposed to add a confusion loss to the task loss on top of
the Embeddings from Language Model (ELMo) model during thefine-tuning stage. This confusion loss penalizes a negative cosine
distance between the Language Model (LM) representations of eachconfusion word pairs, forcing the language model to generate similar contextualized embedding representation for the confusion wordpairs. In Zhu et al. [21], the problem was approached from the perspective of training and evaluation mismatch. The proposed approach
uses one layer of shard BiLSTM encoder and three task-oriented
BiLSTM decoders. During model training, it forms a multi-task
learning problem where the first task is a slot-tagging task over annotated transcription; the second is an unsupervised reconstruction
task for both transcript and ASR hypotheses; the third is anothertask-invariant task where for each intermittent hidden output fromthe shared encoder layer, another FNN is used to classify them as
either label or ASR hypotheses or transcription. The experimental
results suggest that with the extra loss, the learned representation
makes the model robust to ASR errors.
There are also approaches that aims to solve this problem fromthe ASR perspective. Soni et al. [16] proposed to model additivenoise and channel mismatch distortion using a parametric generativemodel. They demonstrated that their proposed approach reduces theword error rate for ASR in unseen conditions.
Finally, our Data Augmentation and Adversarial Training implementation is closely related to the one proposed in Ruan et al.
[11]. This paper is a followup investigation focusing on deployment
considerations of these approaches in a large-scale SLU system.

3

MODEL ROBUSTNESS TO ASR ERRORS

A large NLU system usually consists of three statistical models:a domain classifier (DC), intent classifier (IC) and a named entity
recognition (NER) components. As mentioned above, in this paper
we specifically focus on DC robustness, however, approaches described here can be applied with little or no modification to IC or
any other classification task. Some modifications will be required
for NER component [11].
Here we detail three techniques that we used to improve DC robustness to ASR errors: Data Augmentation, Adversarial Training
and Confidence-Aware Layer. These approaches were designed withthe following production considerations in mind: (i) be data-efficient:no additional requirements for data annotation should be imposed
and the approach should not significantly degrade training and inference time, (ii) be applicable to a wide range of models in a plug andplay fashion.
Our Data Augmentation (DA) is a relatively straightforward approach: for utterances in the training set we add the corresponding
1-best hypotheses from the ASR output with the same domain label.
DA is completely model-agnostic and can be applied during thedata pre-processing step, which makes implementing it completelytransparent to model builders in a large organization. The main design consideration for DA is the selection of the source of the ASR
hypotheses. As the ASR component evolves, the error distribution
shifts, potentially making NLU model trained on the old distribution
less effective. Updating the entire training set every time the ASR
is changed can be impractical as it is both costly to perform andtakes time. To avoid this issue, we show the effect of different ASR

Towards NLU Model Robustness to ASR Errors at Scale

DeMaL2021@KDD2021, 14–18 Aug, 2021, Virtual Conference

ho1u tpu t

ReLU
ReLU

…

hin pu t

confidence score:

0.56

ho2u tpu t

ReLU

0.0

ho3u tpu t

ReLU

0.2

ReLU

0.4

pooling

ho u tpu t…

ho4u tpu t

0.6

ho5u tpu t

0.8

Figure 1: Overview of the Confidence-Aware Layer architecture.
update strategies on model performance. Our models were trainedwith an early stopping criteria on a validation set. Curiously, boththe baseline and the augmented system took comparable number of
updates to converge, making DA training time similar to baseline.
Note that hypotheses beyond the 1-best can also be used to augment
the training set, however, the more noisy the hypotheses, the more
likely it is to change the semantics of the input utterance and cause
generalization problems during training.
To allow for the greater control over the model behavior a generalization of DA called Adversarial Training (AT) was proposed [11].
Inspired by recent Adversarial Training approaches the idea is toconsider ASR errors as a source of input perturbations. Formally,
given the loss function L (𝑦𝑡𝑟𝑢𝑒 , 𝑦𝑝𝑟𝑒𝑑 ) for the target model, where
𝑦𝑡𝑟𝑢𝑒 is a true label, 𝑦𝑝𝑟𝑒𝑑 is a predicted label given the input 𝑥𝑝𝑟𝑒𝑑
the resulting objective function will be defined as follows:
𝐽 = 𝛽 L (𝑦𝑡𝑟𝑢𝑒 , 𝑦𝑡𝑟𝑎𝑛𝑠 ) + 𝛾 L (𝑦𝑡𝑟𝑢𝑒 , 𝑦𝑎𝑠𝑟 ) + 𝛼𝐷𝐾𝐿 (𝑦𝑡𝑟𝑎𝑛𝑠 ||𝑦𝑎𝑠𝑟 )where 𝛽 = 1−𝛼1+𝛾 and 𝛼 ∈ [0, 1). 𝑦𝑡𝑟𝑎𝑛𝑠 and 𝑦𝑎𝑠𝑟 are target model
predictions given respectively the transcription 𝑥𝑡𝑟𝑎𝑛𝑠 and the corresponding ASR hypothesis 𝑥𝑎𝑠𝑟 as input; 𝐷𝐾𝐿 is a KL-divergence.
KL-divergence term ensures consistency of predictions for the sameutterance regardless of whether it is based on a clean transcription
or on ASR output containing an error. Note that when 𝛼 = 0 thisapproach turns into DA. Parameter 𝛾 is either set to 1 or equal to the
ASR confidence score for the corresponding input sample.
In the above formulation semantics of the input utterance canchange due to an ASR error: user saying "bye iphone" can be misheard as "buy a phone", yielding completely different domain in DC.
Thus, having the same label when computing L𝑎𝑠𝑟 and 𝐷𝐾𝐿 can potentially cause performance degradation. This is akin to label noiseproblem and, to achieve robustness to such noise, we have tested avariation of AT replacing the categorical cross entropy (CCE) loss
for the ASR hypothesis with a Q-loss [20], a generalization of CCE
and the mean-absolute error (MAE) losses, that allows controllingnoise-robustness properties of a loss. Formally:𝑞L𝑞 (𝑦𝑡𝑟𝑢𝑒 , 𝑦𝑎𝑠𝑟 ) = 𝑦𝑡𝑟𝑢𝑒(1 − 𝑦𝑎𝑠𝑟 )𝑞(1)parameter 𝑞 ∈ [0, 1], which controls the balance between noiseresistant MAE (achieved when 𝑞 = 1) behaviour and CCE (approximated as 𝑞 → 0), can be set depending on the ASR confidence score′(𝑐𝑎𝑠𝑟 ) like so: 𝑞 = (1 − 𝑐𝑎𝑠𝑟 )𝑞 , we refer to this variation as Adaptive
Q-loss.

Just like DA, Adversarial Training does not degrade inference
performance and is applicable to a wide range of statistical models in a plug-and-play fashion due to being agnostic to the model
structure. However, some modification of the model and the training
workflow were required to adopt AT making it slightly harder to productize compared to DA. Similarly to DA, AT does not significantly
affect training time, however, we did observe some performancedegradation due to required realignment of transcriptions and ASRs
during the computation of the loss. Note that in all cases the ASR
confidence score is only used during the training. As the ASR component updates, confidence score distributions can change as wellpotentially making the ASR confidence score an unreliable feature.
Finally, Confidence-Aware Layer (CA) is simple modification
aimed at exploiting utterance-level ASR confidence score (𝑐𝑎𝑠𝑟 ) during training and optionally during inference. Figure 1 provides an
overview of the proposed approach. CA is applied to an existinglayer in a neural network (in picture – a regular fully-connected
layer with ReLU activation function), duplicating target layer 𝑛times forming "buckets", each corresponding to the specific 𝑐𝑎𝑠𝑟
threshold. Then, during the forward pass only the buckets that correspond to thresholds that are lower than the input 𝑐𝑎𝑠𝑟 are activated.
The resulting outputs are pooled using either maximum, average
or attention. Note that with this formulation each bucket receives
all available transcripts (which have 𝑐𝑎𝑠𝑟 = 1.0) and progressively
more noisy ASR hypotheses as threshold is reduced. Pooling suchbuckets create an ensemble effect similar to pooling multiple dataaugmented models with different amounts of ASR hypotheses added
into the training set. Threshold values and the number of buckets
can be varied: we used 5 and 10 buckets, selecting either uniformly
distributed threshold values or split to achieve roughly the same
number of ASR hypotheses for each resulting score interval. Suchtechnique can be applied to any layer or groups of layers in the
network, however, when applied to a fully-connected layer, a very
efficient implementation is possible allowing computing all bucket’stransformations in a single matrix operation, making CA equivalent
to a single wider layer. CA can be used without a confidence score
during inference by activating all buckets together, avoiding adding
the runtime dependency on 𝑐𝑎𝑠𝑟 .

4

EXPERIMENTAL SETTING

In this section, we describe the datasets and metrics we used in our
experiments as well as define domain classification models that wereused as targets for our robustness approaches. All experiments were

DeMaL2021@KDD2021, 14–18 Aug, 2021, Virtual Conference

Yaroslav Nechaev, Weitong Ruan, and Imre Kiss

done on our internal data and designed to be as close to production pipeline as possible to reflect real challenges that arise fromupstream ASR errors in a large-scale NLU. Following evaluation
criteria defined above, we evaluate both on the human-annotatedtranscriptions and on the ASR 1-best hypotheses aiming to balance
model generalization with robustness to a specific ASR input distribution.

Table 1: Relative change in SEMER when DA is applied to eachof the models. Performance on transcripts and ASR 1-bests
is reported using ASR-5M and ASR-9M datasets respectively.
Lower the better.

Model

ASR-5M (SEMER)
Transcripts ASR

ASR-9M (SEMER)
Transcripts ASR

4.1

BOW
FFNN
Ensemble
LSTM
LSTMCNN+14.05%+2.52%+6.34%+0.09%-0.58%+0.73%-0.75%-0.29%–-0.29%

ASR hypotheses

In our experiments we use ASR hypotheses and the associatedutterance-level confidence scores to augment the target models. To
showcase model performance, we use two datasets including bothtranscriptions and ASR 1-bests: ASR-5M and ASR-9M. The former
includes utterances covering four years and includes 5.5M utterances. The latter represents utterances covering a one year period
after ASR-5M, has 9M samples and has significantly lower word
error rate in its ASR hypotheses.
To perform distribution shift analysis in the later stages of ourexperiments we have produced two additional datasets: ASR-19a
and ASR-19b, representing utterances coming from two fixed production ASR engine releases roughly half a year apart. ASR-19a+b
is a combination of the two.
In production NLU systems lots of synthetic textual data is often
used to assist in development of new features and for such utterances
we would not have the corresponding ASR hypotheses. In our case,the datasets described above cover roughly a third of the training set
depending on a domain and training setup. To mitigate this issue, we
employ a recent text-level technique for simulating ASR errors [4]to produce ASR hypotheses for synthetic utterances mimicking realerror distribution and word error rate. This procedure completes
the augmentation of the training set yielding the ASR-ALL dataset
containing 29M samples.

4.2

Metrics

To measure the overall NLU performance we use Semantic Error
Rate (SEMER) defined as follows:
SEMER =

𝑁 errors
𝑁𝑆 + 𝑁 𝐼 + 𝑁 𝐷=,
𝑁 reference slots
𝑁𝑆 + 𝑁𝐷 + 𝑁𝐶

(2)

where 𝑁𝑆 , 𝑁𝐼 , 𝑁𝐷 and 𝑁𝐶 are the number of substitution, insertion,
deletion errors and the correct slots respectively. This metric is similar to Slot Error Rate (SER) employed in information extraction [10].
When assessing the DC component performance separately, we use
per-domain error rate (fDCER) defined as 1 − 𝐹 1. To aggregate theperformance across all domains, we calculate DC error rate definedas the corresponding weighted average: 1 − 𝐹 1𝑚𝑎𝑐𝑟𝑜 .

4.3

ModelsWe employ five different models of increasing complexity to evaluate
robustness approaches. In our experiments, DC models are binaryclassifiers for each individual domain. Domain prediction is thenacquired in a one-vs-all fashion.
BOW uses a simple logistic regression on top of sparse vector
comprised of unigram, bigram and trigram features and serves as
the simplest model in our tests. FFNN is a shallow neural modelusing fastText [1] embeddings as input followed by two layersof size 256 and ReLU activation functions. An Ensemble of these-3.94%-4.98%-5.19%-6.37%-7.53%-3.00%-4.14%-3.75%–-3.44%

Table 2: Performances (fDCER and SEMER) of the proposed
approaches on transcripts and ASR 1-bests compared to Data
Augmentation (DA) using ASR-ALL dataset. AT parameters:
𝛼 = 0.3 and 𝛾 = 1. CA uses uniform split on confidence score.
Lower the better.

ApproachfDCER
Transcripts ASR

SEMERTranscripts ASR

DA
AT
CA
DA+AT+CA–-0.56%-3.91%-4.75%–-0.50%-1.41%-1.81%–-0.78%-1.95%-3.70%–-0.35%-0.76%-0.81%

two classifiers is used combining the BOW and embedding-based
features in a single network.
The forth system, LSTM, has two bi-directional LSTM layers withhidden state of size 256. Finally, we employ LSTMCNN, inspired byrecent sequence tagging approach [9], as the most complex modelin our tests. LSTMCNN exploit both character-level and word-level
information via the combination of convolutional and recurrent layers. First, character embeddings (of size 16) are aggregated into
word-level representations using a convolutional layer (with hidden
state of size 32). Then, they are concatenated with an additional
word vector and fed to two Variational LSTM layers (with hidden
state of size 768).

5

EVALUATION

To assess the impact of the proposed robustness approaches weevaluate our models both on clean human-annotated transcripts andon the ASR 1-best hypotheses. Ideally, we would want to show an
improvement on ASR hypotheses while maintaining performanceon transcripts as a proxy measure for generalization allowing ourmodels to show reasonable performance without the mandatory
retraining as the ASR model updates.
Firstly, we evaluate Data Augmentation (DA) on models introduced above on datasets covering two time periods. Then, we
compare Data Augmentation against Adversarial Training and theConfidence-Aware Layer approaches. Finally, we dive deep into
practical considerations of applying such robustness techniques by(i) investigating the performance on different NLU domains, (ii)

Towards NLU Model Robustness to ASR Errors at Scale

DeMaL2021@KDD2021, 14–18 Aug, 2021, Virtual Conference

Table 3: Data Augmentation performance (fDCER) against the baseline (non-augmented model) in the presence of the error distribution drift for different domains. Same ASR release as in ASR-19b is used for ASR evaluation. Relative improvements against thebaseline are reported. Lower the better.

ApproachAcross 3 domains
Trans
ASR

Trans

ASR

Knowledge
Trans
ASR

Shopping
Trans
ASR

Baseline
DA (ASR-19a)
DA (ASR-19b)
DA (ASR-19b, 𝑐𝑎𝑠𝑟 > 0.2)
DA (ASR-19a+b)–-0.04%-0.57%-1.33%+1.00%–-0.24%-0.39%-0.60%+1.99%–-1.51%-2.39%-2.60%-1.60%–-1.56%-0.73%-2.97%+3.57%–-2.33%-1.59%-2.29%-0.18%–-1.96%-2.78%-2.81%-1.81%

Globalstudying the effect of ASR error distribution drift on different ASR
hypothesis update strategies.

5.1

Data Augmentation

To evaluate Data Augmentation approach, we setup a set of controlled experiments assessing its impact on the NLU performance using different DC models. IC and NER models are left fixed throughout our experiments, always making their predictions based on cleantranscripts for each utterance: this way in this experiment we effectively consider IC and NER to be perfectly robust to ASR errors
to further highlight the DC performance. To showcase the effectof overtime improvements in ASR and the training set design weinclude evaluation results for two disjoint time periods: ASR-5M
and ASR-9M (Table 1).
In ASR-5M setting, as the models are getting more sophisticated,
SEMER improves in absolute terms but the gap stays the same.
When Data Augmentation is employed, all models are exhibiting
significant improvements ranging from 4% to 7.5% SEMER on ASR
1-best. However, performance on transcripts is a different story: lesssophisticated models (BOW, FFNN and Ensemble) there show significant performance degradation hinting at such models’ limited
ability to generalize. LSTMCNN, being the most complex model in
our testing, was able to benefit the most from the Data Augmentation
yielding 7.53% improvement on ASR 1-bests and 0.58% improvement on transcripts. Evaluation for the ASR-9M setting paints asimilar picture: Data Augmentation was able to outperform baselines across the board both on transcripts and ASR 1-bests. Smaller
improvements can be attributed to lower word error rate for ASRs inthis dataset.
For the remainder of the paper LSTMCNN is adopted as a baseline
for robustness modifications.

5.2

Advanced approaches

To evaluate the techniques described in Section 3, we employ the
ASR-ALL dataset to augment the training set and use the same test
sets as for ASR-9M for testing. Here we report two sets of results:
the overall SEMER impact across the NLU stack and the fDCER
numbers to isolate impact on just the DC component. Differently
from the previous experiment, IC and NER components will also
receive ASR hypotheses as input. This means that even if DC prediction is correct, the overall performance can still suffer in case thedownstream task (IC or NER) is not able to recover from the ASR–-2.28%-3.00%-3.37%-1.51%–-2.39%-3.70%-3.25%-4.92%

error giving us a complete picture on the effect of our modifications
on the overall NLU stack.
As can be seen in Table 2, each of the approaches provide additional performance benefits on top of the Data Augmentation withthe combination of all techniques exhibiting the best performance.
Less pronounced SEMER improvements is due to IC and NER
dampening the overall NLU performance. Q-loss was expected toperform well, however, in our experiments we did not observe statistically significant improvements (𝑞 = {0.1, 0.01} both in constantand adaptive settings). For CA, choice of thresholds significantly
affected per-domain performance. In terms of pooling strategy, average worked the best, while attention showed overall performancedegradation.

5.3

Distribution drift analysis

Finally, in Table 3 we study how Data Augmentation performs
for different domains in presence of error distribution drift due
to the ASR component updates. Specifically, we used the test set
with ASR hypotheses coming from the same ASR release as the
ASR-19b dataset, while the training set was augmented with either"old" (ASR-19a) or "new" (ASR-19b) data or a combination of
the two. In almost all cases, ASR augmentations allowed for improved performance as evaluated on ASR. Only the old hypotheses(ASR-19a) have caused overall degradation on ASR even though
performance improvements on the three selected domains were observed. Due to this, we recommend updating ASR hypotheses periodically as practical. On transcripts, the combination of two ASR
releases (ASR-19a+b) have shown degradation, warning about potential issues when combining different error distributions. Filteringout noisy ASR hypotheses (𝑐𝑎𝑠𝑟 > 0.2) yielded much greater stability on transcripts without loss of ASR performance. This behaviour
persisted when we evaluated across all domains beyond the threereported — -1.11% improvement on transcriptions and -2.81% on
ASR hypotheses.

6

CONCLUSIONSIn this paper we tackle a problem of NLU model robustness to
upstream ASR errors in a large-scale decoupled SLU system. To
this end, we tested three techniques: Data Augmentation, Adversarial Training and Confidence-Aware Layer. These techniques aredesigned to be scalable, i.e. without major training and inference
penalties and applicable to a wide range of machine learning models.
We tested these approaches in different scenarios, such as, ASR error

DeMaL2021@KDD2021, 14–18 Aug, 2021, Virtual Conference

distribution drift, usage changes as reflected in NLU training andtest sets across different years and different recipient models. While
the results differ significantly from one scenario to another changing
the expected benefit depending on the quality of ASR engine andthe base model, in most cases we report meaningful performance improvements both as evaluated on ASR hypotheses and on annotatedtranscripts. These results demonstrate that for sufficiently complex
model robustness to ASR errors is achievable without overfitting
to a particular ASR error distribution. However, as the ASR engine
updates, periodic update of the ASR hypotheses to mitigate the effectof the distribution drift is becoming more important.
In future, we will continue experimenting with other techniquesapplicable to large-scale SLU systems as well as covering the rest of
the NLU components.

REFERENCES[1] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching Word Vectors with Subword Information. Transactions of the Association
for Computational Linguistics 5 (2017), 135–146.
[2] Yu-An Chung and James R. Glass. 2018. Speech2Vec: A Sequence-to-Sequence
Framework for Learning Word Embeddings from Speech. In Interspeech 2018,
19th Annual Conference of the International Speech Communication Association,
Hyderabad, India, 2-6 September 2018, B. Yegnanarayana (Ed.). ISCA, 811–815.
https://doi.org/10.21437/Interspeech.2018-2341[3] Anoop Deoras and Ruhi Sarikaya. 2013. Deep belief network based semantic
taggers for spoken language understanding. In INTERSPEECH 2013, 14th Annual
Conference of the International Speech Communication Association, Lyon, France,
August 25-29, 2013, Frédéric Bimbot, Christophe Cerisara, Cécile Fougeron,
Guillaume Gravier, Lori Lamel, François Pellegrino, and Pascal Perrier (Eds.).
ISCA, 2713–2717. http://www.isca-speech.org/archive/interspeech_2013/i13_
2713.html[4] Maryam Fazel-Zarandi, Longshaokan Wang, Aditya Tiwari, and Spyros Matsoukas. 2019. Investigation of Error Simulation Techniques for Learning Dialog
Policies for Conversational Error Recovery. CoRR abs/1911.03378 (2019).
[5] Dilek Hakkani-Tür, Frédéric Béchet, Giuseppe Riccardi, and Gökhan Tür. 2006.
Beyond ASR 1-best: Using word confusion networks in spoken language understanding. Comput. Speech Lang. 20, 4 (2006), 495–514. https://doi.org/10.1016/j.
csl.2005.07.005[6] Chao-Wei Huang and Yun-Nung Chen. 2019. Learning ASR-Robust Contextualized Embeddings for Spoken Language Understanding. CoRR abs/1909.10861(2019). arXiv:1909.10861 http://arxiv.org/abs/1909.10861[7] Faisal Ladhak, Ankur Gandhe, Markus Dreyer, Lambert Mathias, Ariya Rastrow,
and Björn Hoffmeister. 2016. LatticeRnn: Recurrent Neural Networks Over
Lattices. In Interspeech 2016, 17th Annual Conference of the International SpeechCommunication Association, San Francisco, CA, USA, September 8-12, 2016,
Nelson Morgan (Ed.). ISCA, 695–699. https://doi.org/10.21437/Interspeech.20161583[8] Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua
Bengio. 2019. Speech Model Pre-Training for End-to-End Spoken Language
Understanding. In Interspeech 2019, 20th Annual Conference of the International
Speech Communication Association, Graz, Austria, 15-19 September 2019, Gernot
Kubin and Zdravko Kacic (Eds.). ISCA, 814–818.
[9] Xuezhe Ma and Eduard Hovy. 2016. End-to-end Sequence Labeling via Bidirectional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the
Association for Computational Linguistics (Volume 1: Long Papers). Association
for Computational Linguistics, Berlin, Germany, 1064–1074. https://doi.org/10.
18653/v1/P16-1101[10] John Makhoul, Francis Kubala, Richard Schwartz, Ralph Weischedel, et al. 1999.
Performance measures for information extraction.
[11] Weitong Ruan, Yaroslav Nechaev, Luoxin Chen, Chengwei Su, and Imre Kiss.
2020. Towards an ASR Error Robust Spoken Language Understanding System. In
Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, Helen
Meng, Bo Xu, and Thomas Fang Zheng (Eds.). ISCA, 901–905.
[12] Dmitriy Serdyuk, Yongqiang Wang, Christian Fuegen, Anuj Kumar, Baiyang Liu,and Yoshua Bengio. 2018. Towards End-to-end Spoken Language Understanding.
In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2018, Calgary, AB, Canada, April 15-20, 2018. IEEE, 5754–5758.
[13] Prashanth Gurunath Shivakumar and Panayiotis G. Georgiou. 2019. Confusion2Vec: towards enriching vector space word representations with representational ambiguities. PeerJ Computer Science 5 (2019), e195. https://doi.org/10.
7717/peerj-cs.195

Yaroslav Nechaev, Weitong Ruan, and Imre Kiss

[14] Prashanth Gurunath Shivakumar, Mu Yang, and Panayiotis G. Georgiou. 2019.
Spoken Language Intent Detection using Confusion2Vec. CoRR abs/1904.03576(2019). arXiv:1904.03576 http://arxiv.org/abs/1904.03576[15] Edwin Simonnet, Sahar Ghannay, Nathalie Camelin, and Yannick Estève. 2018.
Simulating ASR errors for training SLU systems. In Proceedings of the EleventhInternational Conference on Language Resources and Evaluation, LREC 2018,
Miyazaki, Japan, May 7-12, 2018, Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Kôiti Hasida, Hitoshi Isahara, Bente
Maegaard, Joseph Mariani, Hélène Mazo, Asunción Moreno, Jan Odijk, Stelios
Piperidis, and Takenobu Tokunaga (Eds.). European Language Resources Association (ELRA). http://www.lrec-conf.org/proceedings/lrec2018/summaries/827.
html[16] Meet Soni, Sonal Joshi, and Ashish Panda. 2019. Generative Noise Modeling andChannel Simulation for Robust Speech Recognition in Unseen Conditions. Proc.
Interspeech 2019 (2019), 441–445.
[17] Natalia A. Tomashenko, Antoine Caubrière, Yannick Estève, Antoine Laurent,and Emmanuel Morin. 2019. Recent Advances in End-to-End Spoken Language
Understanding. In Statistical Language and Speech Processing - 7th International
Conference, SLSP 2019, Ljubljana, Slovenia, October 14-16, 2019, Proceedings(Lecture Notes in Computer Science, Vol. 11816), Carlos Martín-Vide, Matthew
Purver, and Senja Pollak (Eds.). Springer, 44–55.
[18] Gökhan Tür, Anoop Deoras, and Dilek Hakkani-Tür. 2013. Semantic parsing using
word confusion networks with conditional random fields. In INTERSPEECH 2013,
14th Annual Conference of the International Speech Communication Association,
Lyon, France, August 25-29, 2013, Frédéric Bimbot, Christophe Cerisara, Cécile
Fougeron, Guillaume Gravier, Lori Lamel, François Pellegrino, and Pascal Perrier(Eds.). ISCA, 2579–2583. http://www.isca-speech.org/archive/interspeech_2013/
i13_2579.html[19] Xiaohao Yang and Jia Liu. 2015. Using word confusion networks for slot
filling in spoken language understanding. In INTERSPEECH 2015, 16th Annual Conference of the International Speech Communication Association, Dresden, Germany, September 6-10, 2015. ISCA, 1353–1357. http://www.iscaspeech.org/archive/interspeech_2015/i15_1353.html[20] Zhilu Zhang and Mert R. Sabuncu. 2018.
Generalized Cross Entropy
Loss for Training Deep Neural Networks with Noisy Labels. In Advancesin Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December
2018, Montréal, Canada, Samy Bengio, Hanna M. Wallach, Hugo Larochelle,
Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (Eds.). 8792–
8802.
http://papers.nips.cc/paper/8094-generalized-cross-entropy-loss-fortraining-deep-neural-networks-with-noisy-labels[21] Su Zhu, Ouyu Lan, and Kai Yu. 2018. Robust Spoken Language Understanding
with Unsupervised ASR-Error Adaptation. In 2018 IEEE International Conferenceon Acoustics, Speech and Signal Processing, ICASSP 2018, Calgary, AB, Canada,
April 15-20, 2018. IEEE, 6179–6183. https://doi.org/10.1109/ICASSP.2018.
8461831