Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors
Authors: Marek Kubis
Paweł Marek Skórzewski
Marcin Sowanski
TOMASZ ZIĘTKIEWICZ
Venue: EMNLP 2023 Main (2023)
Keywords: ASR
NLU
Speech Recognition
TTS
Back Transcription
Evaluation
Robustness
TL;DR: This paper proposes a method for investigating the impact of speech recognition errors on the performance of natural language understanding models.

Abstract: In a spoken dialogue system, an NLU model is preceded by a speech recognition system that can deteriorate the performance of natural language understanding.
This paper proposes a method for investigating the impact of speech recognition errors on the performance of natural language understanding models.
The proposed method combines the back transcription procedure with a fine-grained technique for categorizing the errors that affect the performance of NLU models.
The method relies on the usage of synthesized speech for NLU evaluation.
We show that the use of synthesized speech in place of audio recording does not change the outcomes of the presented technique in a significant way.

Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors
Marek Kubis
Marcin Sowański
Pawel Skórzewski
Tomasz Ziętkiewicz'
Adam Mickiewiez University, Poznań ul. Uniwersytetu Poznańskiego 4 61-614 Poznañ, Poland (mkubis, pms y@muu. edu. pl
Samsung Research Poland
Plac Europejski I 00-844 Warsaw, Poland
{m. sowanski,, ziet com
Abstract
NLU model robustness. Contrary conventional adversarial attacks, which aim determining the samples that deteriorate the model performance un- der study (Morris et al., 2020), our method also takes into considcration samples that change the NLU outcome in other ways. The robustness crite- ria that we formulate are then used to construct a model for detecting speech recognition errors that impact the NLU model in the most significant way.
a spoken dialogue system, an NLU model is preceded by a speech recognition system that can deteriorate the performance of natural lan- guage understanding. This paper proposes a method for investigating the impact of speech recognition errors on the performance of natu- ral language understanding models. The pro- posed method combines the back transcription procedure with fine-grained technique for cat- egorizing the errors that affect the performance of NLU models. The method relies on the us- age of synthesized speech for NLU evaluation. We show that the use of synthesized speech in place of audio recording does not change the outcomes of the presented technique in significant way.
The proposed method depends on speech pro- cessing models,
does not rely on the avail- ability of spoken corpora. Therefore, it suitable for inspecting NLU models for which only textual evaluation data are present. It makes use of the semantic representation of the user utterance, but it does not require any additional annotation of data. Thus, the dataset used for training and test- ing the NLU model can be repurposed for robust- ness assessment at no additional costs. For illus- tration, we decided to apply the presented method to Transformer-based models since they demon- strate state-of-the-at performance in the natural language understanding task, but the method does not depend the architecture of the underlying NLU model. The limitations of our approach are discussed the end of the paper.
1 Introduction
Regardless of the near-human accuracy of auto- matic speech recognition in general-purpose tran- scription tasks, speech recognition errors can still significantly deteriorate the performance of a natu- ral language understanding model that follows the speech-to-texl module in a conversational system. The problem is even more apparent when auto- matic speech recognition system from an external vendor is used as component of virtual assistant without any further adaptation. The goal of this paper to present method for investigating the impact speech recognition errors on the perfor- mance of natural language understanding models in a systematic way.
Related Work
Data augmentation method for improving the performance of neural models of vision, speech and language. Ma (2019) developed nlpaug, a tool that encompasses a wide range of augmentation techniques for text and au- dio signal. Morris et al. (2020) presented a frame- work for adversarial attacks for the NL. domain, called TextAttack, that can be utilized for data aug- mentation and adversarial training. The framework has modular design attacks can be built from four components: goal function, set of con- straints, a transformation, and a search method. It also provides oott-of-he--ee box implementations for
commonly employed
The method that we propose relies on the use of back transcription, a procedure that combines text-to-speech model with automatic speech recognition system prepare dataset contam- inated with speech recognition errors. The aug- mented dataset used to evaluate natural language understanding models and the outcomes of the eval- uation serve as a basis for defining the criteria of
The author performed the work while being affiliated with hoth organizations.

---

BACKTRANSCRIPTION
ROBUSTNESS ASSESSMENT
ERROR ANALYSIS
e(5)
TTS
((8)
AUDIO
MCDEL
NLU
EVALUATE
COMPARE
TEXT ALIGNMENT
ERROR MODEL
h(6)
a(s)
ASR
Criterfon
NEGATIVE
Crileroo (2) Inoorrec--inon einoomeet
IRRELEVANT
WHOLE
AFFIX
SPLITIOIN
oorrect
Figure 1: The proposed method that consists of back transcription, robustness assessment, and error detection.
several popular attacks described in the literature. Back translation (Sennrich et al., 2016) a tech- nique developed to improve the performance of neural machine translation. In this method, addi- tional monolingual data for training the model are obtained by translating utterances from the target language the source language.
errors. Liu et al. (2021) defined three aspects of NLU robustness (language variety, speech charac- teristics, and noise perturbation) and proposed a method of simulating natural language perturba- tions for assessing the robustness in task-oriented dialog systems. The influence of the noisy text in- put the NLU performance
also studied by Sengupta al. (2021), who tested intent classifica- tion and slot labeling models' robustness seven types of input data noise (abbreviations, casing, misspellings, morphological variants, paraphrases, punctuation, and synonyms). Peng et al. (2021) cre- ated a benchmark for evaluating the performance of task-oriented dialog models, designed to favor models with strong generalization ability, i.e., ro- bust to language variations, speech errors, unseen entities, and out-of-domain utterances.
The first experiments with augmenting ASR data with text-to-speech tools were conducted by Tjan- dra et al. (2017). They generated speech data from unpaired texts to create a paired speech-text cor- pus. Hayashi et al. (2018) proposed a novel data augmentation method for end-to-end ASR. Their method uses a large amount of text not paired with like manner.
speech signals in The authors claim to achieve faster attention learn- ing and reduce computational costs by using hid- den states as a target instead of acoustic features. Laptev et al. (2020) built a TTS system on an ASR training database, then synthesized speech to ex- tend the training data. The authors distinguish two main approaches to ASR: hybrid (DNN+HMM) and end-to-end. They noted that both approaches achieve similar performance when there large amount of training data (Lüscher et al., 2019), but hybrid models perform better than end-to-ent mod- els when the amount data smaller (Andrusenko et al., 2020). Park et al. (2021) construct a parallel corpus of texts and their back-transcribed counter- parts for the purpose of training a post-processor for ASR system. They show that their method is effective in correcting spacing, punctuation and foreign word conversions.
Method
The proposed method of evaluation consists of three stages: the execution of the back transcription procedure that transfers NLU data between text and audio domains, the automatic assessment of the out- come from the NLU model on per-sample basis, and the fine-graine -reigga method of inspecting the re- sults with the use of edit operations (see Figure 1). The method relies on the availability of an NLU dataset containing user utterances along with the semantic representation of the uttered commands. We do assume any particular annotation scheme for the NLU commands. However, in the exper- iments discussed in Section 4, we work with the dataset that provides conventional annotations for domains, intents, and slots.
Recently, there have been several papers address- ing the issue of the robustness of natural language understanding systems to various types of input

---

3.1 Back Transcription
2. I+1
The back transcription procedure applied with re- spect to the NLU dataset consists of three steps. First, textual data are synthesized with the use of text-to-speech model. Next, the automatic speech recognition system converts the audio signal back text. In the last step, both the input utterances and the recognized texts are passed to the NLU model to obtain their semantic representations.
An incorrect result returned for the reference text is replaced by another incorrect result in the case of the back-transcribed text, i.e. b(s) + e(s) a(s) + e(s) A b(s) + a(s).
3. 1-C
An incorrect result obtained for the reference text is changed a correct result in the case of the back bocc-transcribed text, i.c. b(s) + E(S) A a(s) e(s).
The NLU commands are tracked in consecutive steps. As result, we obtain an augmented NLU dataset providing the following data for each sam- ple s:
The first category is always considered to have a negative impact on the robustness of the NLU model. However, with respect to + and + C' categories of samples, alternative options can be considered. For I + samples, it is reason- able to treat them as negative if we want to obtain the definition of robustmess that penalizes changes. It is also sensible consider them be irrelevant since such samples do not affect the performance of the NLU model. -> samples, once again, can be considered to be negative if we want to penalize all changes. They can be treated as irrelevant, mak- ing the definition of robustness unaffected by the changes that improve the performance of the NLU model. Finally, they can also be considered to have positive impact the robustness of the model since they improve the NLU performance.
1.
r(s): the reference text that comes from the initial NLU dataset;
2. h(s): the hypothesis, i.e. the r(s) text syn- thesized with the text-to-speech model and transcribed with the automatic speech recog- nition system;
3. e(s): the expected outcome of the NLU model for r(s) as given in the initial NLU dataset;
4. b(s): the outcome of the NLU model for r(s) (i.e. the NLU result for the text before back transcription);
5. (s): the outcome of the NLU model for hss) (i.c. the NLU result for the text after back transcription).
3.3 Problems of A-Messurements
A common practice of measuring the difference in accuracy before and after back transcription treats + C' samples as positive and ignores - sam- ples. Such a procedure underestimates the impact of C + I samples on the NLU module due to the performance gain introduced by / C samples. It also does not track - I changes which can deteriorate the behavior of downstream modules of dialogue system that consume the outcome of the NLU model. As we show in Section 4, - and + C cases account respectively for up to 30% and 10% of all the changes introduced by back tran- scription. Thus, the decision to ignore or promote them should be result of careful planning.
3.2 Robustness Criteria
A simple method for coarse-grain assessment of NLU robustness relies on measuring performance drop with respect to the commonly used metrics such as accuracy for intent classification F-score for slot values extraction. This a widely accepted practice in the case of adversarial attacks (Morris et al., 2020); however, such a procedure does not distinguish specific cases that arise in the compari- son of the model outcomes. Let us divide samples that differ in NLU outcomes obtained for reference utterances and their back-transeribed counterparts along three criteria:
The relationship between the outlined categories of changes and the building blocks of the F-score is even more complicated. Let C., -> Ig denote the change from correct label a incorrect label 3, Ta > T3 the change from incorrect label a to incor- rect label 3, and lo + C3 the change from incor- rect label o to correct label 3. Let TP, FP, FN,,
1. +
A correct result obtained for the reference text is changed to an incorrect one in the case of the back-transcribed text, i.c. b(s) e(s) A a(s) + e(s).

---

Category
TPa
FP
FNa
TP3
FNg
Pa
Ra
P3
R
Ca lg
+
+
+
la + I1
f
La Cg
Table 1: Relationship the building blocks of F-measure and the robustmess criteria.
Name
I-1
I-C
Domain (D)
Definition
R123
negative
negative
{s: h(s) + r(s))
8:8C Drb(a)
1
{8 Va(s)==(s)]]
(b(s) c(s)
soNE (Ab(s) (x))
R13
irrelevant
negative
1
(s ()s))(((s)))))))) r(()) # d(())
a (sbs))((s)=a
R12
negative
irrelevant
11
Ri
irrelevant
irrelevant positive
{s h(s) + ((s) <(S)}
1| H DAb(s) (x))
1
R1231
negative
{s: h(s)) # r(s))
HaSNE (M [
h(s) + r(s) (b(s) e(s)
(M)) (aaa)
R13+
irrelevant
positive
V Va(s) = c(s)))
Table 2: NLU robustness measures.
Pi and Ri denote true positives, false positives, false negatives, precision and recall with regard to label I. Table 1 shows how the building blocks of F-measure change due to the changes in NLU outcome between the reference utterances and their back-transcribed counterparts. One can observe that for each category of changes, some compo- nents of the F-measure increase (t) while others decrease (4) or remain unchanged (-). Therefore, measuring the difference in F-scores between refer- ence utterances and bock-transcribed texts leads to results that difficult to interpret meaningfully.
sonable if we assume that the downstream module behaves correctly when presented with correct in- put. However, if the downstream module relies the outcome of NLU regardless of its status I then R123 should be preferred. R12- which penalizes changes between incorrect labels but neglects the impact of I + changes, is a rational choice for assessment of an NLU model that precedes a down- stream module dedicated to correcting incorrect NLU outcomes such as a rule-based post-processor. R1, which penalizes the drop in accuracy but ne- glects any other changes, is metric that tracks the volume of samples that become incorrect due the use of an ASR system. Therefore, it is suitable for monitoring the regressions of the ASR-NLU pair across consecutive revisions of the ASR model. The penalization of positive changes by Rı3 makes this metric also reasonable choice for tracking the robustness of NLU models that should act consis- tently in the presence of reference texts and their transcribed counterparts. This is the case of NLU models that are designed to handle both the input typed by the user and the input that comes from an ASR system. The same holds for R123, which, contrary to R13, also takes into account the impact of I + I changes on downstream modules of a dialogue system.
3.4 Robustness Assessment
Proper combinations of the aforementioned cate- gories of NLU outcome changes lead to six alterna- tive robustness measures with their own rationale. We present them in Table 2 and report their values for the NLU models under study in Table The R13+ measure, which neglects / - changes and treats I + changes as positive, is a counterpart measuring the difference accuracy for back- transcribed utterances and reference texts. This approach is sufficient for testing an NLU model in isolation, but it does not take into account that the behavior of downstream modules of dialogue sys- that consume the outcome of NLU model can deteriorate due to the change in labeling of incorrect results. Such cases are revealed by R123 and R123+. The second one promotes changing incorrect outcomes to correct ones, which rea-
'A common case in the industrial setting where NLU U re- sults are post-processed.

---

NLU model
TTS model
R123
R13
R12
R1
R123+
R13+
domain
FastSpeech
0.8583
0.8631
0.8704
0.8759
0.8722
0.8778
intent slots
FastSpeech
0.8017
0.8156
0.8131
0.8283
0.8157
0.8309
FastSpeech
0.3391
0.3902
0.3470
0.4021
0.3617
0,4199
domain
Tacotron
0.8903
0,9001
0.8977
0.9080
0.8985
0,9089
intent
Tacotron
0.8449
0.8616
0.8569
0.8752
0.8589
0.8772
slots
Tacotron
0.3663
0,4260
0.3744
0,4382
0.3878
0.4539
Table 3: NLU models robustness.
type
name del replace_(r)
description
example
delete a token
"a" - n
whole
replace token with string r insert word w before current token
"cat" + "hat"
word
insert_before_(w.
"cat" 4 "a cat"
operations
insert_after_ww)
insert word after current token
"cat" -> "cat that"
add add_prefix_(p)
prepend prefix p to the token append suffix s the token remove n characters from the end remove n characters from the start replace last len(s) (s) characters with 8 replace substring with string r
"owl" "howl"
add [dd_sufii__(s}
"he" + "hey"
affix
del_suffix_(n)
" "cats" + "cat"
operations
del_prefix_(n)
"howl" + "owl"
replace_suffix_(s)
"houl" + "hour"
sreplace_(s)_Ar}
"may" - "my"
join_(s}
join tokens using character split word after n-th character split word on first character
"run in" "run-in"
split/join operations
split_aftert_(n)
"today" "to day"
split_on_first_(c)
"run-in" "run in"
split_on_last_(c)
split word on last character
"forenoon" "for noon"
Table 4: Examples of edit operations.
3.5 Speech Recognition Errors Detection
the NLU outcome between the reference utterance and its back-transeribed counterpart is assessed in accordance with the criteria given in Section 3.2. Having a method for identifying differences be- tween reference utterances (U) and their back- transcriptions (bt(U)) and a set of guidelines for assessing the impact of the change in the NLU outcome as either positive or negative, we build logistic regression model with the goal to predict if the extracted edit operations deteriorate the robust- ness of the model (Y 1) or (Y 0) on the basis of the extracted edit operations (editops).
To detect speech recognition errors that deterio- rate the robustness of the NLU model in the most significant way, we determine the differences be- tween the reference texts and their tack-transerieed counterparts and confront them with the impact caused by the change in the NLU outcome. For identifying the differences between reference and back-transcribed utterances, we align them with the use of the Ratcliff-Obersheh algorithm (John W. Ratcliff, 1988). Alignment identifies spans of both texts which are different (either inserted, miss- ing, or replaced). These spans are then recursively compared to identify differences on word level. Using handcrafted rules, the differences are con- verted into edit operations that transform incorrect words appearing in transcribed texts into correct words present in the reference utterances. The set of edit operations is modeled after similar sets presented in previous works on ASR error correc- tion (Ziętkiewicz, 2020, 2022; Kubis al., 2022). Types of operations, together with examples, are shown in Table The impact of the change in
Y (U
Afterward, we assess the impact of speech recog- nition errors on the robustness of the NLU model by extracting the regression coefficients that corre- spond to the edit operations that transform correct utterances into incorrect ones.
Framing the problem as supervised classifica- tion task has several advantages. First, it allows us to incorporate any combination of the criteria outlined in Section 3.2 into the detection process.

---

NLU model
TTS model
Metric
before BT
after BT
A
domain
PastSpeech
accuracy
0.92
0.88
-0.04
intent slots domain intent
PastSpeech
accuracy
0.88
0.82
-0.05
Peech
micro Fi
0.80
0.61
-0.19
Tacotron
accuracy
0.92
0.89
-0.03
Tacotron
accuracy
0.88
0.84
-0.04
slots
Tacotron
micro F1
0.80
0.61
-0.19
Table 5: NLU models performance.
Second, it allows to consider different dimen- sions of the semantic representation of NLU command, such as domain, intent, and slot values, either separately or in conjunction, enabling the evaluation of joint NLU models. Third, any classi- fication method that quantifies the importance of the features specified at the input can be used to study the impact of speech recognition errors on the robustness of the NLU model. We rely on logis- tic regression because the regression coefficients are easy interpret and the logistic model fits well the provided data. However, a more elaborate model such as gradient boosted trees (Friedman, 2001) could be used instead to prioritize speech recognition errors by feature impurity.
and augmented data and to measure the impact of back transcription the NLU performance.
4.2 Models
4.2.1 Natural Language Understanding
We trained three separate XLM-RoBERTa (Con- neau et al., 2020) models for three separate tasks: domain classification (Kubis et al., 2023a), intent classification (Kubis et al., 2023b) and slot fill- ing (Kubis et al., 2023c). Although joint mod- els present advantages over independent mod- els (Zhang et al., 2019), since these three tasks are interdependent, we chose train separate models to explore the impact of adversarial examples on each of these models separately. Following FitzGer- ald et al. (2022), we adopted the XLM-RoBERTa model architecture and fine-tuned the models on the MASSIVE dataset. We chose this model because it can be compared to models presented in MAS- SIVE and achieves better results in a multilingual setting when compared to mBERT (multilingual BERT). We also considered DeBERTaV3 (He al., 2021), which outperforms both mBERT and XLM- RoBERTa on most NLP tasks, but when tested on the MASSIVE dataset, did perform worse.
4 Experiments
4.1 Data
Given that the back transcription technique does not require spoken data on the input, we decided to use the MASSIVE dataset (FitzGerald et al., 2022) to conduct the experiments. MASSIVE is multilingual dataset for evaluating virtual assis- tants created the basis of SLURP (Bastianelli al., 2020). It consists of 18 domains, 60 intents, and 55 slots. The English subset of MASSIVE includes 11515 train utterances, 2033 development utterances, and 2974 test utterances MASSIVE was designed in such a way that test utterances are well separated semantically and syntactically from the trainset, and therefore, the results of state-of- the-art models do not exceed 90% F1-score with just hyper paraee-parameters fine-tuning. This presents good opportunity test augmentation techniques like the one presented in this paper. We split data into train, validation, and test sets following the par- tition provided by MASSIVE. For learning NLU models, we use the train split. Finally, the test set is used to evaluate NLU models with respect to imitial
As base model for all tasks, we used the pre- trained version of multilingual XLM-RoBERTa that trained 2.5TB of filtered Common- Crawl data containing 100 languages. Adam (Kingma and Ba, 2015) was used for optimization with an initial learning rate of - 5. The Domain and Intent models were trained for 5 epochs and the Slot model was trained for 20 epochs. For the In- tent and Domain models, we tested scenarios where models were fine-tuned for more than 5 epochs, but none of them brought big improvements, while at the same time, this could decrease model gen- eralization powers. In contrast, FitzGerald et al. (2022) experimented with model training between and 30 epochs, which means that our models are tuned light-weighed, but as seen in the zero-
Datasett version 1.1 from https: https://hugginngaaee co/

---

shot scenario presented in the MASSIVE paper, the XLM-RoBERTa has remarkably good baseline generalization power, therefore we tried not to over- models to data which would harm measuring the robustness of speech recognition models.
4.3 Error Analysis
We report the robustness scores determined using the metries proposed in Section 3.2 for the NLU models under study in Table The results show that the evaluated models are far from perfect, even if we consider the most permissive metrics such as R1 which neglects I - and I + C' changes, and R13+, which rewards / + C. The poor perfor- mance of slot models suggests that the accuracy- based measures of robustness that qualify the whole sample as incorrect if any slot value changes an incorrect one due back transcription may be too restrictive. Devising more fine-grained robustness measures for slot models is one of the issues that we plan to investigate in the future. Table 6 gives in- sight into the impact of the formulated robustness criteria on the discussed measures. Considering that I + I samples are responsible for 9%-30% of changes in the NLU outcome after back transcrip- tion and that - C samples cause up 10% of changes, it clear that these criteria should not be neglected without a deliberate decision.
Table 5 reports the performance of NLU models measured with standard evaluation metries before and ufter back transcription (BT) is applied to the MASSIVE dataset. One can observe that the NLU models that we trained for the experiments demon- strate state-of-the-art performance with respect to the standard evaluation measures before the back transcription procedure is applied.
4.2.2 Speech Processing
Our evaluation method relies on combination of speech synthesis and automatic speech recognition models. For speech synthesis, we two mod- cls. The first, FastSpeech 2 (Ren al., 2021), is a non-autoregressive neural TTS model trained with the Fairseq 52 toolkit (Wang et al., 2021) on the LJ Speech 'ataset² (part of LibriVox, Ito and John- son, 2017). Text input is converted to phoneme sequences using the g2pE (Park and Kim, 2019) grapheme-tor to-honmmmeee library. Phoneme sequences are fed to the Transformer encoder, resulting in hidden sequence, which is then enriched with variance adaptor using duration, pitch, and energy information. The hidden sequence is then decoded with a Transformer decoder into mel spectrograms and waveforms with HiFi-GAN vocoder (Kong et al., 2020). The second TTS model is Tacotron 2 (Shen et al., 2018 8), trained on the same dataset (LJ Speech) and using the same vocoder (HiFFi GAN). In contrast to FastSpeech 2, Tacotron 2 is an auforegressive model trained directly on text in- put. Prediction mel spectrograms performed manner using recurrent
A qualitative comparison of the top 20 most fre- quent speech recognition errors demonstrated in Table 7 with the top 20 errors determined by the error detection model that treats all three criteria as negative (counterpart of metric R123) presented in Table shows that the rankings differ signifi- cantly. There are between 14 and 16 errors that ap- pear in the top 20 lists of the error detection model and are not present on the corresponding lists of the most frequent errors, e.g.: max| jeplace_macs). which is the result of homophone substitution; and ""[add_before bel__beffoooooree __do], which indicates the prob- lems caused by the missing period at the end of the utterance.
in a neural networks with attention. Automatic speech recognition is performed with Whisper (Radford et al., 2022), a weakly supervised model trained on a massive collection of 680,000 hours of labeled audio data from diverse sources. The model is reported generalize well on out-of-distribution datasets in training data and be robust domain changes and noise addition, making a challenging choice for our method.
4.4 Quality of Synthesized Audio
We also checked if the overall quality of the syn- thesized audio is acceptable. For this purpose, we back-transcribed the dataset with both TTS models. Then, for each TTS model, we randomly sampled 10% of the prompts for which the result of back transcription differed from the input. The back- transcribed prompts were presented to the annota- tor along with the original prompts and the record- ing of the TTS output. The goal of the annotator was to choose which of the two transcripts was closer the content of the recording. The order of the options was randomized so that the annotator did not know which was the original prompt and
co/facebook/
Bastsspecch2-ee--1jseecc
"https://hugginge httpp//aa.....eepppccccaaaaaaaa.
tts-tacotronn-ljsseech htttpp///wiggiiiaaa.e..oooooooor

---

NLU model
TTS model
C->T
T-1T
T
Const
domain
FastSpeech
133
14
16
2811
intent
FastSpeech
176
36
16
2746
slots
FastSpeech
507
227
26
2214
domain
Tacotron
104
19
10
2841
intent
Tacotron
134
37
17
2786
slots
Tacotron
509
233
26
2206
Table 6: The number of NLU outcomes that changed due to back transcription or remained constant (Const).
FastSpeech
Tacotron
FastSpeech
Tacotron
ollie ollie/replac__oo polly
mail|addd_preii___
pmlreplace_m.
pm|replace_ m. "" ""[add_befor" beforeeffee________ee doo]
mail _aalladdddeiii_____ prefix_e]
a[del]
chants(replace_suffie suffix_ce)
pm[replace m.]
olliefreplace olly]
bow](sreplace w_i]
emmy[replace amyl
at[replace add]
pm[replace_ m.]
enn[replace_nt nn.]
to-dolsplit_ ceoooppittoo_fiteeeteeee oon_irstt_]
any[del suftix_
only[sreplace I] in[del] pratfixed__eeiix_el i[del]
inner[replace inr]
[replace one] paul s|sreplace __eeeeee________________________ rarefreplace_rat
10[replace_ [__e]]]]]__]]__]]]]]]_
9am]del_ppet p_ppppeee
andrreplace_ing
dollarfach before_us] jefffadd suftix_reyy r_yl
tofde]]
maylreplace [ereppacce_maal]]] email]
afdel]
at repreeace add] afadd suffix n] 4freplace four] 10[replace ten] atfdd after__itt and[del]
reburne([replace burnettel at affadd after six] repaaee letlsletss 6am[del. prefix______ today's s[sreplace y' max max(replace macs] natic[replace_neg __ay]] sassy freplace__ _actaceeepee__eeeeeeeee___ pondicherry ssreplace___aa __rrer_rrrrr__ ordered] suffix_21 mr [add suftix..] itfreplace_al
today' sIsreplace y'__l yyl afjoin liceldel]
only (sreplacee I.]]
cnn(replace n.]
at[add after six]
heladd suffix_
and]replace suffix_
atfadd after six] and[del]
enlightening(replaee lighting]
all[replacc_on __oly]] olifreplace to-dol so-iitsplit_nee split_on pplit_on__ii first_-]
pondicherryssepllace
4freplace_fow
barnfdel_suffix___
hammdel__pree [ammfee_prrefx____
malifadd_suffix..o
I'IIfreplae_oll
light add__sffii_ss__ the[del] today's "isrrplace__" yy__
byldell willfdel] 430freplace_thing
his [del_ prefix_11
today's "aday's[sreppe s[sreplace_y' _y
Table 7: Top 20 most frequent errors.
Table 8: Top 20 errors that deteriorate R123.
which was the back-transcribed one. If both options were equally viable, the annotator was allowed to choose both as the answer.
synthesized samples similar to the scores ob- tained for the recordings. For this purpose, we con- ducted an experiment using audio samples from the SLURP dataset (Bustianelli et al., 2020), which contains recordings of over a hundred speakers gathered in acoustic conditions that match typical home/office environment with varying locations and directions of speakers toward the microphone array. Thus, the recordings use for evaluation are not overwhelmingly noisy, but they provide realistic use case for a virtual assistant.
Table 9 presents sample sizes (total), the number of TTS outputs recognized as closer to the origi- nal prompt (utr), closer the back mack-transcribed text (aug) or marked as close to both. The last column shows the percentage TTS outputs that resem- ble the original prompt (i.e., wdt++both). The results show that about 85% of TTS-generated readings of the selected prompts were good enough to at least such extent that the annotator indicated them to be either equally close closer to the original prompt than the recognized text, which confirms the acceptable quality of the synthesis. Furthermore, both TTS models have reached sim- ilar results, which implies that they can used interchangeably.
total
First, we applied the back transcription proce- dure to the text prompts extracted from SLURP. Next, we ran the ASR model on the audio record- ings corresponding to the extracted prompts and applied the NLU models to the transcribed texts. Finally, we compared the robustness scores calcu- lated for back- transcrende and transcribed texts. As shown in Table 10, the robustness scores obtained on TTS TTS-generated speech samples closely resem- ble those obtained on real speech samples from SLURP. The difference between scores is 0.03 on average and not greater than 0.08, which confirms
4.5 Robustness Scores for Voice Recordings
To confirm that TTS-generated speech samples can be used in place of voice recordings, we ver- ified that the robustness scores obtained for the

---

TTS
total
utt
aug
both
resemblance
Tacotron
121
73
19
29
84.30%
PastSpeech
115
66
16
33
86.09%
Table 9: TTS evaluation results.
NLU model
Audio
R123
R13
R12
R1
R123
R13
domain
SLURP
0.8675
0.8755
0.8786
0.8875
0.8802
0.8890
intent
SLURP
0.8209
0.8402
0.8348
0.8562
0.8375
0.8588
slots
SLURP
0.4308
0.4813
0.4413
0.4968
0.4547
0.5126
domain
FastSpeech
0.8590
0.8642
0.8710
0.8770
0.8727
0.8788
intent
FastSpeech
0.7962
0.8084
0.8137
0.8283
0.8177
0.8324
slots
FastSpeech
0.3629
0.4162
0.3705
0.4281
0.3835
0.4439
domain
Tacotron
0.8879
0.8966
0.8966
0.9059
0.8976
0.9069
intent
Tacotron
0.8302
0.8475
0.8446
0.8638
0.8473
0.8664
slots
Tacotron
0.3875
0.4476
0.3945
0.4585
0.4054
0.4714
Table 10: NLU models robustness determined with SLURP.
that using TTS- -eneereee speech samples in place audio recordings is justified, taking into consid- eration that Whisper ASR achieves the word error rates of 0.1625, 0.1121 and 0.1165 for SLURP, Tacotron and Fastspeech samples, respectively.
variation. Configuration of different systems might generate different outcomes, so it is more diffi- cult to draw general conclusions for other TTS and ASR systems used. Additionally, the set of possible variations will be, in most cases, limited to errors produced by TTS and ASR. While those types of errors were the primary goal of this article, it could be extended measure further NLU robustness problems.
5 Conclusion
this paper, we proposed a method for assessing the robustness of NLU models to speech recogni- tion errors. The method repurposes the NLU data used for model training and does not depend on the availability of spoken corpora. We introduced criteria for robustness that rely the outcome of the NLU model but do not assume any particu- lar semantic representation of the utterances. We showed how these criteria can be used to formu- late summary metrics and constructed an analyti- cal model that prioritizes individual categorics of speech recognition errors on the basis of their im- pact on the mrobutsobsss of the NLU model. Fi- nally, we performed an experimental evaluation of the robustness of transformer-based models and in- vestigated the impact of using text-to-speech mod- els in place of audio recording.
A Acknowledgements
This research was partially funded by the CAIMAC: Conversational Al Multilingual Augmentation and Compression project, a cooperation between Adam Mickiewicz University and Samsung Electronics Poland.
References
Andrei Andrusenko, Aleksandr Laptev, Ivan Meden- nikov. 2020. Towards a competitive end-to-end speech recognition for CHiME-6 dinner party tran- scription. Proc. Interspeech 2020, pages 319-323.
Alexei Baevski, Henry Zhou, AbdeIrahman Mohamed, and Michael Auli. 2020. Wav2vec 2.0: A framework for self-supervised learning of speech representations, In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS' 20, Red Hook, NY, USA. Curran Associates Inc.
Limitations
The presented method compares input utterances with the same input synthesized by TTS and pro- cessed by ASR. This setting introduces two limi- tations for the NLU component. First, the archi- tecture, the training data, and finally, the quality of TTS and ASR systems impact generated data
Emanuele Bastianelli, Andrea Vanzo, Pawel Swieto- janski, and Verena Rieser. 2020. SLURP: Spo- ken Language Understanding Resource Package. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

---

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451.
Marek Kubis, Zygmunt Vetulani, Mikotaj Wypych, and Tomasz Ziętkiewicz. 2022. Open challenge for cor- recting errors of speech recognition systems. In Human Language Technology. Challenges for Com- puter Science and Linguistics, pages 322-337, Cham. Springer International Publishing.
Aleksandr Laptev, Roman Korostik, Aleksey Svischev, Andrei Andrusenko, Ivan Medennikov, and Sergey Rybin. 2020. You do not need more data: Improving end-to-end speech recognition by text-to-speech data augmentation. In 2020 13th International Congress on Image and Signal Processing, BioMedical Engi- neering and Informatics (CISP-BMEI) pages 439- 444.
Jack FizzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rotimann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, et al. 2022. MASSIVE: A IM-example multilin- gual natural language understanding dataset with 51 typologically-diveer languages. arXiv preprint S.Xii:2204.08882
Jiexi Liu, Ryuichi Takanobu, Jiaxin Wen, Dazhen Wan, Hongguang Li, Weiran Nie. Cheng Li. Wei Peng, and Minlie Huang. 2021. Robustness testing of language understanding in task-oriented dialog. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2467-2480, Online, Association for Computational Linguisties.
Jerome H. Friedman. 2001. Greedy function approxi- mation: gradient boosting machine. The Annals of Statistics, 19(5:11189-133
Tomoki Hayashi, Shinji Watanabe, Yu Zhang, Tomoki Toda, Takaaki Hori, Ramon Astudillo, and Kazuya Takeda. 2018. bock-translation-stlee data augmen- tation for end-to-end ASR. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 426- 433.
Christoph Lüscher, Eugen Beck, Kazuki Irie, Markus Kitza, Wilfried Michel, Albert Zeyer, Ralf Schlüter, and Hermann Ney. 2019. RWTH ASR Systems for LibriSpeech: Hybrid vs Attention. In Proc. Inter- speech 2019, pages 231-235.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021, I, DeBERTaV3: Improving deBERTa using ELECTRA- style pre-training with ding sharing.
embed-
Edward
Ma. 2019.
NIp augmentation.
Keith Ito and Linda Johnson. 2017. The LJ speech dataset.
https://keii thieeihtittt tociito.coom com/
LJ-Speech- -SSpeecc--aasset/.....
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. Textattack: A frame- work for adversarial attacks, data augmentation, and adversarial training in NLP. In Proceedings of the 2020 Conference Empirical Methods in Natu- ral Language Processing: System Demonstrations, pages 119-126.
David E. Metzener John W. Rateliff. 1988. Pattern matching: The gestalt approach.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A
method for stochastic optimization. In Proc. of the 6th International Conference on Learning Represen- tations (ICRL 2015), San Diego, CA.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEÉE international conference acoustics, speech and signal processing (ICASSP), pages IEEE
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. Hifi-GAN: Generative adversarial networks for ef- ficient and high fidelity speech synthesis. In Pro- ceedings of the 34th International Conference on Neural Information Processing Systems, NIPS 20,
Red Hook, NY, USA. Curran Associates Inc.
Chanjun Park, Jachyung Seo, Seolhwa Lee, Chanhee Lee, Hyeonseok Moon, Sugyeong Bo, and Heuiseok Lim. 2021. BTS: Back TranScription for speech-to- text post-processon using text-to-speech-to-eee Proceedingy of the 8th Workshop on Asian Transla- tion (WAT2021), pages 106-116, Online. Association for Computational Linguistics.
Marek Kubis, Pawel Skórzewski. Marcin Sowañski, and Tomasz Zietkiewiez. 2023a. LLM-RoaERTa domain classification model trained on MASSIVE dataset (revision 8cdbdfa).
In
Marek Kubis, Pawel Skórzewski, Marcin Sowaríski, and Tomasz Ziętkiewicz. 2023b. XLMRRoBERTa intent classification model trained on MASSIVE dataset (revision 26d0272)
Kyubyong Park and Jongseok Kim. 2019. g2pE. https: //gi thub. wwt.//oowwog///////eeo g2p.
Marek Kubis, Pawel Skórzewski, Marcin Sowaríski, and Tomasz Ziętkiewicz. 2023c. XLMMRoBERTa slot filling model trained on MASSIVE dataset (revision 5d94h64).
Baolin Peng, Chunyuan L., Zhu Zhang, Chenguang Zhu, Jinchao Li, and Jianfeng Gao. 2021. RADDLE: An evaluation benchmark and analysis platform for robust task-oriented dialog systems. In Proceedings

---

of the 59th Annual Meeting of the Association for Computational Linguistics and the 1ith International Joint Conference Natural Language Processing (Volume I: Long Papers), pages 4418-4429, Online. Association for Computational Linguistics.
Tomasz Ziętkiewicz. 2020. Post-editing and rescor- ing of ASR results with edit operations tagging. In Proceedings of the PolEval 2020 Workshop, pages 23-31.
Tomasz Ziętkiewicz. 2022. Tag and correct: high pre- cision post-editing approach to speech recognition emoors correction. In Proceedings of the 17th Confer- ence on Computer Science and Intelligence Systems. volume 30 of Annals of Computer Science and Infor- mation Systems, pages 239-942. IEEE
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak su- pervision. Published as preprint https://cdn. openai. com/ papers/ sapperswwhih http//wwiiepppppeee .dff.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. 2021. Fastspeech 2: Fast and high-quality end-to-end text to speech. In 9th International Conference Learning Represen- tations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. Oeenneview.net.
A Wav2Vee Experiments
Due to the limited space the main text presents only the back transcription experiments conducted with the use of Whisper (Radford et al.. 2022) ASR model. However, in order to verify that the presented evaluation method generalizes to other speech recognition models we also carried out periments with Wav2vec 2.0% (Baevski al., 2020), model pre-trained in self-supervised manner on audio-only data from LibriVox (LV60k) using masking of latent speech representation and fine- tuned on labeled data (LibriSpeech 960h, Panay- otov et al., 2015) for the speech recognition task. Following the main text we report the robustness results determined with the use of Wav2Vec model in Table 11 and 12 and present back transcription evaluation results in Table 13 and 14.
Sailik Sengupta, Jason Krone, and Saab Mansour, 2021. On the robustness of intent classification and slot la- beling in gral-oriented dialog systems to real-world noise. Proceedings of the 3rd Workshop on Nat- ural Language Processing for Conversational AI, pages 68-79, Online. Association for Computational Linguisties.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers). pages 86 66.96, Berlin, Germany, Association for Computational Lin- guistics.
Jonathan Shen, Ruoming Pang. Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, Rif A. Saurous, Yannis Agiomvrgiannkkis, and Yonghui Wu. 2018. Natural TTS synthesis by con- ditioning wavenet MEL spectrogram predictions. In 2018 IEEE International Conference on Acous- fics, Speech and Signal Processing (ICASSP), pages 4779-4783.
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. 2017. Listening while speaking: Speech chain by deep learning. 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 301-308.
Changhan Wang, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Ann Lee, Peng-Jen Chen, Jiatao Gu, and Juan Pino. 2021. Fairseq $2: A scalable and inte- grable speech synthesis toolkit. In Proceedings of the 2021 Conference Empirical Methods in Nat- ural Language Processing: System Demonstrations, pages 143-152, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Zhichang Zhang, Zhenwen Zhang, Haoyuan Chen, and Zhiman Zhang. 2019. A joint learning frame- work with BERT for spoken language understanding. IEEE Access, 7: 7:168849-16888 168858.
"https://huginggfaa co/ facebook// wav2vec2-base-966h

---

NLU model
TTS model
R123
R13
R12
R1
R123+
R13-
domain
FastSpeech
0.7829
0.7884
0.7906
0.7967
0.7926
0.7988
intent
FastSpeech
0.6903
0.7030
0.7016
0.7159
0.7065
0.7210
slots
FastSpeech
0.3037
0.3436
0.3085
0.3506
0.3194
0.3634
domain
Tacotron
0.8050
0,8141
0.8156
0.8256
0.8181
0.8281
intent
Tacotron
0.7254
0.7379
0.7389
0.7533
0.7437
0.7583
slots
Tacotron
0.3050
0.3473
0.3103
0.3549
0.3219
0.3688
Table 11: NLU models robustness.
NLU model
TTS model
C+1
1
I
Const
domain
FastSpeech
407
43
21
2503
intent
FastSpeech
543
94
35
2302
slots
FastSpeech
1093
384
34
1463
domain
Tacotron
332
45
27
2570
intent
Tacotron
449
82
38
2405
slots
Tacotron
1027
378
35
1534
Table 12: The number of NLU outcomes that changed due to back transcription or remained constant (Const).
TTS
total
utt
aug
both
utt+bboth
percentage
Tacotron
207
171
19
17
188
90.82%
FastSpeech
217
183
18
16
199
91.71%
Table 13:
TTS evaluation results.
NLU model
Audio
R123
R13
R12
R1
R1231
R131
domain
SLURP
0.6539
0.6590
0.6642
0.6700
0.6694
0.6755
intent
SLURP
0.5910
0.6012
0.6014
0.6131
0.6083
0.6206
slots
SLURP
0.2943
0.3304
0.3004
0.3391
0.3144
0.3561
domain
FastSpeech
0.7844
0.7901
0.7924
0.7988
0.7945
0,8010
intent
FastSpeech
0.7007
0.7089
0.7155
0.7256
0.7214
0.7319
slots
FastSpeech
0.3016
0.3428
0.3061
0.3495
0.3163
0.3618
domain
Tacotron
0.8024
0.8085
0.8130
0.8200
0.8155
0.8225
intent
Tacotron
0.7251
0.7373
0.7405
0.7548
0.7459
0.7605
slots
Tacotron
0.3034
0.3425
0.3086
0.3502
0.3203
0.3644
Table 14: NLU models robustness determined with SLURP.


Reviews:
[{"review_id": "ZqPCGkXSMD", "reviewer": "Reviewer_mLMm", "summary": "", "questions": "What is the ASR WER performance for synthesized and original speech?\n\nWill the authors make this dataset available to public?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "As for \"The authors do show however that there is very small difference in robustness between real and TTS generated speech. In my opinion, this should be surveyed in more detail and discussed in the paper.\" we agree with the reviewer that difference in robustness between real and TTS generated speech that we show is small. For the purpose of evaluating the proposed method (Sec. 4.5) we used the SLURP dataset which contains recordings of 100+ speakers gathered in acoustic conditions that match a typical home/office environment with varying locations and directions of speakers towards the mic array (cf. https://arxiv.org/pdf/2011.13205.pdf). Therefore, the recordings are not overwhelmingly noisy, but they provide a realistic use case for an intelligent personal assistant. Especially, taking into account that they encompass eighteen NLU domains. We will highlight the limitations of the dataset used for the study in the final version of the paper. We also welcome any suggestions for other open, multi-domain, spoken language understanding dataset that we can use along SLURP.\n\nAs for \"Using transformer based language models models for NLU modeling or Tacotron for speech synthesis is fairly common today. \",\nplease notice that we present \"a method for investigating the impact of speech recognition errors on the performance of natural language understanding models\". Therefore, It would be a methodological error on our side, if we proposed both a new family of NLU or TTS models alongside a new method for evaluating NLU models. We deliberately used state-of-the-art open TTS, ASR and NLU models to have a reproducible environment for conducting the experiments. An environment which is unbiased by in-house research on NLU or TTS models.\n\nAs for \"The only novelty is the dataset creation.\". Well, to be fair the dataset is not a novelty either. As in the case of TTS, ASR and NLU models we worked on well recognized datasets (MASSIVE/SLURP) with the intent of having a reproducible environment. However, the novel aspects of our method are as follows:\n\n1. In section 3.2. w introduce criteria for robustness formed on the basis of reference and back-transcribed utterances.\n2. In section 3.2 we construct a family of robustness metrics that exhibit different characteristics with regard to the modules that precede and follow the NLU model in a spoken dialogue system. We also show that a common practice of measuring the difference in accuracy for evaluating the impact of ASR on the NLU model is just one of six alternative options (R13+).\n3. In section 3.3 we propose a method for detecting speech recognition errors on the basis of the presented robustness criteria.\n\nAs for the questions: We will release the dataset on GitHub. Currently it is hidden due to anonymity requirements.\n\nThe ASR WER results for the entire SLURP dataset are as follows:\n1. Audio: 0.1625\n2. Tacotron: 0.1121\n3. Fastspeech: 0.1165\n\nThe ASR WER results for utterances that change in NLU outcome due to back-transcription are as follows:\n1. Audio: 0.4842\n2. Tacotron: 0.4362\n3. Fastspeech: 0.4758", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a dataset creation technique called Back transcription generate additional data points for NLU tasks by transcribing synthesized speech. This technique can be replicated for several spoken language processing tasks to augment datasets for evaluation. ", "reasons_to_reject": "The main aim of the paper is unclear to me. It is common for speech recognition errors to persist and degrade the performance of NLU  impacting the intent, domain and slot predictions and therefore the overall conversational task. The authors seem to propose the technique to augment datasets for evaluation of NLU tasks. There is significant difference between synthesized speech and real speech and the quality of transcripts for each scenario. The quality of ASR transcripts of real speech is often impacted by background noise, L1 backgrounds, vocabulary usage, etc. while in TTS the speech can be from a fixed speaker. The authors do show however that there is very small difference in robustness between real and TTS generated speech. In my opinion, this should be surveyed in more detail and discussed in the paper.\n\nIn terms of the range of experiments and evaluations, I think there is limited work demonstrated in the paper and the novelty is minor. Using transformer based language models models for NLU modeling or Tacotron for speech synthesis is fairly common today. The only novelty is the dataset creation. \n\nFurther, no metrics for ASR quality (WER performance) for real and synthesized speech are shared in the paper. \n\n", "questions_to_authors": "What is the ASR WER performance for synthesized and original speech?\n\nWill the authors make this dataset available to public?", "paper_topic_and_main_contributions": "In this paper, the authors investigate the impact of speech recognition errors on natural language understanding system by using back transcription technique that transcribes TTS generated speech using an ASR. The technique is investigated to simulate speech recognition errors and evaluate the quality of predictions from NLU models.   ", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Kv8JHxnnmp", "reviewer": "Reviewer_PdVj", "summary": "", "questions": "a.For the speech processing part, the authors use the pre-trained models for both TTS and ASR tasks. Were these models trained on any data from the dataset? What is the performance of these models? How noisy are the generated transcriptions?  For TTS generated audios, is there only human evaluation explained in Section 4.4? \n\nb. Can changing an ASR system resulting noisier transcript be more dramatic to observe the model robustness?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please find below the answers to your questions:\n\nQ1. For the speech processing part, the authors use the pre-trained models for both TTS and ASR tasks. Were these models trained on any data from the dataset?\n\nThe authors of TTS and ASR models that we use did not indicate MASSIVE or SLURP as a source of training data.\n\nQ2. What is the performance of these models?\n\nWe selected state-of-the-art models for the tasks. The Whisper ASR model that we use achieved WER of 2.7 on LibriSpeech (https://cdn.openai.com/papers/whisper.pdf). The FastSpeech2 model achieved MOS of 4.18±0.06 on LJSpeech corpus (https://arxiv.org/pdf/2109.06912.pdf). Tacotron2 based models achieve MOS of up to 4.53 (https://arxiv.org/pdf/1712.05884.pdf).\n\nQ3. How noisy are the generated transcriptions?\n\nWe did not inject additional noise into the synthesized samples. Hence, they have to be considered clean.\n\nQ4. For TTS generated audios, is there only human evaluation explained in Section 4.4?\n\nYes, we considered the evaluation procedure presented in Section 4.4 to be sufficient taking into account that we use state-of-the-art pre-trained TTS models for the task.\n\nQ5. Can changing an ASR system resulting noisier transcript be more dramatic to observe the model robustness?\n\nYes, indeed. Please check Appendix A were we report results for a weaker ASR system.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper investigates the robustness of NLU using a back transcription method and showed that using synthetic audio didn’t change the result compared to using recording. This findings is a proof to show that labeled audio data is not necessary to investigate the robustness. Using several TTS models also supports the claim, since they don’t possess much differences in the result. Although the authors use the XLM-Roberta model to test their proposition, the method is modular so that each method used in the modules can be changed.", "reasons_to_reject": "a. As indicated in Section 3, the proposed method requires a dataset with user utterances and semantic representation of the uttered commands. This requirement can limit the usage of the model especially with the low-resource languages. \n\nb. The choice of XLM-Roberta model is not convincing, since the paper investigates the robustness of the NLU model, not the best performing one. Along with XML-Roberta, other models can be also investigated on the MASSIVE dataset. Also, seeing the results other than MASSIVE dataset could be more supportive.", "questions_to_authors": "a.For the speech processing part, the authors use the pre-trained models for both TTS and ASR tasks. Were these models trained on any data from the dataset? What is the performance of these models? How noisy are the generated transcriptions?  For TTS generated audios, is there only human evaluation explained in Section 4.4? \n\nb. Can changing an ASR system resulting noisier transcript be more dramatic to observe the model robustness?", "paper_topic_and_main_contributions": "The paper proposes a new method to investigate the robustness of natural language understanding (NLU) models against speech recognition errors. The problem they tackle is the lack of assessment procedure for NLU systems. The proposed method that is based on the back transcription can be split into three steps: back transcription, robustness assessment and error analysis. The back transcription module takes a reference text, generates audio and its transcription. The reference and the transcription are the output of this module. The robustness assessment evaluates the model prediction using the expected result from reference text and the predicted result from transcription. For the assessment, they investigated three classes of changes. The choice of the criterias depends on how to penalize the model. Finally, the error analysis identifies the differences between the reference and transcription by aligning them. They build a logistic regression model to predict the edit operations between the two sequences. The experimental results showed that using generated audio samples vs recordings didn’t represent much differences.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YKBwA4v7fF", "reviewer": "Reviewer_jqhQ", "summary": "", "questions": "1. Have you considered evaluating the robustness of NLU models on real spoken data rather than relying solely on TTS-generated speech samples? How do you believe the findings might differ in such a scenario?\n\n2. The paper mainly focuses on Transformer-based models like FastSpeech and Tacotron. Have you considered comparing the robustness of NLU models with other NLU architectures or models? How do you think different architectures might perform in terms of robustness?\n\n3. The experiments are conducted on the MASSIVE dataset, which comprises 18 domains. Could you provide insights into the domain diversity and the potential implications of the findings on NLU robustness for other domains not covered by the dataset?\n\n4. The paper introduces robustness criteria and summary metrics. Have you considered incorporating additional evaluation metrics to capture other aspects of NLU model performance and robustness?\n\n5. While TTS-generated speech samples show reasonable resemblance to real speech, have you conducted any user studies or surveys to assess the perceptual quality and naturalness of the synthesized utterances? How confident are you that the TTS models sufficiently capture the nuances of real speech?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please find below the answers to your questions:\n\nQ1: Have you considered evaluating the robustness of NLU models on real spoken data rather than relying solely on TTS-generated speech samples? How do you believe the findings might differ in such a scenario?\n\nYes, we performed evaluation with the use of real spoken data from the SLURP dataset. We show the results in Table 8 (Section 4.5).\n\nQ2: The paper mainly focuses on Transformer-based models like FastSpeech and Tacotron. Have you considered comparing the robustness of NLU models with other NLU architectures or models? How do you think different architectures might perform in terms of robustness?\n\nTaking into consideration that Transformer-based models exhibit state-of-the-art performance in the NLU task, we think that the other NLU architectures will perform worse in the terms of robustness.\nUnfortunately, due to time constraints and limited space we focused solely on Transformer-based models, which are currently the most popular SOTA method.\n\nQ3: The experiments are conducted on the MASSIVE dataset, which comprises 18 domains. Could you provide insights into the domain diversity and the potential implications of the findings on NLU robustness for other domains not covered by the dataset?\n\nMASSIVE and SLURP cover utterances spanning 18 domains, 60 intents, and 55 slots. The range of covered domains is broad, it includes calendar, weather, transport, cooking, etc., as well as general topics.\nTaking into consideration that several categories of ASR errors occur across different domains in our data, it is reasonable to assume that the robustness for other domains will be at least partially related to the robustness determined for the domains under study.\n\nQ4: The paper introduces robustness criteria and summary metrics. Have you considered incorporating additional evaluation metrics to capture other aspects of NLU model performance and robustness?\n\nThe method for detecting speech recognition errors outlined in Section 3.3 can be adapted to other categories of errors. We consider generalizing it to errors that are not easily summarized by edit operations (e.g. restarts in spontaneous speech).\n\nQ5: While TTS-generated speech samples show reasonable resemblance to real speech, have you conducted any user studies or surveys to assess the perceptual quality and naturalness of the synthesized utterances? How confident are you that the TTS models sufficiently capture the nuances of real speech?\n\nWe present a study on quality of synthesized audio in Section 4.4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel Methodology: The paper presents a novel and creative method for evaluating NLU model robustness in the presence of speech recognition errors. By repurposing NLU training data and using TTS models to generate synthetic utterances, the paper overcomes the need for spoken data input, making it easier for researchers to study NLU robustness.\n\n2. Broad Applicability: The proposed method is not tied to any specific semantic representation, allowing it to be applied to various NLU models and architectures. This broad applicability makes it relevant to a wide range of NLP tasks and scenarios.\n\n3. Data Efficiency: The method utilizes existing NLU training data, which saves computational resources and reduces the need for additional labeled spoken data, especially for low-resource languages or domains where obtaining spoken data can be challenging.\n\n4. Insights into NLU Robustness: By quantifying the impact of speech recognition errors on NLU outcomes, the paper provides valuable insights into the robustness of Transformer-based NLU models. These insights can help researchers and practitioners understand how NLU models perform in real-world scenarios with imperfect speech recognition input.\n\n5. Potential for Further Research: The paper opens up avenues for further research in NLU model robustness assessment. The proposed criteria and summary metrics can be extended and adapted to investigate other sources of errors and vulnerabilities in NLU models.", "reasons_to_reject": "1. Limited Comparison: The paper mainly focuses on comparing Transformer-based models, specifically FastSpeech and Tacotron, for NLU model robustness. While this is insightful, a broader comparison with other NLU architectures or models would strengthen the paper's impact and generalizability.\n\n2. Overemphasis on TTS Models: The paper heavily relies on TTS models for generating synthetic utterances to evaluate NLU model robustness. While TTS-generated speech samples show reasonable resemblance to real speech, there may be inherent limitations in TTS systems, such as prosody and naturalness, which could affect the robustness assessment.\n\n3. Lack of Evaluation on Real Spoken Data: The paper primarily evaluates the NLU models on synthetic utterances generated by TTS models. While this approach is efficient, the absence of evaluation on real spoken data might limit the real-world applicability and generalizability of the findings.\n\n4. Limited Discussion on Error Types: The paper discusses specific types of speech recognition errors, such as insertions, deletions, and replacements, but it lacks a comprehensive analysis of all possible error types, which could provide a more holistic understanding of NLU robustness.\n\n5. Limited Domain Diversity: The experiments conducted on the MASSIVE dataset might lack domain diversity, potentially limiting the generalizability of the findings to other domains and real-world applications.", "questions_to_authors": "1. Have you considered evaluating the robustness of NLU models on real spoken data rather than relying solely on TTS-generated speech samples? How do you believe the findings might differ in such a scenario?\n\n2. The paper mainly focuses on Transformer-based models like FastSpeech and Tacotron. Have you considered comparing the robustness of NLU models with other NLU architectures or models? How do you think different architectures might perform in terms of robustness?\n\n3. The experiments are conducted on the MASSIVE dataset, which comprises 18 domains. Could you provide insights into the domain diversity and the potential implications of the findings on NLU robustness for other domains not covered by the dataset?\n\n4. The paper introduces robustness criteria and summary metrics. Have you considered incorporating additional evaluation metrics to capture other aspects of NLU model performance and robustness?\n\n5. While TTS-generated speech samples show reasonable resemblance to real speech, have you conducted any user studies or surveys to assess the perceptual quality and naturalness of the synthesized utterances? How confident are you that the TTS models sufficiently capture the nuances of real speech?", "paper_topic_and_main_contributions": "This paper addresses the problem of assessing the robustness of Natural Language Understanding (NLU) models to speech recognition errors. The main contribution of the paper is the proposal of a method to evaluate NLU model robustness without the need for spoken data input. Instead, the method repurposes the NLU training data and uses Text-to-Speech (TTS) models to generate synthetic utterances. These synthetic utterances are then processed by Automatic Speech Recognition (ASR) models to simulate speech recognition errors. By comparing the NLU model's performance on the original text and the TTS-generated text, the paper quantifies the impact of speech recognition errors on the NLU outcomes.\n\nThe paper introduces criteria for robustness that rely on the outcome of the NLU model without assuming any specific semantic representation. It formulates summary metrics based on these criteria to prioritize different categories of speech recognition errors based on their impact on NLU model robustness. The paper also evaluates Transformer-based NLU models using this method and investigates the use of TTS-generated speech samples in place of audio recordings.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eac6chAN1P", "reviewer": "Reviewer_ki6r", "summary": "", "questions": "It would be good to provide ASR error rates.\n\nIf you could identify a specific type of ASR error which is impacting on NLU performance, how would you fix it?\n\nSome ASR developers are now using TTS to augment their training data.  How would this impact on your analysis method?\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "It is stated in the review that \"Table 8 provides a large set of numbers to 4 figure accuracy, but its not clear to me that there is any information which could not have been deduced from simple ASR WER rates and standard NLU F1 measures. The paper would be much more compelling if WER/F1 rates had been provided, and the additional value of the proposed metrics clearly demonstrated.\"\n\nIn order to clarify on this issue let us analyze three cases:\n\n1. Can ASR WER be used in isolation to deduce the robustness of an NLU model?\n\n    ASR WER does not track the NLU outcome. Hence, it cannot be used to deduce NLU performance.\n\n2. Can ASR WER be used in combination with F1 measure to deduce the robustness of an NLU model?\n\n    No. Unfortunately, due to the limited size of the paper we postponed the analysis of F1 measure to Appendix B. Table 13 shows how the building blocks of F-measure change due to the changes in NLU outcome between the reference utterances and their back-transcribed counterparts. One can observe that for each category of changes that we distinguished in Section 3.1 some components of F-measure increase (↑) while others decrease (↓) or remain unchanged (=). Therefore, measuring the difference in F-scores between reference utterances and back-transcribed texts leads to results that cannot be interpreted in a meaningful way. This is also the reason why we refrained from reporting delta F1 scores in the main part of the paper. However, F-scores for NLU models are shows in Table 14 (Appendix C). We will move the analysis of F-score from Appendix B to the main part to make it clear in the final version of the paper.\n\n3. Can ASR WER be used in combination with Exact Match Accuracy to deduce the robustness of an NLU model?\n\n    Yes. However, it is just one of six alternative measures (i.e. R_13+) that can be used to study the robustness of an NLU model in terms of the criteria outlined in Section 3.1. In Section 3.2 (lines 222-265) we provide **the insights** on using these alternative measures. Just to summarize:\n\n    a) R13+ measure neglects I → I changes and treats I → C changes as positive. Thus this approach is sufficient for testing an NLU model in isolation, but it does not take into account that the behaviour of downstream modules of a dialogue system that consume the outcome of an NLU model can deteriorate due to the change in labelling of incorrect results.\n\n    b) R123+ promotes changing incorrect outcomes to correct ones which is reasonable, if we assume that the downstream module behaves correctly when presented with correct input.\n\n    c) If the downstream module relies on the outcome of NLU regardless of its status, then R123 should be preferred in place of R123+.\n\n    d) R12 which penalizes changes between incorrect labels, but neglects the impact of C → I changes is a rational choice for assessment of an NLU model that precedes a downstream module dedicated to correcting incorrect NLU outcomes such as a rule-based post-processor.\n\n    e) R1 which penalizes the drop in accuracy, but neglects any other changes is a metric that tracks the volume of samples that become incorrect due to the use of an ASR system. Therefore, it is suitable for monitoring the regressions of the ASR-NLU pair across consecutive revisions of the ASR model.\n\n    f) The penalization of positive changes by R13 makes this metric also a reasonable choice for tracking robustness of NLU models that should act consistently in presence of reference texts and their transcribed counterparts. This is the case of NLU models that are designed to handle both the input typed by the user and the input that comes from an ASR system.\n\n    g) The same holds for R123 which contrary to R13 also takes into account the impact of I → I changes on downstream modules of a dialogue system.\n\nAs for the questions:\n\nQ1. It would be good to provide ASR error rates.\n\nThe ASR WER results for the entire SLURP dataset are as follows:\n\n1. Audio: 0.1625\n2. Tacotron: 0.1121\n3. Fastspeech: 0.1165\n\nThe ASR WER results for utterances that change in NLU outcome due to back-transcription are as follows:\n\n1. Audio: 0.4842\n2. Tacotron: 0.4362\n3. Fastspeech: 0.4758\n\nWe will report WER in the final version of the paper.\n\nQ2. If you could identify a specific type of ASR error which is impacting on NLU performance, how would you fix it?\n\nFor this purpose we can utilize the back-translated dataset to select samples to be included in the NLU model training/augmentation dataset. If you are interested in fixing a specific error, you can extract from the back-transcribed dataset hypotheses h(s) that contain this error along with the expected NLU outcomes e(s) and include them in the NLU training dataset.\n\nThe aforementioned procedure combined with the method for detecting speech recognition errors presented in Section 3.3 can be used to form the NLU training/augmentation dataset that addresses the errors that deteriorate the robustness of the NLU model in the most significant way.\n\nQ3. Some ASR developers are now using TTS to augment their training data. How would this impact on your analysis method?\n\nThe presented method does not rely on the use of a specific TTS model, hence one can counter this problem by selecting a TTS that was not used for augmenting ASR training data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is clearly written and presented and it provides a practical method of leveraging an NLU training set to also determine robustness to ASR errors.  The paper also provides some evidence that modern TTS systems such as Tacotron provide reasonable surrogates for natural speech in testing an ASR system.\n", "reasons_to_reject": "There is nothing very original in the paper.   The authors do provide a way of automatically generating analytics for an ASR-NLU tandem but they do not demonstrate any particular insights that are gained from the study.  Table 8 provides a large set of numbers to 4 figure accuracy, but its not clear to me that there is any information which could not have been deduced from simple ASR WER rates and standard NLU F1 measures.  The paper would be much more compelling if WER/F1 rates had been provided, and the additional value of the proposed metrics clearly demonstrated.\n", "questions_to_authors": "It would be good to provide ASR error rates.\n\nIf you could identify a specific type of ASR error which is impacting on NLU performance, how would you fix it?\n\nSome ASR developers are now using TTS to augment their training data.  How would this impact on your analysis method?\n", "paper_topic_and_main_contributions": "This paper uses a speech synthesiser to assess the impact of asr errors on a semantic decoder designed to extract domain/intent/slot values from natural language utterances.  Given a set of NLU training texts, each text is synthesised and then recognised to give a second possibly corrupted version of the text.  Various error metrics are proposed depending on whether the asr error generates an actual semantic error or serendipitously corrects an error in the source text.   In addition, heuristics are used to find a set of edit operations which restore the asr error.  Logistic regression is then used to relate the edit operation to its impact on NLU robustness.\n\nExperiments are conducted using the MASSIVE NLU dataset with XLM-RoBERTa decoders.  Two synthesisers, FastSpeech and Tacotron are tested alongside Whisper ASR.  Also naturally spoken utterances are extracted from SLURP to confirm that TTS input is representative of human speech input.  The results show that where errors occur in the source text, the ASR actually corrects a substantial number of them so simple NLU error rates do not tell the whole story.  They also claim that edit operation regression allows them to identify the type of ASR error that should be prioritised.  \n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 461 qualify -> quality ?\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work addresses the problem of assessing the robustness of Natural Language Understanding (NLU) models to speech recognition errors. The method proposed repurposes NLU training data and uses Text-to-Speech (TTS) models to generate synthetic utterances. These synthetic utterances are then processed by Automatic Speech Recognition (ASR) models to simulate speech recognition errors. By comparing the NLU model's performance on the original text and the TTS-generated text, the paper quantifies the impact of speech recognition errors on NLU.

Beyond automating combined ASR-NLU testing, one of the key contributions that the paper provides is in proposing a number of additional metrics that can be derived by this method and explaining the insights that they provide versus existing metrics. That said, more than one reviewer would prefer that WER/F1 rate be provided in the main text to make the argument for these new metrics more compelling.
Decision: Accept-Main