MCLF: A Multi-grained Contrastive Learning Framework for ASR-robust Spoken Language Understanding
Authors: Zhiqi Huang
Dongsheng Chen
Zhihong Zhu
Xuxin Cheng
Venue: EMNLP 2023 Findings (2023)
Keywords: Spoken Language Understanding
ASR Robustness
Multi-grained Contrastive Learning
Data Augmentation

Abstract: Enhancing the robustness towards Automatic Speech Recognition (ASR) errors is of great importance for Spoken Language Understanding (SLU). Trending ASR-robust SLU systems have witnessed impressive improvements through global contrastive learning. However, although most ASR errors occur only at local positions of utterances, they can easily lead to severe semantic changes, and utterance-level classification or comparison is difficult to distinguish such differences. To address the problem, we propose a two-stage multi-grained contrastive learning framework dubbed MCLF. Technically, we first adapt the pre-trained language models to downstream SLU datasets via the proposed multi-grained contrastive learning objective and then fine-tune it on the corresponding dataset. Besides, to facilitate contrastive learning in the pre-training stage, we explore several data augmentation methods to expand the training data. Experimental results and detailed analyses on four datasets and four BERT-like backbone models demonstrate the effectiveness of our approach.

MCLF:
A Multi-grained Contrastive Learning Framework for ASR-robust Spoken Language Understanding
Zhiqi Huang, Dongsheng Chen, Zhihong Zhu, and Xuxin Cheng
Peking University, China
zhiqihuang@pku. edu.cn cn
(chends, zhihongzhu, chengxx] @stu. pku. edu. on
Abstract
100
$3.00
25
Enhancing the robustness towards Automatic Speech Recognition (ASR) errors is of great importance for Spoken Language Understand- ing (SLU). Trending ASR-robust SLUU sys- tems have witnessed impressive improvements through global contrastive learning. However, although most ASR errors occur only local positions of utterances, they can easily lead to severe semantic changes, and utterance-level classification comparison is difficult to dis- tinguish such differences. To address the prob- lem, we propose two-stage multi-grained contrastive learning framework dubbed MCLF. Technically, we first adapt the pre-trained lan- guage models downstream SLU datasets via the proposed multi-grainer contrastive learn- ing objective and then fine-tune on the cor- responding dataset. Besides, to facilitate con- trastive learning in the pre-training stage, we explore several data augmentation methods to expand the training data. Experimental results and detailed analyses on four datasets and four BERT-like backbone models demonstrate the effectiveness of our approach.
S5.9 A0
90-
70
15
26.66
50-
RoBEKI& (Baclinc)
(Dues)
005
dlean
low (9. 9.1.66)
aodiun
high
10
(0.16,0.1
-0.4
Figure Comparison of RoBERTapsee and our MCLF on different WER intervals split from the SLURR test set. The performance of MCLF in the case of larger WER is greater than that RoBERTalaas-
been proposed, e.g., Speech Transformer (Dong et al., 2018), wav2vec (Baevski al., 2020), and SPIRAL (Huang et al., 2022a). Despite the suc- cess, ASR systems still generate transcripts with errors (mainly insertions, deletions, and substitu- tions errors) when facing more complicated real- world scenarios (Fang et al., 2020a). These ASR may confuse the downstream SLU tasks and lead performance degradation. Previous works towards ASR-robust SLU can be divided into categories: 1) Detecting and correcting ASR errors directly (Zhou al., 2022), or injecting additional ASR output information such as N-best hypothesis and phoneme sequence to the downstream SLU models (Li et al., 2020; Sundararaman et al., 2021; Wang et al., 2022b). However, both hypotheses with ASR errors and extra ASR information are hard to obtain due the limitation of the ASR system (Dutta al., 2022). 2) Adapting the ASR system architecture towards improving language correctness and fluency (Kim et al., 2021a; la; Le et al., 2021). An obvious shortcoming is the additional technical burden of modifying and retraining exist- ing ASR systems according to specific tasks.
1 Introduction
The rise in popularity of intelligent assistants like Apple Siri and Amazon Alexa has aroused interest in intelligent speech technology. A fundamental component of these smart assistants is the Spo- ken Language Understanding (SLU), framework that aims comprehend the semanties of human speech (Tur and Mori, 2011). Traditionally, seman- tic information from utterances are obtained by initially applying Automatic Speech Recognition (ASR) to convert human speech into text. After- wards, the SLU model is trained using this textual transcript input (Huang et al., 2021c; Chen et al., 2022a: Zhu al., 2023c.b).
Since speech-to-text has a wide range of appli- cation scenarios in both academia and industry, many ASR systems based on neural networks have
Recent pre-trained language models (PLMs) like

---

tal results and detailed analyses demonstrate the superiority and competitiveness of MCLF.
Rockbure
Pretmiing
Pre-trained MCLF
Fue-unin
Fine iine-tuned MCLF
In summary, the contributions this paper are three-fold: 1) We propose MCLF, which is the first attempt to combine global contrastive and fine-grained contrastive objectives for establishing ASR-robust SLU models; 2) Experiments four backhone models and four datasets demonstrate the effectiveness of our proposed method; 3) Detailed analysis shows that our proposed framework brings effectiveness to the robustness of noise data.
SLU danset
Auginstood
SI doneut
Figure 2: Overview of MCLF learning.
BERT (Devlin et al., 2019; Liu et al., 2019b) have shown appealing performance on various down- stream NLP tasks. However, most PLMs are trained on clean texts and are compromised in- evitably ASR errors. As shown in Figure 1, WER gets higher, the SLU model suffers se- vere classification performance degradation (from 95.69% to 56.71%) Taking ASR hypotheses as inputs may introduce an issue that words in an ut- terance may be misrecognized. For instance, fair and fare are acoustically similar, so an ASR system may fail to distinguish between them, resulting in a substitution error. Such substitution errors might be recovered by humans because humans are aware of the acoustic confusability of words. However, the errors may significantly degrade the testing per- formance when the models are trained on oracle transcripts (i.e., clean texts). order to enhance the ASR robustness in contextualized word em- beddings, Chang and Chen (2022) first introduced contrastive pre-training objectives with only tex- tual information input. However, they neglect finer-level alignment between the oracle transcript and the ASR hypothesis.
2 Approach
In this section, we first detailedly elaborate on the pre-training tasks used in MCLF (Figure 3) in Sec- tion 2.1. Then we introduce the proposed data aug- mentation methods for contrastive learning in the pre-training stage (Section 2.2). Finally, we briefly describe the fine-tuning stage Section 2.3.
2.1 Pre-training for MCLF
In Figure 3a, illustrate the architecture of MCLF as well as the three pre- pre-training objectives, namely masked language modeling (Cmim), global con- trastive learning (Cg), and fine-grained contrastive learning (Ly). The combination of Lg and L is called multi-grained contrastive learning.
2.1.1 Global Contrastive Learning
Contrastive learning has been proven effective to learn better representations in both computer vi- sion (Chen et al., 2020; Li et al.. 2021; Yao et al.. 2022) and natural language processing (Gao al., 2021; Kim et al., 2021 research. It aims learn highly discriminative features by pulling semanti- cally related samples together and pushing apart unrelated ones. Here we adopt contrastive learning to learn the ASR-robust utterance encoder f. which is initialized with the original RoBERTabass model. Assume training batch with B paired examples D (a=f,=9)) where (xg, xx9) denotes the i-th clean oracle transcript and ASR hypothesis pair. They are both fed into the shared encoder, and hę and hę represent the & normalized output embedding of the [CLS] token, respectively. We then apply contrastive learning over h; and hę by optimizing symmetric InfoNCE loss. The ASR- to-Oracle contrastive loss Lazo is:
In this work, we propose novel two-stage multi- grained contrastive learning framework (MCLF) for ASR-robust SLU (illustrated in Figure 2). Specifically, in the first stage, we devise fine- grained interactive module to align the tokens of the oracle transcript with the ones of the ASR hy- pothesis. We also leverage contrastive learning to facilitate global representation learning at the ut- terance level. Inspired by Fang et al. (2020b) that consider the phonetic confusion, we explore vari- ous task-specific data augmentation (DA) methods to mitigate the deficiency of large amounts of la- beled data for the PLM training procedure. In the second stage, we fine-tune the pre-trained MCLF model with several robust training techniques. We conduct extensive experiments on four benchmark datasets (i.e., SLURP, ATIS, TREC6, and TREC50) and four BERT-like backbone models. Experimen-
B
1
Luzo =
Σ log 1=1
Mooe DA details are in Scction 2.2 and Appendix B.
B
E,
(he hg/ra)

---

Fine-grained Contrastive b
La Lee
1. maaa
Token Wiken-wise Similarity
01
0g
A₃
Ag
Ag
Ag
0
Linear
A2
0
meaa
111
Au
0g
ASR to Oracle
Oracle to ASR Contractive
Contrastive
0.3
miim
Pooling
Lg
h
f
Oracle Encoder f()
shered
ASR Encoder f())
[CLS] when
hawaii become a state
[SEP]
[CLS] when did
hawaii became a estate [SEP]
(a) Overview of MCLF pre-training.
(b) MCLF fine-tuning.
Figure 3: (a) Overall schematic diagram of MCLF pre-training. MCLF is a dual-stream model consisting of a single weight-shared encoder /. Besides canonical objective masked language modeling (Cmton). we introduce ASR-Oracle global contrastive loss (C.g) and novel fine-grained contrastive loss (Ly) align the representations of a clean-noisy utterance pair at both instance-level and token-level. (b) The fine-tuning objective includes a cross-entropy loss (Coe), two supervised contrastive learning loss (C, and C1), and a self- distillation loss (Ca).
where = 0.2 is temperature parameter. Simi- larly, the OraRle-to-ASR contrastive loss La is:
m = maxo<r [f(x)) as its token- wise maximum similarity with z.. We then use the average moken-wise maximum similarity of all non- padded tokens in the ASR hypothesis (resp. Oracle transcript) as the similarity of ASR-to-Oracle (resp. Oracle-to-ASR). The similarity of the i-th ASR hy- pothesis to the j-th Oracle transcript can thus be formulated as:
B
eheThg/ra hg/re)
Loze = B
log
Σje e (hy hT/Te)
1=1
thus the ASR-Oracle global contrastive loss is:
Lg = (fa2o + Lo2a).
m1 Σ (OGN m1
(2)
2.1.2 Fine-grained Contrastive Learning
Besides the global contrastive alignment, it re- mains a challenge to establish finer-grained token- level ASR-Oracle alignment.
Similarly, the similarity of the j-th Oracle tran- script to the i-th ASR hypothesis is
Recent work CASLU (Wang et al., 2022b) proposes to utilize single cross-attention layer to explicitly model the fine-grained interactions between ASR hypotheses and phonemes sequence. However, it still relies additional phoneme information and is inefficient. To this end, inspired by FILIP (Yao et al., 2022) that learns fine- ane-grained image-text alignment through a cross-modal late interaction mechanism, we de- sign the fine-grained contrastive learning which strengthens the model's robustness towards the lo- cal word error caused by phonetic confusion.
fftt) f( Miggigg (fgg(
(3)
where mi arg maxosrcn Note that <6, in Equation (2) does not necessar-
[ [
ily equal o6 in Equation (3). Thus the fine-grained contrastive loss is:
B
L,
log
Σ cloigtTed
2B
1=1
(4)
cloletTe) log Σj E
Specially, denote nı and n2 the token number of the i-th ASR hypothesis and j-th Oracle tran- script, respectively, and the corresponding encoded features are f (a?) E xd and f (x/)) E xR For the k-th ASR token, we compute its similarity with all oracle tokens of 29.. and use the largest one
2.1.3 Overall Pre-training Objective
Following Gururangan et al. (2020) that lever- ages the masked language modeling (MLM) loss

---

for Domain-Adaptive Pretraining on downstream datasets, we keep the MLM loss in the pre-training for MCLF. Specially, the overall pre-training loss function is formulated as:
Algorithm 1: Procedure for Insertion Set and Confusion Set Construction
Input: Training set D {(O,, AD)
Params: O₁: the 8th Oracle transeript
Ay: the tth ASR hypothesis
S = Lgg + XyL Ammmm
(5)
T: the size of training set
where the x and Am are weight parameters.
Output: 1: the Insertion set
2.2 Data Augmentation for Contrastive Learning in Pre-training
C: the Confusion set
1: +- Ø; C. 2;C +
2: for t <-1 to T do
Traditionally, ASR system uses Word Error Rate (WER) as the performance metric to measure the percentage of incorrectly transcribed words. It is defined as follows:
It 1+=0
4:
m - -length(O))::
length(Ad)
5:
for + to n do
6:
if [AgJs & O2 then
WER I+D+S
7:
1, T..Inser((AA)])) end if
N
8:
where I, D. and denote the number of inserted words, deleted words, and substitution words in the ASR hypothesis, respectively. N is the number of words in the oracle text. From the perspective of WER, ASR errors can be divided into three cate-
9: end for
10:
I + IUL
11:
for i < to do
12:
if [Or]; $ At then
13:
C][O.]] < C[[O,]] h
gories (Fang al., 2020b): insertions (/), deletions (D), and substitutions (S).
14: end if
15: end for
In this work, we design different data augmen- tation (DA) methods for contrastive pre-training following different ASR error types. 1) Insertion DA. According to preliminary experiments, we find that some words are easier to appear in the ASR hypothesis. So we augment the data by repeat- ing the common words selected from the oracle transcript of training set. 2) Deletion DA. We ran- domly delete some words from the oracle transcript. 3) Substitution DA. We randomly replace some words from the oracle transcript.
16: end for
17: // Sort each word list in descending order of
longest common subsequence (LCS)
18: Keys C.keys()
19: for key in Keys do
20: Clkey] (kkcy))sor(LCCS(((ee)),. key))
21: end for
22: return I,C
First, we obtain the insertion words set and sub- stitution words set (i.e,, confusion set) which ac- quires a new set of words that each word is most likely to be replaced with based on the training set. The construction procedure for insertion set and confusion set is in Algorithm And we sort all of the confusion list for each word in descending order of longest common subsequence. Detailed sorting algorithm can be seen in Algorithm 2 in Appendix. Then, we perform different DA meth- ods (i.e., InsetiionDelleioo/Suustttuuuooi on part of sentences position. The target of position selection is to let WER reach certain value. Empirically, for sentences with length less than 5, we keep it unchanged, for sentences with length between and 10, we perform DA one random position, for sentences with length larger than 10, we perform DA in two random positions.
2.3 Fine-tuning MCLF for SLU Task
We further adapt the pre-trained MCLF model the corresponding SLU datasets in fine-tuning.
Following Gunel et al. (2021), we leverage the supervised contrastive learning (SCL) objective, which is meant to pull the examples of the same label closer and push away the examples from other labels. As shown in Figure 3b, two examples with green color (the same color represents the same label) should be closer in the latent space, and be pulled away from the red example. Such technique can make full use of the hard label information during the training process. Denote h, and h, as the Mean-Mas Max pooled features (Zhang et al., 2018) of all the output tokens from the utterance encoder.

---

The hard SCL loss is:
scenario and action used the evaluation metric of SLURP. Following Chang and Chen (2022). we use Google Web API (an off-the-shelf ASR system) to obtain the ASR hypothesis from the audio and adopt the same dataset split way. The median WER is 25% for Google Web API.
1
NYC
N
(h,, by/Te)
Ln.
log
N
9
. Σ A e (h? ba/Te)
Moreover, we adopt Chang and Chen (2022) that utilizes self-distillation to mitigate the impact of label noises in the training set by minimizing the KL divergence between current step prediction and the previous one. Denote p P( () as the posterior probability of data x; predicted by the model followed by a single linear layer the I-th epoch. The loss is formulated as:
ATIS ATIS is widely used SLU dataset with 22 different intents on flight reservation. The average WER for ATIS is 29.1 19.11%
TREC6 TREC50 TREC question classi- fication dataset with two versions, i.e., TREC6 (6 classes) and TREC50 (50 classes). The difference is that the class labels of TREC50 are more fine- grained than TREC6. The average WER for both datasets is 32.93%. The ASR hypotheses of both ATIS and TREC are synthesized by TTS and later transcribed by the ASR system, and they have been released by PhonemeBERT
N N
La
(p) (lp!).
where Pa is the one-hot vector of the label y/- We also consider using the soft supervised con- trastive learning as a complement of Cn, to further relieve the label noise (Yun et al., 2020):
3.2 Experimental Settings
N
N
( hy/To)
Backbone Models In our experiments, we choose four pre-trained language models with different architectures and pre-trained tasks as backbones, ie., 1) RoBERTa/ao. (Liu et al., 2019b); 2) BERT (Devlin al., 2019); 3) ELECTRAsmal (Clark et al., 2020); 4) TinyBERT, (Jiao et al., 2020). BERT the most widely used PLM in various NLP tasks. RoBERTa improves the pre-training of BERT with various optimization techniques. ELECTRA replaces the MLM task in BERT with the more compute- efficient replaced token detection pre-training task. Tiny BERT accelerates inference and reduces the model size compared with BERT through a novel Transformer distillation method. To demonstrate the consistent improvement brought by the pro- posed multi-graine contrastive objective, we con- duct experiments on all different backbones while keeping BERT's s related training parameters fixed, as can be seen in Appendix A.
C.
log
N
Lkzs e(h, he/7e)
Finally, the overall fine-tuning loss function is:
ft Lee + Seo(Ch + AdCs) + daha.
where Lee the cross-entropy loss, Ase and da are weight parameters.
3 Experiments
In this section, we evaluate the effectiveness of the proposed method four benchmark datasets, i.e., SLURP (Bastianelli et al., 2020), ATIS (Hemphill al., 1990), TREC6 and TREC50 (Li and Roth, 2002). Data statistics are shown Table 1.
Dataset
#Class
Avg. Length
Train
Test
WER
SLURP
18 46
6.93
50,628
10,992
25%
ATIS
11.14
4.978
893
29.11%
TREC6
8.89
5,452
500
32.93%
TREC50
8.89
5,452
500
32.93%
Training Details We follow Chang and Chen (2022) for the data split and preprocessing. The training batch size selected from (32, 64, 128) for each dataset. For each experimental setting, we pre-train the model with 20000 steps and then fine-tune it on the corresponding ASR hypotheses with 10 epochs. Unless otherwise specified, all
Table 1: Datasets statistics information.
3.1 Datasets
SLURP SLURP a publicly available multi- domain SLU dataset with a collection of 722k audio recordings. It includes 18 different scenar- ios and 46 defined actions. Joint accuracy of both
Pttpps: / //gl thub. com/ Observeal 1 Research/Phoneme-

---

PLM
Global CL
Fine-grained CL
SLURP
ATIS
TREC6
TREC50
Average
RoBERTabae
x
K
84.0410.19
94.4220.54
84.7610.67
75.08
84.58
X
85,33+0.88
94.93=0.46
87.0 87.0440.53
76.20
85.88
X
V
80.01
94.58.0.50
85.76
78. 08.72
86.02
/
85.39 1.0.28
95.22..0.5
87.00 L.0.2
78.84
86.62
BERT base
X
X
83.87
94.27=00.4
85.40
74.56
84.53
x
84.44
95.15
86. +0.37
85,55
X
V
84.13
94.83.40.44
86.04 1:0.02
48.40
85.85
84.85 200.1
94.95
86.40 20..5
78.92
86.28
ELECTRA ml
X
60.77
90.57=0.55
82.44
58.32
75.80
/
X
74.82.1:40
92.83
85.08
61. 61.12..0.4
78.46
/
82.70
94.25..0.6
84.20
1.22
83.36
82.76:0.24
95.00
85.56=1108
74.52
84,46
TinyBERT,
X
X
72.05
88.45
77. 77.8440.46
55 55.28
73.41
/
X
78. 14..13
93.4610.15
82.90
64. 10
79.65
10.10
95.00
82.64 10.84
70.60
82.51
V
81.83+0.15
95.35+0.24
84.04
72.60
83.46
Table 2: Average accuracy (%) across 5 seeds on the benchmarks under different pre-trained language models and their standard deviation. For each PLM, results in row I with fine-tuning only, and the results of to are pre-trained first (all equipped with MLM loss by default) and then fine-tuned.
experimental training processes are kept consistent, Detailed experimental settings are in Appendix A.
Method
SLURP
ATIS
TREC6
TREC50
RoBERTa
83.97
94.53
PRRT
83.78
94.83
85.96
76.16
SimCSE
84.47
94.07
84.92
74.82
SpokenCSF
85.26
95.10
86.36
76.20
3.3 Main Results
MCLF (Ours)
85.39
95.22
87.00
78.84
Table 3: Accuracy (%) comparison of baseline works and the proposed MCLF on four datasets. For a fair comparison, all of the methods in this table employ RoBERTapase model as the backbone.
In Table 2, we show the performance of four back- bones introduced in Section 3.2 under different pre-training objective settings Equation (5), i.e.. I) no pre-training stage; 2) with only Ly and Lmimi 3) with only Cy and Lmtmi and 4) with the full pre- training objective. And they correspond to row row for each backbone, respectively. As can be seen, when compared with the baseline result in row 1, using global contrastive learning (global CL) and fine-grained CL respectively in pre-training can boost the performance of all datasets for all back- bone models, and combining both Ly and Σ the same time brings further gains, which indicates effectiveness of the proposed multi-grained con- trastive learning. Another observation that using our proposed MCLF upon smaller backbome (i.e., parameters scale-wise) usually has higher accu- racy gain than upon the larger one, this indicates that MCLF is more efficient to dig the potential of smaller models and has the potential to make up for the inefficient models' capacity. Fine-grained CL., most of the time, performs better than those that only employ global CL under smaller backbones.
3.4 Comparison with Baselines
To validate the superiority of MCLF, in this sec- tion, we implement various previous methods as baselines for comparison:
RoBERTa (Liu al., 2019b): A RoBERTabes model directly fine-tuned on the SLU tasks using cross-entropy loss, and with ASR hy- pothesis as input.
PhonemeBERT (Sundararaman al., 2021):
RoBERTabase model first jointly trained on ASR hypothesis augmented with phoneme se- quence (generated from the ASR hypothesis with Phonemizer tool') and then fine-tuned for SLU tasks with cross-entropy loss.
htttps://gtt thub. ctt.uubb.co/boootttooo//oo phon/ phonemizer

---

SimCSE (Gao al., 2021): We first simply feed the same ASR hypothesis twice and get the two embeddings with different dropout noise (both/ - y for computing b in Equation (1), then it was fine-tuned on the same dataset with cross-entropy loss.
due to the ASR errors that make them indis- tinguishable, for example, we found two oracle utterances with different labels but they share same ASR assumption in the dataset.
4.2 Effect of Pooling Strategy in Fine-tuning
In Table 4, we evaluate the effect of different fea- ture pooling strategies in fine-tuning. Specifically, we first pre-train the BERT model with sev- eral contrastive learning objectives, then during fine-tuning, we try different output features pool- ing strategies for utterance representations, i.e., 1) [CLS]: global [CLS] representation; 2) Mean: averaging of all output token representations; 3) Mean-Max: concatenating the mean and maxi- mum of all token features (Zhang et al., 2018).
SpokenCSE (Chang and Chen, 2022): The closest method to our MCLF. A major dif- ference is that they only consider global con- trastive learning during pre-training while ig- noring finer-graine interactions (e.g., token- level ASR-Oracle alignment). And we intro- duce the fine-grained contrastive objective C, in Equation (4) to alleviate this problem.
As shown in Table 3, MCLF surpasses previ- ous methods on all four datasets. Among them, PhonemeBERT, SimCSE, and SpokenCSE focus on feature representation in pre-training. Phoneme- BERT needs extra phoneme sequence as input and is less effective on SLURP. SpokenCSE leverages global contrastive learning by taking oracle tran- scripts and the corresponding ASR hypotheses as input and gets rid of the dependence on phoneme information. By introducing an extra novel fine- grained contrastive objective together with the DA method, our MCLF improves considerably over the latest SpokenCSE. as well as all other baselines.
As can be seen, compared to the global [CLS] representation, method Mean can further improve performance. Especially, the model got the best results using Mean-Max pooling. Thus, we can conclude that, after adding fine-grained contrastive learning loss, layer Mean pooling strategy outper- forms (CLS). Intuitively, we attribute this to that the fine tne-ggained interaction provides fruitful infor- mation to the tokens. This proves that the whole sentence representation is more meaningful and dis- tinguishable with fine-grained contrastive learning.
4.3 Case Study Different Error Types
4 Discussions
We take several typical ASR errors from TREC6 for analysis in Table 5. The predictions are collected based on the RoBERTabase and MCLF predictions. For six labels of TREC6, the total mismatch pre- dictions (e.g., the golden label is HUM but pre- dicted as LOC) can be up to 30. Here, we list examples for analyzing the origin of the ASR er- rors between and after employing the MCLF, Take the first Oracle-ASR pair for example, when the key information Philippines is deleted, the baseline model tends to treat the whole utterance as DESC label, while the MCLF can capture fine-grained information like word in thus to predict true label as LOC. There are many such examples. However, we also find some interesting cases that are hard to solve even employing the MCLF, for example, in the third example, both the RoBERTa ba and MCLF mispredict the label as HUM, because the ASR hypothesis is what is nikki, for this example, the ASR error will cause both the local and global information of the utterance to change. In such a case, even MCLF will not have much effect.
4.1
UMAP Visualization
To further understand how the proposed method plays an important role, we extract the 768-d ut- terance embeddings from the TREC6 dataset and use the Uniform Manifold Approximation and Pro- jection (UMAP) (Mclnnes and Healy, 2018) algo- rithm to project these embeddings onto 2-d plane in Figure 4. First, from Figure 4a and Figure 4b, we can see that the semantics of ASR utterances are very scattered, farther than oracle utterances. Second, Figure 4a and 4c show the representations of ASR hypothesis on RoBERTapase and MCLF, respectively. We can see that most of the ASR ut- terances with same class are pulled together in both models. In RoBERTajases there are some ASR ut- terances that cannot be assigned to specific classes, while nearly all the utterances in MCLF model are pulled closer. Third, from Figure and Figure 4d, even in MCLF, there are still some ASR utterances that are indistinguishable, which we believe that

---

(a) RoBERTa ASR
(b) RoBERTa Oracle
(c) MCLF- ASR
(d) MCLF Oraace
Figure UMAP visualization of
Oracle utterances on RoBERTa and MCLF model on TREC6 dataset.
SLURP
TREC6
BERT base
[CLS]
Mean
Mean-Max
[CLS]
Mean.
Mean-Max
Global CL.
83.98
84.10
84.44
85.91
85.95
86.12
w/ Fine- Fine-grained CL
83.71
83,93
84.13
85.68
85.90
86.04
w/ Multi Multi-grained CL
84.39
84.71
84.85
85.80
86.22
86.40
Table 4: Performance SLURP and TREC6 datasets under different output feature pooling strategies.
Example
ASR Error Type
Golden Label
RoBERTa Prediction
MCLF Prediction
O: What hemisphere is the Philippines in
Deletion
LOC
DESC
LOC
What hemisphere is the in
O: Why is a ladybug helpful
Insertion
DESC
ENTY
DESC
A: Why is a lady bug helpful
What nicotine
Substitution
DESC
HUM
HUM
A: nikki
Table 5: ASR error analysis TREC6 test dataset. "O" and "A" denote the Oracle transcript and ASR hypothesis, respectively. Label in red denotes an error while the one in blue denotes the correct.
5
Related Work
correct candidates by leveraging the minimum edit distance (MED). Ruan al. (2020) introduced novel loss to minimize the distribution differences between correct and incorrect output texts. Spo- kenCSE (Chang and Chen, 2022) aimed to learn the invariant representations between clean text and erroncous hypothesis by utilizing a contrastive objective to adapt PLM to ASR results. Another stream considers making full use of the ASR output information (e.g., N-best hypothesis, phoneme quence). For example, Zhu et al. (2021) proposed two novel approaches (i.e., N-best Plain, N-best Alignment) that combine the N-best hypothesis of ASR as input for error correction. Wang et al. (2022b) proposed to trigger the fine-grained inter- actions between phoneme and word embeddings to derive phonetic-aware ASR-robust features. Cheng et al. (2023a) used manual transcripts improve ASR robustness by mutual learning. Xie et al. (2023) proposed incorporate syntax information into SLU system to strengthen ASR-robust ability by constraining attention scopes based on relation- ships within the syntactic structure. Cheng et al.
SLU an essential task for machines to infer cor- rect semantic meaning (e.g., intent detection, slot filling) from human speech (Huang et al., 2020; Zhou et al., 2020; Huang and Chen, 2020; Huang et al., 2022b; Chen et al., 2022b; Cheng et al., 2023b,d,e; Zhu et al., 2023a). Traditionally, it can be solved by fine-tuning NLP models (especially PLMs) with the ASR hypothesis as input (Wang al., 2022a). However, the ASR hypothesis of- ten contains errors caused by ASR systems. the SLU models are trained with clean texts, the per- formance will be poor when predicting using noisy text as input. Also, models trained with noisy texts may suffer performance degradation over datasets without ASR errors.
The core problem lies in improving the ASR robustness for SLU models, and several types of methods have been proposed to address this issue. One stream treats texts with or without noise two modalities. For example, Tan and Ling (2019) pro- posed to rebuild tuples between ASR results and

---

(2023c) utilized cross attention fuse the features between manual transcripts and ASR transcripts and contrastive attention to capture unique features of ASR transcripts.
experiments can be performed on single NVIDIA V100 GPU, it still incurs additional training time and cost. However, this does not change the range of devices on which the technique can be applied, since data augmentation does not change the size of the entire SLU model, and at the same time, the inference speed is basically unchanged.
Considering that the ASR output information may not be easily obtained due to the constraint of ASR systems, we adopt the first stream in this paper. In addition to following Chang and Chen (2022) that builds alignment between ASR hypotheses and the oracle transcripts through contrastive learning objective, we design the fine-grained contrastive learning which strengthens the model's robustness towards the local word error caused by phonetic confusion, and it serves as a complement to the global contrastive learning. The combination of both types of contrastive objectives is called multi- grained contrastive learning, aiming learn both utterance and token level alignment between ASR hypotheses and the oracle transcripts, which mines fruitful utterance-level and token-level information from downstream datasets.
Ethics Statement
Our goal in developing an ASR-robust SLU model is to enable practical intelligent assistant applica- tions more useful and appealing to human users through robust semantic recognition with ASR er- roneous outputs. For example, for people with different accents, different language habits, and whether they use slang, their inputs to the ASR sys- tem will be very different. So it is difficult for the ASR model to fully consider all of these situations and correctly identify all spoken errors. Therefore, the design of a robust spoken language understand- ing system for ASR is very important. And our proposed approach based on multi-granularity con- trastive learning effectively adapts the SLU model to the error of the ASR system. Overall, ASR error a crucial problem in application scenarios such as intelligent speech assistant, and having an ASR- robust SLU model that can filter such errors would have many positive applications.
6 Conclusion
In this paper, we propose MCLF, an ASR-robust SLU model towards multi-grained Oracle-ASR contrastive pre-training. Besides, several data aug- mentation techniques are used to facilitate the con- trastive pre-training stage. To our best knowledge, this is the first attempt improve the ASR robust- ness of SLU models by simultaneously exploiting both global and fine-grained features alignment between ASR hypothesis and oracle transcripts. Empirical results on four benchmark datasets show the effectiveness of the proposed MCLF and the superiority compared with several baseline works.
References
A. Baevski, Y. Zhou, A. Mohamed, and M. Auli. 2020. wav2vee 2.0: A framework for self supervisd leam- ing of speech representations. In Proc. of NeurIPS.
E. Bastianelli, A. Vanzo, P. Swictojanski, and V. Rieser. 2020. SLURP: A spoken language understanding resource package. In Proc. of EMNLP.
Limitations
As with any research, MCLF is not without lim- itations. We split the discussion into limitations that are inherent to our method, and limitations of our present study, the latter can be overcome by extensions of our work. Previous studies (e.g., SpokenCSE (Chang and Chen, 2022)) conducted contrastive learning with ASR and oracle utterance pairs using the Transformer- based model. Our em- pirical results can be seen further mining the potential of contrastive learning and illustrating the generalization of our method under multiple baseline backbones. Another technical limitation is that our method requires the cost of additional training data, and while we have shown that our
Y. Chang and Y. Chen. 2022. Contrastive learning for improving ASR robustness in spoken language derstanding. In Proc. of INTERSPEECH.
D. Chen, Z. Huang, X. Wu, S. Ge, and Zou. 2022a. Towards joint intent detection and slot filling via higher-order attention. In Proc. of ICAII.
D. Chen, Z. Huang, and Y. Zou. 2022b. Leveraging bilinear attention to improve spoken language under- standing. In Proc. of ICASSP.
D. Chen, C. Tao, L. Hou, L. Shang, X. Jiang, and Q. Liu. 2022c. Litevl: Efficient video-language learning with enhanced spatial-temporal modeling. In Proc. of EMNLP.

---

T. Chen, S. Komblithh M. Norouzi, and G. Hinton. 2020. A simple framework for contrastive learning visual representations. In Proc. of ICMI.
S. Gururangan, Marasovic, S, Swayamdipta, K. Lo, 1. Beltagy. D. Downey, and N. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proc. of ACL.
Cheng, B. Cao. Q. Ye, Z. Zhu, H. Li, and Y. Zou. 2023a. MI-Imel: Mutual learning and large-margin contrastive learning for improving asr robustness in spoken language understanding. In Proc. of ACL Findings.
C. Hemphill, J. Godfrey, and G. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley.
X. Cheng, W. Xu, Z. Zhu, H. Li, and Y. Zou, 2023b. Towards spoken language understanding via multi- level multi-grained contrastive learning. In Proc. of CIKM.
L. Hou, Z. Huang, L. Shang. X. Jiang, X. Chen, and Q. Liu. 2020. Dynabert: Dynamic BERT with adap- tive width and depth. In Proc. of NeurIPS.
C. Huang and Y. Chen. 2020. Learning asr-robust con- textualized embeddings for spoken language under- standing. In Proc. of ICASSP.
Cheng, Z. Yao, Z. Zhu, Li, H. Li, and Y. Zou. 2023c. C2A-SLU: Cross and Contrastive Attention for Improving ASR Robustness in Spoken Language Understanding. Proc. of INTERPPECCH.
W. Huang, Z. Zhang. Y. Yeung, X. Jiang, and Q. Liu. 2022a. SPIRAL: self-supervised perturbation- invariant representation learning for speech pre- training. In Proc. ICLR.
X. Cheng, 7. Zhu, B. Cao, Q. Ye, and Y. Zou. 2023d. Mrrl Modifying the reference via reinforcement learning for non-au autorrrrssssvvveeeeeee joint multiple intent detection and slot filling. In Proc. of EMNLP Find- ings.
Z. Huang, L. Hou, L. Shang, X. Jiang, X. Chen, and Q. Liu. 2021a. Ghostbert: Generate more features with cheap operations for BERT. In Proc. of ACI.
Cheng. Zhu, Xu, Li, Li, and Y. Zou. 2023e. Accelerating multiple intent detection and slot filling via targeted knowledge distillation. In Proc. of EMNLP Findings.
Z. Huang, F. Liu, X. Wu, S. Ge, H. Wang. W. Fan, and Y. Zou. 2021b. Andio-oriented multimodal machine comprehension via dynamic inter- and intra-modality attention. In Proc. of AAAT.
K. Clark, M. Luong, Q. Le. and C. Manning. 2020. ELECTRA: pre-training text encoders discrimina- tors rather than generators. In Proc. of ICLR.
Z. Huang. F. Liu, P. Zhou, and Y. Zou. 2021c. Sentiment injected iteratively co-interactive network for spoken language understanding, In Proc. of ICASSP.
J. Devlin, M. Chang, K. Lee. and K. Toutanova. 2019. BERT: pre-training of deep bidirectional transform- ers for language understanding. In Proc. of NAACL.
7. Huang, F. Liu, and Y. Zou. 2020. Federated learn- ing for spoken language understanding. In Proc. of COLING.
L. Dong, S. Xu, and B. Xu. 2018. Speech trancc-tranformmer no-recurrence sequence-to-sqquence model for speech recognition. In Proc. of ICASSP.
7. Huang, M. Rao, A. Raju, 7. Zhang, B. Bui, and C. Lee. 2022b. MTL-SLT: multi-task learning for spoken language tasks. In Proc. of ConvAI@ACL.
S. Dutta, S. Jain, A. Maheshwari, G. Ramakrish- nan, and P. Jyothi. 2022. Error correction in ASR using sequence-to-sqqueee models. CoRR, abs/2202.01157.
X. Jiao, Y. Yin, L. Shang. X. Jiang. X. Chen, L. Li. F. Wang, and Q. Liu. 2020. Tinybert: Distilling BERT for natural language understanding. In Proc. of EMNLP.
Fang, S. Filice, N. Limsopatham, and O. Rokhlenko. 2020a. Using phoneme representations to build pre- dictive models robust to ASR errors. In Proc. of SIGIR.
M. Kim, G. Kim, S. Lee, and J. Ha. 2021a. St-bert: Cross-modal language model pre-training for end- to-end spoken language understanding. In Proc. of ICASSP.
A. Fang, S. Filice N. Limsopatham, and O. Rokhlenko. 2020b. Using phoneme representations to build pre- dictive models robust to ASR errors. In Proc. of SIGIR.
T. Kim, K. Yoo, and S. Lee. 2021b. Self-guided con- trastive learning for BERT sentence representations. In Proc. ACL.
T. Gao, X. Yao, and D. Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. Proc. of EMNLP.
D. Le, M. Jain, G. Keren. S. Kim, Y. Shi, J. Mahadeokar, Fuegen, O. Kalinli, Y. Saraf, and M. Seltzer. 2021. Contextualized streaming end-to-end speech recognition with trie- based deep biasing and shallow fusion. In Proc. of INTERSPEECH.
J. Chan, Y. Shangguan,
B. Gunel, J. Du, A. Conneau, and V. Stoyanov. 2021. Supervised contrastive learning for pre-traines lan- guage model fine-tuning. In Proc. ICLR.

---

J. Li. Selvaraju, A. Gotmare, S. Joty, Xiong, and Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. In Proc. of NeurIPS.
Z. Yang, Dai, Yang, Carbonell, Salakhutdinov, and Q. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Proc. of NeurIPS.
M. Li, W. Ruan, X. Liu, L. Soldaini, W. Hamza, and C. Su. 2020. Improving spoken language understand- ing by exploiting ASR N-best hypotheses. CoRR. abs/2001 .05284..
L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu. 2022. FILIP: fine-grained interactive language-image pre-training. In Proc. of ICLR.
X. Li and D. Roth. 2002. Learning question classifiers. In Proc. of COLING.
S. Yun, J. Park, K. Lee, and J. Shin. 2020. Regularizing class-wise predictions via self-knowledge distillation. In Proc. of CVPR.
F. Liu, Y. Liu, X. Ren, X. He, and X. Sun. 2019a. Aligning visual regions and textual concepts for semantic-grounded image representations. In Proc. of NewrIPS.
M. Zhang, Y. Wu, W. Li, and W. Li. 2018. Learning universal sentence representations with mean-max attention autoencoder. In Proc. of FMNLP.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, Zetllemoerr, and V. Stoyanov. 2019b. Roherta: A robustly optimized hert pretrain- ing approach. CoRR, (bs/1907.11692
P. Zhou, D. Chong, H. Wang, and Q. Zeng. 2022. Cal- ibrate and refine! A novel and agile framework for ASR error robust intent detection. In Proc. of INTER- SPEECH.
P. Zhou, Z. Huang, F. Liu, and Y. Zou. 2020. PIN: A novel parallel interactive network for spoken lan- guage understanding. In Proc. of ICPR.
McInnes and J. Healy. 2018. UMAP: uniform man- ifold approximation and projection for dimension reduction. CoRR, abs/1802.03426.
L. Zhu, W. Liu, L. Liu, and E. Lin. 2021. Improving ASR error correction using N-Best hypotheses. In Proc. of ASRU.
Radford, J. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and Sutskever. 2021. Learning transfer- able visual models from natural language supervision. In Proc. of ICML.
Zhu. X. Cheng, D. Chen, Z. Huang, and Y. Zou, 2023a. Mix before align: towards zero-shot cross- lingual sentiment analysis via soft-mix and multi- view learning. Proc. of INTERSPEECH.
W. Ruan, Y. Nechaev, L. Chen, C. Su, and I. Kiss. 2020. Towards an ASR error robust spoken language under- standing system. In Proc. of INTERSPEECH.
Z. Zhu, X. Cheng, Z. Huang, D. Chen, and Y. Zou. 2023b. Enhancing code-switching for cross-lingual slu: A unified view of semantic and grammatical coherence. In Proc. EMNLP.
M. Sundararaman, A. Kumar, and J. Vepa. 2021. Phone- mebert: Joint language modelling of phoneme se- quence and ASR transcript. In Proc. of INTER- SPEECH.
Zhu, X. Cheng, Z. Huang, D. Chen, and Y. Zou. 2023c. Towards unified spoken language understand- ing decoding via label-aware compact linguistics rep- resentations. Proc. of ACL Findings.
Tan and Z. Ling. 2019. Multi-classification model for spoken language understanding. In International Conference on Multimodal Interaction.
G. Tur and R. De Mori. 2011. Spoken language under- standing: Systems for extracting semantic informa- tion from speech. John Wiley & Sons.
C. Wang, S. Dai, Y. Wang, F. Yang, M. Qiu, K. Chen, W. Zhou, and J. Huang, 2022a. Arobert: An ASR ro- bust pre-trained language model for spoken language understanding. IEE ACM Trans. Audio Speech Lang. Process.
Z. Wang, Y. Le, Y. Zhu, Zhao, M. Feng. M. Chen, and X. He. 2022b. Building robust spoken language understanding by cross attention between phoneme sequence and ASR hypothesis. In Proc. of ICASSP.
Y. Xie, 7. 7hu, X. Cheng, 7. Huang, and D. Chen. 2023. Syntax matters: Towards spoken language understanding via syntax-awar aware attention. In Proc. of EMNLP Findings.

---

A Experimental Details
Algorithm 2: LCS-based Sorting Algo-
rithm for Utterance X and Y
The detailed hyperparameters for training MCLF are shown in Table 6.
Input: X= .m >
Y= Y1... >
Output: c/m, m]: the LCS and 1: m <- [engthh[X]
Config
Pre-training
Fine-tuning
+- length Y|
2: for i 4-1 to m do
Batchsize
[32, 128)
(32, 64, 128)
3: c(t.0) <
Learning rate (Ir)
2e-5
2e-5
4: end for
Lr shedule
linear decay
linear decay
5: for j <-1 to do
Gradient clipping
1
6: c[0.j] +
Dropout rale
0.1
7: end for
Attention dropout
0.1
0.1
8: for i 4-1
to do for j 4-1 to do
ML.M ratio
15%
9:
Af:2m Am
0.1,
10:
i == == then
Aass An
1.0, 10
11:
c 1,j-,1 1]
To
0.2
0,2
12:
else
13:
c(t.j]] max(cli.) 1]. 1.5))
7d
14:
end if
Data Augmentation
y 20000 Steps
15: end for
Training Steps/Epochs
10 Epochs
16:
end for
17:
return c|m,
Table 6: Experimental setup for two stages in training four backbone models on four benchmark datasets.
We can see that, 1) different datasets do not share a uniform best batch size, while having different opti- mal batch sizes for optimal performance; 2) smaller dataset has larger results variance than larger one (e.g., TREC to SLURP), showing that MCLF gets consistent performance when the training data is larger; 3) for ATIS, large fine-tune batch size get better results, for SLURP, large pre-trained batch size get better results.
B Data Augmentation Details
LCS Sorting Algorihtm
We show the Longest Common Subsequence (LCS) sorting details Al- gorithm 2.
To further illustrate the effect of our data augmentation approach, in this section, we show the ablation results of MCLF in Table 7 that remove the DA part, while keeping other training details unchanged. We find that DA is important as part of the training step for multi-grained con- trastive learning objectives, and can help PLMs to learn more representative features.
Ablation Study
D Implementation Details
We outline the key components of the fine fine-grained and global contrastive objectives in PyTorch. Please refer to Listing for details.
Ablation Settings
ATIS (AAcc)
RoBERTa
G-CT.: Insertion-DA
+0.2
G-CL: Deletion-DA DA
+0.1
G-CL: Substitution- DA
+0.2
FG-CL: Insertion-DA
+0.2
FG-CL: Deletion-DA
+0.3
FG-CL: Substitution-DA
+0.3
MCLF
+0.8
Table 7: Ablation study of DA on the MCLE,
C Effect of Training Batch Size for MCLF with Different Backbones
In this section, we analyze the effect of batch size in pre-train and fine-tune stages. We show the re- sults on MCLF of different batch size of Table

---

PLM
batch size
FT batch
SLURP
ATIS
TREC6
TREC50
RoBERTabuse
32
32
85.14
94.3210.36
86.40 0.83
77.56
32
64
85,39
94.9511..00
86.76_o.0
17.88
32
128
85.30=0.14
95.05=0.25
86,40_0.38
77. 77.64
64
32
85.33
94. 12..38
86.76_0.61
77.20
64
85.27:10.11
94.93 94.93=0.41
97.00_0.52
78. 12.60
64
128
85.08
95.07 #0.46
86.32 0.39
77.80 77.80
128
32
85.24
94.67 =0.94
77.92 06.52_0073
128
64
85.31 10.21
95.20 10.32
86.72
0.63
78.84
128
128
85.01
95.22 10.55
86.20 0.63
78.28
BERT
32
32
84.82220116
94,82
86.16_0.82
78.32
bace
32
84.54=0.18
94.75=0.35
85.88=0.11
78. 78.20=vv61
32
128
84.11 10.20
94.95 1.0.20
85.96 0.71
78.04....8
64
32
84.82=00.00
94.77
85. $5.48_1.14
78,92
64
84.67
94. 10.46
85.72 0.48
78.56
64
128
84.20.1..00
94.95 10.33
86.32_ 0.33
78.32
128
84.85=0.31
24.58=0.42
85.92-0.72
78. 78.7660056
128
64
84.8 84.81
94. 10.0088
86.40 0.85
18.84
128
84.11 111.15
20.39
0.93
18.92
ELECTRAamaa
32
32
79,79
90.80=0.26
84.24
71.72=00.94
32
64
81.48
94.55
84.44 0.53
72.76.1111
32
128
81.6910.33
94.8810.11
85,56 1.09
73.52
64
80.24-40.27
94. 44.70
84.96_1.18
72.08+0.41
64
64
81.81
94.82
84.80
74,24
128
82.43
05.05=0048
85.52_0.89
74.52.20.00
128
32
80.81 10.19
94.63
83.36 0.97
71. 71.56
128
64
89.19
94.45 10.19
84.64_0.86
73.56
128
128
82.76
95.00 10.24
84. 0.55
74. 74.12.0.69
TinyBERT4
32
79.80
94.75 10.38
83.36 0.66
70.76
32
64
81.55
94.35
83,48_0.70
71.40
32
128
81.65
95. 95.12=0.96
83.00
72.08=1.13
64
32
80.15
94.55
83.68 1.00
71.3610.75
64
81.77=0.24
94.95
84.04
72. 72.16
64
128
81.71 10.07
95.35 10.24
83.32 0.57
72.60 1.085
128
32
80.42_...19
94.7510.38
83.68 0..7
70.12 12.332
128
81.83+0.21
95.30-0.47
83,84_0.45
70. 70.84
128
128
81.83
95.22.1019
83.32_0.30
71.20
Table 8: Results of four backbones on benchmark datasets under different pre-train and fine-tune batch sizes.

---

import terch import terch.nc Import torch.nn. functional as Import math
toker_xiso_simileriin (repl, rep2, chunk_sizo=1024) batch_sizel, n_tokenl, feat_dim repl.shape
batch_size2, n_token2, rep2. shape
num_folds math. (batch size? / chunk_size)
output []
for i in range (num folds):
rep2_sse
rep2 [i chank [runk_size:( 1) chunk_size]
out r reshape (-1, feat_aim) 8 reshapppo (-1, reat_dim) T
out out_i.reshin
beaatch_sizell n_tokenl, batch_size2, r_taken2)
out_: = out_i.man (3) [C] (1) output.appente append
return torch.ca cat (output, dim=1)
def get fitt_2b_sime (b_sim oracle):
m1 token_ wwwwwwwsseiilleeee siniassiyallllryyaassss asr)
m3 token_wise__smmilartty (asr, oracle]
m2
(oracle, asr]
m4
(oracle,, oracle)
matrix_up = torch.cat ([ml, m21, dim=l)
matrix_down = torch.cat (Lm3, m4l, aim=l)
return torch. [[mttrix_us matrix_down), dim=0)
class ContrastiveLoss (nn. Module):
temperature 0.05
det (sll, sim_mat, batch_size, 2_i):
numerator torch.exp (sin self temperature)
identity_oxcope (111), 0.0) (zt.zii. denominator torch. www.ddddteeeeeeeeeeeeeeee temperature))
torch.ones batch_size, )) sccattor_ torch. tensor
torch. (sim mat :1 self.
loss -torch. loacch.oogaee (nuerrator denominator) return loss 1.ss__)
def forward(self, 2_1, 2_1, gramularity):
z_j, z_j: glopal_shape- [b, fine_shapo= [b, nb,22/ batch_size s_i.shape grarularity rep totch.cat (az (fz_i, z_jl, dim=0)
"globai":
sim mat F. wisesiissssmmmlrtttiaeeaaaaalele wre.utuueep.u elif granularity Eeg.sseeell.llltrrr.......eeeee
(1), rep. (0n) dim=2)
"fine":
2 - 2_1 / a. norm m--1-1 keepdim-True)
2 =-= = z_ / 2_j. norm (dim=-1, koopdin=-rue)
8 sim_mat mat get 2b____ sim 1 2_1)
loss = 0.0 for k in zange (0, batch_size)
loss = loss + self. (k, x + batch_size, simmaa: batch_size, 2_
loss - loss + bejf + batch_size, size, k, sim mat, batch_si2e i_
return 1.0 / (2 batch size) loss
Listing I: Contrastive Loss Implementation


Reviews:
[{"review_id": "sAKXO7Wzhp", "reviewer": "Reviewer_7euW", "summary": "", "questions": "- Q1: What data is used during pretraining? How is augmented data used in constructing positive and negative pairs during pretraining?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed feedback; addressing these suggestions has improved the quality of the paper. Below are some explanations about the remarks of your concerns.\n\n**On the concern about novelty and missing references.**\n\n- We appreciate your feedback, particularly the pointing out of the Multi-level contrastive learning-based SLU systems referenced in [1][2]. Their input will be incorporated into the next version of our study and we will create a clear distinction between our approach and theirs. Additionally, we will compare our work with other frameworks that incorporate multi-grained information.\n\n- It's worth noting that our method differs significantly from the ones mentioned in [1][2]. In [1], contrastive learning (CL) is employed for Cross-lingual SLU tasks, while our goal is to alleviate the ASR-Error problem. Furthermore, in their work, positive samples are drawn from different languages, whereas we generate our positive samples by imitating ASR error.\n\n- In [2], fine-grained contrastive learning is used between different modalities, namely speech and text. However, in our training optimization target, only a single text modality is used.\nMoreover, the concept of multi-grained or fine-grained can also be seen in other areas outside of SLU, such as the field of vision-language pre-train (FILIP, [3]) and the field of passage search (ColBERT, [4]).\n\n- We believe these significant distinctions and additional insights strengthen the validity of our approach and broaden its implications.\n\n**On the concern about marginal contribution.**\n\n- Thank you for your valuable feedback on our paper. We understand your concerns regarding the use of common techniques. However, we would to clarify that the originality of our work is not focused on inventing complex new techniques, but rather on creatively employing these known strategies within a completely novel framework, namely the Multi-Grained Contrastive Learning Framework (MCLF). This uniquely aids in enhancing spoken language understanding (SLU) systems that are prone to ASR errors.\n- The results shown in Table 1 clearly highlight significant system performance improvements following the application of MCLF within the aforementioned paradigm. We believe that these findings represent a meaningful and substantial contribution to the field.\n- Although the utilization of underlying basic data augmentation and fine-tuning techniques is not our first invention, we are the first to successfully apply these techniques in the multi-granularity ASR error SLU scenario. Our paper illustrates their effectiveness in different backbones, illustrating novel applications of these widely used processes.\n- Consequently, we respectfully disagree with the characterization of our work's contribution as being marginal. Instead, we assert that our work offers significant utility to the field.\n\n**On the concern about DA and Loss Ablation.**\n- The ablation of data augmentation can be seen in Table 7 in Appendix C.\n- The ablation of loss ablation can be seen in Table 1 in Section 3.3. We conduct experiments on four backbones (i.e., RoBERTa, BERT, ELECTRA, and TinyBERT). For each backbone, four different loss combinations are experimented for getting insights. Row 1 means the original PLM without any contrastive learning objective; Row 2 means the PLM is trained along with global contrastive learning loss; Row 3 means the PLM is trained along with the fine-grained contrastive learning loss; Row 4 means the PLM is trained with both global contrastive learning loss and fine-grained contrastive learning loss.\n\n**On the concern about implementation details.**\n\n**Q:** What data is used during pretraining?  \n- During pre-training, we use the same task-specific data but the learning target is MLM. Details can be seen in Table 5 and Table 6 in Appendix A & Appendix B. \n\n**Q:** How is augmented data used in constructing positive and negative pairs during pretraining?  \n- A: We apply contrastive learning over oracle and asr sentences by optimizing InfoNCE loss during pre-training, details can be seen in Line 142 - Line 156. The positive data are generated by data augmentation, the negative data are sampled from the in-batch data. For more details on InfoNEC loss, please refer to [5][6]. Thanks for the careful review, for more details, our implemented code is appended along with the submitted manuscript with detailed comments.\n\n**Typos**\n\n- Thank you for your careful reading! We will fix them in the revised version.\n\n\n**References**  \n\n[1] Label-aware Multi-level Contrastive Learning for Cross-lingual Spoken Language Understanding. Liang et al., 2022.  \n[2] Cross-modal Transfer Learning via Multi-grained Alignment for End-to-End Spoken Language Understanding. Zhu et al., 2022.  \n[3] FILIP: Fine-grained Interactive Language-Image Pre-Training. Hou et al., 2021.  \n[4] Efficient and effective passage search via contextualized late interaction over bert. Omar et al., 2020.  \n[5] https://paperswithcode.com/method/infonce.  \n[6] Representation Learning with Contrastive Predictive Coding. Oord et al., 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- S1: Fine-grained pretraining objective along with DA strategies for more ASR-robust system. \n- S2: The approach is proven to be consistently effective across different model backbones, and it outperforms all baselines of previous works.\n- S3: Detailed analyses on the model robustness in visualization, pooling strategies, as well as error case study, demonstrates the effectiveness of the method.", "reasons_to_reject": "- W1: The idea of multi-grained alignment for SLU models is not novel; There have been similar SLU works in applying multi-level contrastive learning methods in SLU systems (see missing reference).\n- W2: The contribution of the paper is a little marginal, where most of the techniques are already commonly used in SLU systems: the data augmentation method is straightforward, and the the finetuning losses are simple combinations of existing classification losses.  \n- W3: More thorough analysis on the contribution of different components within the model such as data augmentation and loss combination of the method is lacking.", "questions_to_authors": "- Q1: What data is used during pretraining? How is augmented data used in constructing positive and negative pairs during pretraining?", "paper_topic_and_main_contributions": "This paper address the problem of automatic speech recognition (ASR) errors for cascaded spoken language understanding (SLU) models that takes ASR transcripts as input. The authors propose MCLF, a two-stage multi-grained contrastive learning framework to enhance the robustness of SLU systems. At the first stage, the approach adopts multi-grained contrastive learning alignment in both sequence and token level as pretraining objective. Different data augmentation (DA) strategies are also introduced to generate more transcripts with a certain level of word error rate (WER). During finetuning, a joint training objective is introduced with various classification losses are proposed.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Multi-grained alignment for SLU systems:\n- Label-aware Multi-level Contrastive Learning for Cross-lingual Spoken Language Understanding; Shining Liang, Linjun Shou, Jian Pei, Ming Gong, W. Zuo, X. Zuo, and Daxin Jiang\n- Cross-modal Transfer Learning via Multi-grained Alignment for End-to-End Spoken Language Understanding; Y Zhu, Z Wang, H Liu, P Wang, M Feng, M Chen, X He", "Typos_Grammar_Style_and_Presentation_Improvements": "- P1[Figure 1]: The legend could be moved to somewhere else not covering the plot;\n- P2: It would be better to have figures in pdf format rather than image format;\n- P3[line 309]: they have -> and they have\n- P4[381]: different -> the same?\n- P5[448]: interactive -> interaction"}}, {"review_id": "CB6uFEBzwf", "reviewer": "Reviewer_dEWS", "summary": "", "questions": "Mentioned previously, how the author derived to their proposed data augmentation? How does it affect the model if they are random and executed casually? Would curriculum learning help with increasing errors in the perturbation and what are the frequency of such augmentation used?  ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude for the insightful reviews you've provided. We value your input and would like to address your concerns as outlined below.\n\n**On data augmentation.**\n\n- Thank you for your valuable questions. First, data augmentation is one of the important techniques for MCLF. \nGenerally, in some scenes, the performance of the model can be improved by using the method of contrastive learning. However, contrastive learning often requires more data to exert its benefits (e.g., [1][2]) Based on this consideration, we designed DA method starting from the construction form of WER. However, we want to clarify that the DA method we use is definitely not optimal. We can replace it with many existing excellent DA methods. We will discuss this part in the next version, thank you for your comments!\n\n**Questions**\n\n**Q:** How the authors derived to the proposed DA method?\n  - We start with the construction form of the ASR error. As can be seen in Line 210, the Word Error Rate to measure the percentage of incorrectly transcribed words consists of Inserted words, deleted words, and substitution words. Thus we design the Insertion DA, deletion DA, and substitution DA. The experimental results show the effectiveness of such DA methods.\n\n**Q:** How does DA affect the model if they are random and executed casually?\n- We provided DA ablation in Table 7 in the Appendix, where we show that all of the Insertion, Deletion, and Substitution DA methods are useful for the training of MCLF. Following your recommendation, we try one more random DA as a comparison under fine-grained contrastive learning setting. Note that except for the augmented data, we keep other implementation settings exactly the same for fair comparison. We run five times by different random seeds. Results show that 1 time we achieved a 0.2 improvement in ATIS (this is nearly the same as Insertion DA), and 1 time did not have any impact on performance, but 3 times, the loss convergence during pre-trained was significantly slower, and final finetuned models' results are 1% less than the Insertion DA method on average. We will discuss this issue in detail in the next version, and sincerely appreciate your suggestions.\n\n**Q:** Would curriculum learning help with increasing errors in the perturbation and what are the frequency of such augmentation used?\n- Curriculum learning could potentially help with increasing errors in the perturbation, as it would allow the model to first learn from less challenging examples and gradually progress towards harder, more perturbed examples. This strategy can help the model develop a more robust understanding of the data and prevent it from overfitting on any specific set of perturbations. We will discuss this issue in detail in the next version, and sincerely appreciate your suggestions.\n\n**References**\n\n[1] FILIP: Fine-grained Interactive Language-Image Pre-Training. Hou et al., 2021.  \n[2] Efficient and effective passage search via contextualized late interaction over bert. Omar et al., 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work is inspired by FILIP to use the fine-grained contrastive learning to improve the representations of the downstream model. Besides, it is interesting and shown to be effective based on how the author uses such contrastive learning loss on corrupted text output. It is good that such technique appears to be easy to reproduce (but requires the understanding of the augmentation techniques used by the author).\n\nEven though the pre-training of the proposed work requires additional computational cost, it is a plus that the final inference remains unchanged to the base model. As such, the model does have strong applicative uses.\n \nThe paper is comprehensive and the ablation work is sufficient with detailed analysis shown in the results and appendix. The visual plots also help to understand the improved representation of such technique.", "reasons_to_reject": "The effectiveness of the proposed work seems to tie partly to the data augmentation technique. Noted on the sensitivity of the data augmentation method in appendix, I am just curious if such augmentation requires careful and thoughtful engineering? Does substitution work with random replacement? ", "questions_to_authors": "Mentioned previously, how the author derived to their proposed data augmentation? How does it affect the model if they are random and executed casually? Would curriculum learning help with increasing errors in the perturbation and what are the frequency of such augmentation used?  ", "paper_topic_and_main_contributions": "This paper addresses the problem of erroneous ASR output that deteriorate the performance of SLU task. While it is common to encounter such problem with noisy prediction output from ASR model, especially with noise corrupted speech, it is also difficult to work on the robustness of the downstream SLU task. Interestingly, this work proposes a two-stage multi-grained contrastive learning framework to handle the noisy ASR output and has achieved improved results.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bmkA9dkOqF", "reviewer": "Reviewer_u9Ji", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our heartfelt appreciation for the thorough reviews you have provided. We value your insights and would like to offer detailed responses to your concerns.\n\n\n**On the concern about error analysis**\n\n- Thank you for your insightful feedback. We agree that providing an error analysis could offer a better understanding of the performance improvements emanating from the use of fine-grained contrastive learning. We value your comments and to address this, we have conducted three different ASR error analysis on the TREC6 test set. The table below presents a snapshot of this analysis.\n\n|Examples|ASR Error Type|Golden Label|RoBERTa Prediction |Global Contrastive Learning Prediction|MCLF Prediction|\n|:---------------------------|:---------------------|:--------------------|:---------------------|:-------------------|:---------------------|\n|O: \"What hemisphere is the Philippines in\" A: \"What hemisphere is the in\"|Deletion|LOC|DESC|LOC|LOC|\n|O: \"Why is a ladybug helpful Insertion\" A: \"Why is a lady bug helpful|Insertion| DESC|ENTY|ENTY|DESC|\n|O: \"What is nicotine\" A: \"What is nikki\"|Substitution|DESC|HUM|HUM|HUM|\n\n\n- Take the first example to elaborate, the Oracle Transcript (O) \"What hemisphere is the Philippines in?\" becomes \"What hemisphere is the in?\" through ASR, due to the omission of \"Philippines\". This omission makes comprehension of the sentence meaning challenging for the traditional SLU backbone model (e.g., RoBERTa), leading to incorrect judgments.\n\n- We also discovered that the sentence vector distances in the mentioned example sentences are as such: Ex3 > Ex2 > Ex1 (measured using BERT-base-uncase [CLS] embedding). We noted that when using the original RoBERTa model, all three cases result in incorrect prediction; when using Global Contrastive Learning (CL), Ex1 is predicted correctly; and when applying MCLF, both Ex1 and Ex2 are accurately predicted. These demonstrations highlight the efficacy of the proposed MCLF model. In addition, it suggests that the farther the sentence embedding with ASR error is from the oracle sentence embedding, the less likely it is to be correctly predicted by CL-trained model, and the benefits of employing fine-grained CL are increasingly noticeable. We will include this analysis in the next version of the paper, along with a detailed discussion on the findings. We appreciate your suggestions and believe that such an analysis will indeed provide the readers with a richer perspective on the effectiveness of the proposed method.\n\n\n\n**On the concern about missing references**\n\n- We appreciate your insightful comments and the associated references. We agree that a discussion with end-to-end spoken language understanding (E2E SLU) systems would be valuable. Indeed, the E2E SLU systems can prevent cascading errors from the ASR system, potentially leading to better performance.\n\n- First, we would like to emphasize that our current focus primarily lies in the examination of performance robustness across different ASR environments. Our proposed method provides a greater degree of flexibility by decoupling the SLU components from the ASR modules, thus facilitating a more comprehensive analysis of ASR errors. Further, our methodology can be seamlessly integrated into a broad spectrum of existing ASR models and interfaces with less costs involved.\n\n- Upon close examination of the papers you recommended, we noted that the E2E SLU methods employed by [1] and [2] yielded impressive results on the FSC, SLURP, and SNIPS datasets. Though, it is worth highlighting that the fundamental differences between these approaches and ours. While E2E SLU designs aim to circumvent cascading errors from ASR systems, our strategy intentionally addresses these errors to enhance system robustness.\n\n- In conclusion, thanks for the value and significance of your suggestion. We will add discussion with references [1][2] in the next version and explore incorporating a broader range of comparisons and discussions related to E2E systems. Your constructive feedback has proved invaluable, and we extend our gratitude for drawing our attention to this important aspect.\n\n**References**\n\n[1] ESPNET-SLU: ADVANCING SPOKEN LANGUAGE UNDERSTANDING THROUGH ESPNET. Arora et al. In ICASSP 2022.  \n[2] Two-Pass Low Latency End-to-End Spoken Language Understanding. Arora et al. In INTERSPEECH 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Clear description of their methodology with mathematical formulations.\n2. Good performance improvements, particularly when the PLM is small like TinyBERT", "reasons_to_reject": "1. The paper lacks a comparison with a relevant line of research, namely E2E SLU systems (https://arxiv.org/abs/2111.14706, https://arxiv.org/abs/2008.06173), which avoid cascading errors from the ASR system. Even if there were no empirical comparisons, the authors should have provided some discussion on the advantages of their robustness approach over end-to-end approaches. Interestingly, E2E SLU approaches (https://arxiv.org/pdf/2111.14706.pdf) appear to perform better than their approach on the SLURP dataset, even without incorporating any pre-trained LM.\n2. The main contribution of the paper is Fine-grained Contrastive Learning, as I understand it. It would have been insightful to include an error analysis to better understand the sources of performance improvements resulting from using fine-grained contrastive learning.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes enhancing the robustness of SLU systems to ASR errors. The authors utilize a two-stage multi-grained contrastive learning framework to adapt a pre-trained language model to ASR hypotheses of SLU datasets. They find that their approach helps improve performance on four publicly available SLU datasets.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "References for E2E SLU models:\nhttps://arxiv.org/abs/2111.14706\nhttps://arxiv.org/abs/2008.06173\n\nReferences for E2E SLU models incorporating pre-trained LM:\nhttps://arxiv.org/abs/2207.06670\nhttps://arxiv.org/abs/2010.13826"}}]

Author rebuttal:

Meta-review: This paper introduces MCLF, a two-stage, multi-grained contrastive learning framework aimed at improving the robustness of SLU systems. In its initial stage, the framework employs multi-grained contrastive learning alignment at both the sequence and token levels as a pre-training objective. Various data augmentation (DA) strategies are utilized to produce additional transcripts with a specific word error rate (WER). In the finetun-ing phase, a joint training objective is applied, and multiple classification losses are proposed. MCLF showcases its efficacy across four public benchmarks.

**Soundness Scores**: (3, 4, 4)
All reviewers agree that the experiments are methodologically sound. The original manuscript includes comprehensive experiments on four public benchmarks as well as an ablation study. Although a detailed error analysis was absent in the initial submission, the authors have subsequently addressed this in their rebuttal. Overall, the primary claims of this work are well supported.

**Excitement Scores**: (3, 4, 3)
All reviewers agree that the novelty and contributions as incremental, primarily because the techniques detailed have already been extensively employed in SLU tasks. In their rebuttal, the authors seem to acknowledge that their proposed method's performance might be comparable to, or only slightly better than, random chance data augmentation.
Decision: Accept-Findings