MoE-SLU: Towards ASR-Robust Spoken Language Understanding via Mixture-of-Experts (2024)

Abstract: As a crucial task in the task-oriented dialogue systems, spoken language understanding (SLU) has garnered increasing attention. However, errors from automatic speech recognition (ASR) often hinder the performance of understanding. To tackle this problem, we propose MoE-SLU, an ASR-Robust SLU framework based on the mixture-of-experts technique. Specifically, we first introduce three strategies to generate additional transcripts from clean transcripts. Then, we employ the mixture-of-experts technique to weigh the representations of the generated transcripts, ASR transcripts, and the corresponding clean manual transcripts. Additionally, we also regularize the weighted average of predictions and the predictions of ASR transcripts by minimizing the Jensen-Shannon Divergence (JSD) between these two output distributions. Experiment results on three benchmark SLU datasets demonstrate that our MoE-SLU achieves stateof-the-art performance. Further model analysis also verifies the superiority of our method.

heading: 1 Introduction
text: Spoken Language Understanding (SLU) is a fundamental task in recent task-oriented dialogue systems, aimed at capturing the comprehensive semantics of human speech. It plays an important role in personal assistants like Amazon’s Alexa, Apple’s Siri, and Microsoft’s Cortana (Young et al., 2013; Cheng et al., 2023b; Zhuang et al., 2024). SLU focuses on two typical subtasks: intent detection and slot filling (Tur and De Mori, 2011; Cheng et al., 2024). Intent detection can be regarded as a semantic classification task (Xu et al., 2021), aiming to predict the user’s intent (Chen et al., 2022b; Zhou et al., 2022). Slot filling could be approached as a sequence labeling task, where the goal is to predict the slot for each token (Zhou et al., 2021; Zhu et al., 2024; Zhao et al., 2024; Song et al., 2024).
† Equal contribution. * Corresponding author.
In the realm of SLU, there are two common approaches: pipeline methods and end-to-end methods. Pipeline methods involve the cascaded combination of automatic speech recognition (ASR) and natural language understanding (NLU). The ASR system transcribes acoustic input into text, which is then fed into the NLU component to tackle the specific task. On the other hand, end-to-end SLU methods directly generate predicted results without explicit separation of ASR and NLU (Huang et al., 2022; Seo et al., 2022; Dong et al., 2023c).
When it comes to pipeline SLU methods, they provide the advantage of seamless integration of external datasets and the utilization of pre-trained language models. However, they are prone to error propagation, where mistakes from the ASR system could adversely affect the accuracy of subsequent NLU processing. As a result, enhancing the ASR robustness of the SLU model becomes crucial.
An effective approach to mitigate the detrimental impact of errors arising from ASR is to learn error-robust representations for SLU. D’Haro and Banchs (2016) proposes a phrase-based machine translation system trained with the words and phonetic encoding to automatically correct the ASR results. Mani et al. (2020) proposes a machine translation model that learns a mapping from the outof-domain ASR errors to in-domain terms found in corresponding reference files. Leng et al. (2021) utilizes edit distance-based alignments between the encoder and decoder to perform ASR error correction. Dutta et al. (2022) bootstraps the BART-based sequence-to-sequence model and leverages several phonetically grounded fine-tuning strategies to enhance the correction of errors in ASR predictions.
In this paper, we propose MoE-SLU, an ASRRobust SLU framework based on the mixture-ofexperts technique to make better use of the clean manual transcripts and the ASR transcripts. We propose three strategies to generate more transcripts from clean transcripts, which could simulate more
14868
kinds of errors in real scenarios. Then we utilize the mixture-of-experts method to weighted average the representations of generated transcripts, ASR transcripts, and the associated clean manual transcripts. By learning the weights of different transcripts in mixture-of-experts, the clean manual and ASR transcripts are handled differently. Both the weighted average of representations and the representations of ASR transcripts are used to calculate the cross entropy with the corresponding labels. In addition, we also regularize these two predictions via minimizing Jensen-Shannon Divergence (JSD) between the two output distributions. Experimental results demonstrate that our MoE-SLU outperforms previous ASR-Robust SLU models, and model analysis also verifies the advantages of our method. To sum up, the contributions of our work are three-fold:
• We propose an ASR-Robust SLU framework MoE-SLU, which uses the mixture-of-experts technique to make better use of the ASR transcripts and the clean manual transcripts.
• Experiments on three public SLU datasets display that MoE-SLU achieves new state-of-theart performance, surpassing previous works.
• Model analysis further verifies that MoE-SLU can indeed improve ASR robustness more effectively than previous works.

heading: 2 Related Work
text: ASR-Robust Spoken Language Understanding SLU aims to understand the user’s current goal via constructing semantic frames (Cheng et al., 2023c; Dong et al., 2022, 2023a). Since SLU usually faces the challenge of error propagation from the ASR system, there are increasing attempts being made to enhance ASR robustness in SLU. Various traditional methods have been proposed to solve SLU, including support vector machine (SVM) and recurrent neural network (RNN) (Haffner et al., 2003). Xu and Sarikaya (2013b) attempts to leverage loglinear models to achieve intent detection.
Recently, many classification approaches based on the neural network such as convolutional neural networks (CNN) (Xu and Sarikaya, 2013a; Ravuri and Stolcke, 2015) have been investigated. For instance, Xia et al. (2018) leverages a capsule-based neural network with self-attention for SLU. With the recent remarkable performance demonstrated by pre-trained models across different tasks (Zhu et al., 2023a,b; Feng et al., 2023; Wu et al., 2024; Huang et al., 2024b; Shen et al., 2024; Chen et al.,
2024), researchers began to explore the application of BERT-based pre-trained models (Devlin et al., 2019) in the field of SLU. Huang et al. (2022) applies LAS (Chan et al., 2016) and BART (Lewis et al., 2020) as the pre-trained models and proposes a multi-task learning framework termed MTL-SLT. In this work, we utilize RoBERTa (Liu et al., 2019) and Data2vec (Baevski et al., 2022) to acquire invariant representations between clean manual transcripts and erroneous ASR transcripts.
Mixture-of-Experts Mixture-of-Experts is proposed by Jacobs et al. (1991); Jordan and Jacobs (1994), which allows for the processing of different examples utilizing independent expert modules and has been adopted in different domains. Hochreiter and Schmidhuber (1997) utilizes mixture-ofexperts to construct the large-scale language models. With the development of attention (Cao et al., 2021; Zhu et al., 2022; Zhuang et al., 2022; Li et al., 2022a; Xin et al., 2022, 2023b; Xin and Zou, 2023; Yin et al., 2023; Xin et al., 2023a), researchers began to use mixture-of-experts to feed-forward networks to enhance the performance of Transformers. Shazeer et al. (2017) uses mixture-of-experts in sequence learning. He et al. (2018); Cho et al. (2019) uses mixture-of-experts in the generation tasks, and Peng et al. (2020) leverages mixture-of-experts to improve the performance of the translation models. In addition, Zhang and Feng (2021) proposes the mixture-of-experts wait-k policy to build a universal simultaneous machine translation method capable of delivering the high-quality translations with arbitrary latency. In our study, we apply mixtureof-experts technique to weighted average the representations of the generated transcripts, the ASR transcripts, and the associated clean manual transcripts, which allows the model to leverage the difference between the clean manual transcripts and the corresponding ASR transcripts more efficiently.

heading: 3 Method
text: In this section, we first propose three strategies to generate additional transcripts from the clean transcripts (§3.1). Then, we begin to introduce our proposed ASR-Robust SLU framework MoE-SLU, including a self-supervised contrastive learning module in pre-training (§3.2) and a mixture-of-experts module in fine-tuning (§3.3). Finally, we introduce the training objective during the pre-training stage and the fine-tuning stage (§3.4).
take out a beauty blender
take out a a beauty blender
Repetition
take out a beauty blender
take out a beauty blender
Deletion
Swap

heading: 3.1 Proposed Data Augmentation Strategies
text: To capture a wider range of the linguistic variations and noise patterns in the real-world environments, we propose three strategies to generate some additional transcripts from the clean transcripts, thereby increasing the diversity of training data. The proposed strategies include repetition, deletion, and swap. Specific schematic illustrations of each operation are depicted in Figure 1 for better clarity. Since the changes brought about by these strategies are minor, they usually do not alter the underlying semantics of the original ASR transcripts. Moreover, in general, there are certain keywords in the transcript which play a crucial role in determining the intent. With the utilization of these three strategies, the ability of the SLU model to capture these keywords can be further improved.
Repetition Strategy As illustrated in the upper part of Figure 1, the repetition strategy involves randomly duplicating certain words in original ASR transcripts. In real environments, it is common for certain words to be repeated in the original speech. The corresponding intent label remains unchanged and the slots corresponding to the repeated words are added to the original slot labels. By incorporating the strategy into the data generation process, we can generate more transcripts that closely resemble the real-life scenarios to enhance the ability.
Deletion Strategy As illustrated in the middle part of Figure 1, similar to the repetition strategy, we can also delete some words which are relatively unimportant to generate some new transcripts. To
achieve this, we utilize CoreNLP1(Manning et al., 2014) to extract non-nouns from the original transcripts. Subsequently, we randomly delete some of these non-nouns to generate new transcripts. The corresponding intent label also remains unchanged and the slots of the deleted words are deleted.
Swap Strategy As illustrated in the lower part of Figure 1, we also propose a swap strategy that randomly swaps two words in the original transcript to obtain the new transcript. The corresponding intent label remains unchanged and the slots of swapped words are swapped accordingly.

heading: 3.2 Self-supervised Contrastive Learning
text: Following previous works (Chang and Chen, 2022), we employ self-supervised contrastive learning during pre-training to develop sentence representations that are robust against to misrecognition and capable of handling the ASR errors. For a fair comparison, we utilize a pre-trained RoBERTa model (Liu et al., 2019) and Data2vec (Baevski et al., 2022). This continuous training process enables the SLU model to learn from the rich patterns and structures present in the input spoken language.
The mini-batch of input data is denoted as B = (xp, xq), where xp represents a clean manual transcript and xq represents its associated ASR transcript. We first apply the proposed three strategies to generate additional transcripts from xp. Then, we employ the pre-trained model and use the last layer representation of the special token [CLS] to obtain the corresponding representation h. By doing so, the model can capture the contextual information and encode it to meaningful representation.
We leverage the self-supervised contrastive loss Lsc (Gao et al., 2021) to adjust sentence representations. Lsc encourages similar representations for semantically related pairs of transcripts while pushing apart the representations of the unrelated pairs. This process could improve the ability of the model to capture underlying semantic information in the transcripts and promote the robustness against ASR errors (Wang and Isola, 2020):
Lsc = − ∑
(h,h+)∈P log
es(h,h +)/τsc
∑ h′ ̸=h e s(h,h′)/τsc
= −EP [ s(h, h+)/τsc ] + E [ log (∑
h′ ̸=h es(h,h
′)/τsc )]
(1)
1https://stanfordnlp.github.io/CoreNLP
𝑝!,#$%&
𝑝!,'$
𝑝!,'$%&
where P denotes the positive pairs, τsc denotes the temperature hyper-parameter to adjust the scale of the cosine similarity function s(·, ·). Positive pairs could be divided into the following three categories. The first category is ASR transcripts and associated clean manual transcripts, the second category is the ASR transcripts and the associated generated transcripts, and the third category is the clean manual transcripts and the associated generated transcripts. The negative pairs are all the remaining transcripts in the same batch, including ASR transcripts, clean manual transcripts, and the generated transcripts.

heading: 3.3 Mixture-of-Experts
text: As shown in Figure 2, to leverage the different characteristics of the clean manual transcripts and ASR transcripts more effectively, we design a mixtureof-experts module in fine-tuning. For a transcript xi = (xi1, x i 2, . . . , x i m), we concatenate these representations obtained by the pre-trained model and feed the concatenation to a multi-layer perceptron (MLP) to predict the confidence score βi:
βi = tanh([hi;hi1; . . . ;h i m]W i + bi) (2)
where Wi and bi are the parameters of MLP, hi denotes the representation of the special token [CLS], hi1, . . . , h i m denote the representations of the original tokens, [·] denotes the concatenation operation. For the input clean manual transcript x1 and its associated ASR transcript x2, we employ repetition, deletion, and swap strategies to obtain x3, x4, and x5, respectively. We consider them as five experts and then apply a softmax function to calculate the weight Gi of the i-th expert:
Gi = softmax(βi) (3)
Then, we utilize x1, x2, x3, x4, and x5 as inputs to the model and calculate the weighted average of
their predictions, denoted as ŷ:
ŷ = 5∑
i=1
Gi · F(xi) (4)
where F(·) denotes the corresponding predictions of the SLU model. Through utilizing this approach, the SLU model can assign different weights to the generated transcripts, manual transcripts, and ASR transcripts, effectively incorporating their respective strengths within the overall system.

heading: 3.4 Training Objective
text: Pre-training Motivated by recent success of pretrained models (Cao et al., 2022; Li et al., 2022b, 2023; Jin et al., 2023; Dong et al., 2023b; Yang et al., 2024; Huang et al., 2024a), we proceed with training the masked language model (MLM) during the pre-training stage. Following Chang and Chen (2022), the training loss Lpt is the weighted sum of self-supervised contrastive learning loss Lsc and an MLM loss Lmlm as follows:
Lpt = λLsc + (1− λ) · Lmlm (5)
where λ is the coefficient balancing the two tasks.
Fine-tuning Following E et al. (2019); Chen et al. (2022a), the intent detection objective is:
L1I = − N∑
i=1
yIi log y 2,I i (6)
L2I = − N∑
i=1
yIi log ŷi I (7)
where yIi denotes the intent label, y 2,I i is the prediction of the original ASR transcript, ŷiI denotes the weighted average of the intent predictions, N denotes the batch size, and n denotes the number of the tokens in the utterance.
For slot filling, we follow Dong et al. (2023c) to perform it in the sequence generation style, and the slot filling objective is as follows:
L1S = − N∑
i=1
n∑
j=1
yj,Si log y 2,j,S i (8)
L2S = − N∑
i=1
n∑
j=1
yj,Si log ŷi j,S (9)
where yj,Si denotes the slot label, y 2,j,S i denotes the prediction of the original ASR transcript, and ŷij,S denotes the weighted average of slot predictions. We also regularize the weighted average of the predictions and the predictions of ASR transcripts by minimizing JSD between output distributions:
LIJSD = N∑
i=1
JSD(y2,Ii , ŷi I) (10)
LSJSD = N∑
i=1
n∑
j=1
JSD(y2,j,Si , ŷi j,S) (11)
The final fine-tuning loss Lft is as follows:
Lft = L1I + L2I + L1S + L2S + γ(LIJSD + LSJSD) (12) where γ is the coefficient weight.

heading: 4 Experiments

heading: 4.1 Datasets and Metrics
text: Following previous works (Chang and Chen, 2022), all the experiments are conducted on three widelyused benchmark datasets, including SLURP, ATIS, and TREC62. Table 1 presents the statistical information of the three datasets.
SLURP is a challenging SLU dataset, which encompasses diverse domains, speakers, and recording settings. Each intent in SLURP is represented
2The SLURP dataset can be accessed at https://gith ub.com/MiuLab/SpokenCSE, while the ATIS and TREC6 datasets can be accessed at https://github.com/Observe ai-Research/Phoneme-BERT.
as a (scenario, action) pair, and the evaluation metric for SLURP is joint accuracy, which considers a prediction correct only if both the scenario and action are accurately predicted. The corresponding ASR transcripts are obtained using the Google Web API3. ATIS and TREC6 are two additional SLU datasets for flight reservation and question classification tasks, respectively. For these two datasets, the data synthesis involves a text-to-speech (TTS) model followed by ASR transcription.

heading: 4.2 Baselines
text: We compare our MoE-SLU with six pipeline SLU baselines, including RoBERTa (Liu et al., 2019), Phoneme-BERT (Sundararaman et al., 2021), SimCSE (Gao et al., 2021), and SpokenCSE (Chang and Chen, 2022), and MCLF (Huang et al., 2023), and ML-LMCL (Cheng et al., 2023a), and five endto-end SLU baselines, including MTL-SLT (Huang et al., 2022), Speech-Brain (Ravanelli et al., 2021), CTI (Seo et al., 2022), HuBERT SLU (Wang et al., 2021), CIF-PT (Dong et al., 2023c). For a fair comparison, we report the results of MoE-SLU using RoBERTa (Liu et al., 2019) and Data2vec (Baevski et al., 2022) as the pre-trained models. The performance of recent large language models including ChatGPT4(OpenAI, 2023) and SpeechGPT (Zhang et al., 2023) are also reported for comparison.

heading: 4.3 Implementation Details
text: During all the experiments, the model is pre-trained for 10k steps on each dataset utilizing a batch size 128. To avoid overfitting, the model is fine-tuned for up to 10 epochs with the batch size 256. Early stop is employed if the loss on dev set does not decrease for 3 epochs. For the SLURP dataset, two separate classification heads are trained for the scenario and the action, sharing the same embeddings. For all the hyper-parameters, we conduct several experiments and choose the values which perform best. The mask ratio of the MLM task is set to 0.15, τsc is set to 0.2, λ is set to 0.4, and γ is set to 2. During both pre-training and fine-tuning, we utilize the Adam optimizer (Kingma and Ba, 2015) with β1 = 0.9, β2 = 0.98, and 4k warm-up updates to optimize the parameters. The whole training process typically lasts a few hours. All the experiments are conducted on an Nvidia Tesla-A100 GPU.
3https://cloud.google.com/speech-to-text 4https://chat.openai.com

heading: 4.4 Main Results
text: The performance comparison between MoE-SLU and baselines is presented in Table 2. Based on the results, we have the following observations:
(1) When RoBERTa is selected as the pre-trained model, our MoE-SLU outperforms all the baselines except CIF-PT which applies Data2vec as the pretrained model. Besides, when Data2vec is used as the pre-trained model, the performance of our MoESLU further boosts and surpasses all the baselines. This improvement can be attributed to the proposed strategies and the use of different weights assigned to generated transcripts, ASR transcripts, and clean manual transcripts. These distinct weights could enable the effective leveraging of the unique strengths of different types of transcript.
(2) We adopt the evaluation method introduced by He and Garner (2023) to assess the performance of ChatGPT and SpeechGPT using ASR transcripts, where the model is presented with 20 examples and prompted to accommodate for ASR errors. From the results, we can obviously observe that there is a performance gap of approximately 20% between these models and MoE-SLU on the SLURP dataset. This performance degradation can also be observed
in other datasets. This discrepancy highlights the challenges that language models might encounter when it comes to comprehending spoken language in the presence of noise. Therefore, enhancing the robustness of LLMs to ASR input errors remains a highly valuable area of exploration.

heading: 4.5 Analysis
text: In this section, several analytical experiments are conducted. To maintain the fair comparison, we report the performance of MoE-SLU with RoBERTa as the pre-trained model unless stated otherwise.

heading: 4.5.1 Ablation Study
text: To provide evidence for the advantages of our MoESLU model from multiple perspectives, we conduct a set of ablation experiments on MoE-SLU. The experiment results are shown in Table 3.
Effectiveness of the Proposed Strategies. One of the core contributions of MoE-SLU is the proposed three strategies, which enlarges the training set. To evaluate the effectiveness of the three strategies, we conduct three ablation experiments where each strategy is individually removed. We denote the processed results without each strategy as “w/o
Repetition Strategy”, “w/o Deletion Strategy”, and “w/o Swap Strategy” in Table 3, respectively. When any of the strategies is removed, we observe a significant decrease. These results verify that the proposed strategies can make a positive contribution to ASR-Robust SLU. The reason is that these strategies simulate more types of noise in the real-world environments and improve the ability of the model to capture the keywords, thereby indirectly enhancing the positive effects of mixture-of-experts.
Effectiveness of Mixture-of-Experts. Another core contribution of our MoE-SLU is the utilization of the mixture-of-experts approach, which is designed to effectively leverage the different transcripts. To validate the effectiveness of mixture-ofexperts, we conduct an ablation experiment where we remove L2I , L2S , LIJSD, and LSJSD in Eq.12, denoted as “w/o MoE” in Table 3. We could also find the obvious performance degradation. We believe the reason is that through learning the weights in mixture-of-experts, the model can benefit from the distinctive strengths of different transcripts.
Contrastive learning benefits from a larger batch size as it provides more negative examples to facilitate convergence (Chen et al., 2020). To delve into whether the proposed mixture-of-experts approach, rather than the indirectly increased batch sizes, is responsible for the improvements, we double the original batch size after not utilizing mixture-ofexperts, denoted as “w/o MoE + batch size↑” in Table 3. The results show that these improvements are indeed attributed to the proposed mixture-ofexperts approach rather than the boosted batch size.
Effectiveness of JSD. To verify the effectiveness of JSD, we also remove LIJSD and LSJSD in Eq.12 and denoted it as w/o JSD in Table 3. The results demonstrate that the accuracy drops by 0.81, 0.71, and 0.49 in intent and 0.74, 0.63, and 0.45 in slot
on the three datasets, respectively, providing the evidence that JSD further enhances the performance. We suggest that the reason is that JSD serves as a regularization mechanism that further promotes the leveraging of the strengths inherent in different transcripts and helps to prevent overfitting during the entire training process.

heading: 4.5.2 Weight Analysis
text: We also report the learned weights of x1, x2, x3, x4, and x5 in Eq. 4 on these three datasets. It is evident that the transcripts after the deletion strategy carry significant weight on all the datasets, aligning with the findings mentioned in Sec. 4.5.1. Furthermore, the transcripts after applying the repetition strategy and the swap strategy retain certain weights, providing further validation of the effectiveness of our proposed three data augmentation strategies.
Dataset G1 G2 G3 G4 G5
SLURP 0.13 0.18 0.21 0.31 0.17 ATIS 0.15 0.17 0.22 0.33 0.13 TREC6 0.14 0.17 0.23 0.34 0.12
Table 4: The learned weights in Eq. 4 on three datasets.

heading: 4.5.3 Visualization
text: To gain a deeper understanding of the impact and contributions of the mixture-of-experts method, we provide a visualization example on SLURP dataset in Figure 3 based on Principal Component Analysis (PCA) (Abdi and Williams, 2010). In our MoESLU approach, the representations of clean manual transcripts, transcripts after applying the proposed strategies, and their corresponding ASR transcripts are closely aligned, showcasing the robustness of our method in handling ASR errors. However, in the case of the previous best method, ML-LMCL,
although the representations of clean manual transcripts and the corresponding ASR transcripts are also relatively close, they are distant from the associated transcripts after using the proposed strategies, which further supports that our MoE-SLU can effectively capture and utilize the unique information provided by different transcripts, resulting in the improved performance in aligning and comprehending the spoken language.

heading: 4.5.4 Performance at Different Noise Levels
text: To better investigate the impact of different noise levels, we separate the test set of SLURP dataset into eight groups based on their WER and demonstrate the intent accuracy in Figure 4. We choose a previous ASR-Robust SLU model ML-LMCL and an SLU model MISCA (Pham et al., 2023) which is not specially designed for improving ASR robustness. We can obviously observe that MISCA is significantly influenced by ASR errors, which proves the necessity of developing the ASR-Robust SLU frameworks. Compared to ML-LMCL, MoE-SLU indeed improves ASR Robustness more effectively, which further verifies the superiority of our method. We believe the reason is that our method leverages different strengths of each type of transcript.

heading: 5 Conclusion
text: In this paper, we propose a new ASR-Robust SLU framework MoE-SLU. We design three strategies to simulate more kinds of errors in the real environment and apply mixture-of-experts to leverage the different transcripts more effectively. Experiments and analysis on three datasets show that our model significantly outperforms previous models. Future
work will focus on exploring how to make better use of the unique strengths of different transcripts.
Limitations
While MoE-SLU demonstrates substantial improvements over existing ASR-Robust SLU models, it is important to note that as previous works (Chang and Chen, 2022), we currently still rely on the ASR transcripts for pre-training and fine-tuning to align with the target inference scenario. However, ASR transcripts may not always be readily available due to the constraints of ASR systems. Therefore, our future research will aim to improve ASR robustness without relying on any ASR transcripts throughout the training and inference process.
Ethics Statement
We perform all experiments applying publicly available datasets that have been pre-processed for academic research purposes. As a result, these datasets do not contain any information which could identify individuals by name or contain offensive content. Though our framework could achieve state-ofthe-art performance, it is essential to acknowledge that the results generated by our SLU framework might not be entirely perfect. So it is advisable for individuals not to depend solely on the generated results. In real-world applications, it is recommended to be assisted by humans to ensure accuracy.

heading: Acknowledgements
text: This paper was partially supported by NSFC (No: 62176008). We thank all the anonymous reviewers for their insightful comments.

heading: Future Work
text: Future
work will focus on exploring how to make better use of the unique strengths of different transcripts. Limitations
While MoE-SLU demonstrates substantial improvements over existing ASR-Robust SLU models, it is important to note that as previous works (Chang and Chen, 2022), we currently still rely on the ASR transcripts for pre-training and fine-tuning to align with the target inference scenario. However, ASR transcripts may not always be readily available due to the constraints of ASR systems. Therefore, our future research will aim to improve ASR robustness without relying on any ASR transcripts throughout the training and inference process