PCAD: Towards ASR-Robust Spoken Language Understanding via Prototype Calibration and Asymmetric Decoupling (2024)

Abstract: Spoken language understanding (SLU) inevitably suffers from error propagation from automatic speech recognition (ASR) in actual scenarios. Some recent works attempt to alleviate this issue through contrastive learning. However, they (1) sample negative pairs incorrectly in pre-training; (2) only focus on implicit metric learning while neglecting explicit erroneous predictions; (3) treat manual and ASR transcripts indiscriminately. In this paper, we propose a novel framework termed PCAD, which can calibrate bias and errors and achieve adaptive-balanced decoupling training. Specifically, PCAD utilizes a prototype-based loss to aggregate label and prediction priors and calibrate bias and error-prone semantics for better inter-class discrimination and intra-class consistency. We theoretically analyze the effect of this loss on robustness enhancement. Further, we leverage a teacher-student model for asymmetric decoupling training between different transcripts and formulate a novel gradientsensitive exponential moving averaging (GSEMA) algorithm for adaptive balance of accuracy and robustness. Experiments on three datasets show that PCAD significantly outperforms existing approaches and achieves new state-of-the-art performance.

heading: 1 Introduction
text: Spoken Language Understanding (SLU) is the core component of intelligent assistants, aiming to capture the semantics information of user queries (Tur and Mori, 2011). The mainstream SLU solutions are divided into two categories: pipeline and endto-end methods (Radfar et al., 2020). Pipeline SLU methods combine automatic speech recognition (ASR) and natural language understanding (NLU) in a cascaded manner, enabling them to easily employ pre-trained language models (Cheng et al., 2023a, 2024; Zhuang et al., 2024; Zhu et al.,
*Corresponding author.
2024a). However, pipeline SLU suffers from error propagation from ASR in actual scenarios, which inevitably damages the performance of cascaded NLU components.
Some studies explore to address this issue through directly correcting ASR errors (Mani et al., 2020; Wang et al., 2020) or adapting networks via masked language modeling (MLM) (Sundararaman et al., 2021; Huang and Chen, 2019; Zhu et al., 2021), which usually introduces extra speechrelated information for correction. Recent works (Chang and Chen, 2022; Cheng et al., 2023b,a) enhance representation through contrastive learning (Chen et al., 2020) to mitigate the impact of ASR errors. Following Chang and Chen (2022); Cheng et al. (2023b,a), we focus on pipeline SLU methods and attempt to improve ASR robustness only by using textual information.
Although these works (Chang and Chen, 2022; Cheng et al., 2023b,a) have achieved promising progress through contrastive learning on ASR and clean transcripts, we discover that they still suffer from (1) sample bias: In the label-agnostic pre-training stage, contrastive learning (Chang and Chen, 2022) may mistakenly sample negative pairs (Zhou et al., 2022), resulting in the separation of features with the same semantics (Mishchuk et al., 2017; Zhou et al., 2022). (2) semantic error: Chang and Chen (2022); Cheng et al. (2023b,a) attempt to alleviate ASR errors by implementing feature alignment in an implicit space. However, they ignore explicit erroneous intentions and treat correct and incorrect predictions equally during metric learning. (3) coupling tendency: Clean and ASR transcripts make different contributions to the accuracy and robustness. Chang and Chen (2022) treat different transcripts indiscriminately and use a mixed set of all transcripts for fine-tuning simultaneously. Despite being aware of this issue, Cheng et al. (2023a) still symmetrically trains different transcripts through dual models.
5235
In this paper, we propose Prototype Calibration and Asymmetric Decoupling (PCAD) to tackle the above issues. For sample bias, we adopt a prototype contrastive loss as a regularizer to calibrate errors between all features and corresponding prototype centroids (Ye et al., 2019; Li et al., 2020) for alleviating representation deterioration. Class prototypes can be used to include label supervision, thereby alleviating sampling bias caused by labelagnostic pre-training. For the second issue, we introduce an error-sensitive prototype loss which adds a calibration term to the regular prototype loss to punish the misclassified predictions during fine-tuning. We emphasize on learning more accurate prototypes by removing misclassified intentions. Figure 1 illustrates our main idea. We further analyze the theoretical effectiveness of our prototype-based approach for calibrating bias and improving robustness from the perspectives of gradient and robust decision-making in section 3.6. For coupling tendency, we innovatively propose an asymmetric algorithm termed GS-EMA for decoupling training, which utilizes the average gradient saliency to achieve the adaptive balance of accuracy and robustness. To our best knowledge, it is the first to calibrate error-prone SLU representation from a prototype perspective and consider adaptivebalanced decoupling training for improving ASR robustness.
Our contributions can be summarized as: (1) We propose an ASR-robust SLU framework termed PCAD, which introduces a prototype-based calibration loss and its error-sensitive variant for calibrating bias in pre-training and reducing errors
in fine-tuning. (2) We introduce a teacher-student model for asymmetric decoupling training and propose a novel GS-EMA algorithm to achieve adaptive balance. (3) Theoretical analysis and extensive experiments on three datasets SLURP (Bastianelli et al., 2020), ATIS (Hemphill et al., 1990) and TREC6 (Li and Roth, 2002) show that PCAD can effectively handle ASR noise and improve robustness in SLU.

heading: 2 Related Work
text: ASR-Robust Spoken Language Understanding. The error propagation of ASR can affect the performance of a series of SLU subtasks. Improving the robustness of SLU tasks to ASR errors has significant practical value. Chang and Chen (2022) first explores utilizing contrastive learning to learn the invariance characteristics between noisy and clean transcripts for improving ASR robustness. Cheng et al. (2023a) realizes the semantic similarity encountered in contrastive learning during the pretraining stage and attempts to alleviate this problem by large-margin contrastive learning (Chen et al., 2021). Cheng et al. (2023b) proposes crossattention mechanisms to differentiate features between clean transcripts and ASR transcripts to improve performance. In this paper, we first improve the ASR robustness of SLU tasks from a prototype perspective, and theoretically demonstrate the capability of our method to calibrate bias and shield partially bounded noise. Contrastive Learning. As a promising paradigm of unsupervised learning, contrastive learning aims to maximize the similarities of positive pairs and
minimize negative pairs in a latent space (Arora et al., 2019; He et al., 2019). Contrastive learning is proposed for the first time in the field of computer vision and has achieved excellent performance in visual unsupervised representation (Chopra et al., 2005; Schroff et al., 2015; Chen et al., 2020; He et al., 2019; Zhu et al., 2024b; Chen et al., 2024). Recent studies introduce contrastive learning into the field of natural language processing and have achieved excellent performance in tasks such as sentence representation (Giorgi et al., 2020; Yan et al., 2021), visual-text modality (Radford et al., 2021), and so on. Chang and Chen (2022) and Cheng et al. (2023a) adopt contrastive learning to improve ASR robustness in SLU tasks. In this paper, we introduce contrastive clustering to achieve error calibration in SLU tasks based on prototypes. Prototype Learning. Prototype learning (Wu et al., 2018) aims to learn prototypes as feature clustering centers from input representation belonging to the same category. Prototypical methods have been widely adopted in the field of computer vision (Snell et al., 2017; Wu et al., 2018; Wang et al., 2022). Pahde et al. (2020); Ji et al. (2020); Li et al. (2021) employ it to extract the overall characteristics of categories as a feature of imbalanced samples and apply it to few-shot learning. Jain et al. (2022); Kushwaha and Fuentes (2023) introduce the idea of prototypes to improve the performance of sound recognition. Wang et al. (2022) proposes a prototype-based method to calibrate erroneous pixels for semantic segmentation. Mustafa et al. (2020) proposes a prototype conformity loss to achieve high-quality adversarial defense. Li et al. (2020) utilizes prototypes as implicit variables and proposes prototypical contrastive learning to handle higher-level characteristics. Benefiting from these works, we focus on misclassified instance samples under noisy SLU tasks and introduce a prototype calibration loss with the capability of bias correction to improve ASR robustness. Decoupling Training. Our method adopts teacherstudent models to achieve decoupling training of clean transcripts and noisy ASR transcripts. Teacher-student models (Hinton et al., 2015) typically extract and compress knowledge through distillation, which has been successfully applied to a wide range of tasks such as semi-supervised learning and model compression. Tang et al. (2021) explores a teacher-student dual model framework for semi-supervised object detection. Cheng et al. (2023a) utilizes mutual learning to symmetrically
train teacher and student models for ASR-robust SLU. In this paper, we introduce the idea of decoupling training to train clean and ASR transcripts. Unlike previous methods, we propose a novel EMA algorithm based on gradient sensitivity to achieve adaptive balance training for updating SLU models.

heading: 3 Method
text: In this section, we will elaborate on the specific pipeline of our PCAD. As shown in Figure 2, our proposed PCAD includes three components: (1) prototype calibration loss in pre-training (in section 3.2); (2) error-sensitive calibration loss in finetuning (in section 3.3); (3) adaptive-balanced decoupling training in fine-tuning (in section 3.4). Further, we provide the total training objective (in section 3.5) and the theoretical analysis (in section 3.6) of our method in ASR robustness.

heading: 3.1 Baselines based on Pre-training
text: Pre-training methods attempt to learn noise invariance and improve ASR robustness by contrastive pre-training on ASR transcripts and manual clean transcripts. Therefore, pre-training-based methods (Chang and Chen, 2022; Cheng et al., 2023b,a) usually consist of two stages: (1) adopting selfsupervised contrastive learning to learn the invariant characteristics of noise between ASR transcripts and clean transcripts, and then (2) finetuning the pre-trained model in downstream tasks. These methods use teacher and student models to process ASR transcripts and clean transcripts respectively, and fine-tune them via distillation loss and cross-entropy loss. We follow two-stage settings and then use a prototype-based strategy to improve the robustness of these baselines.

heading: 3.2 Pre-training with Prototype Calibration
text: Given a mini-batch of input data of N pairs of texts B = {(xcleani , xasri )}Ni=1, we can obtain corresponding implicit states E = {(epi , e q i )}Ni=1 through the last layer of [CLS] in pre-trained RoBERTa (Liu et al., 2019). Following Chang and Chen (2022) and Cheng et al. (2023a), we first utilize self-supervised contrastive learning to preliminarily align clean and noisy features:
Ls = − 1 2N
∑
(e,e+)∈P log
es(e,e +)/τs
∑E e̸=e e s(e,e+)/τs , (1)
where τs is the temperature coefficient, s(·, ·) denotes the cosine similarity function and set P =
{(epi , e q i )} ⋃{(eqi , e p i )} as the matching set of aggregations (epi , e q i ) and (e q i , e p i ).
Prototype Calibration Loss We adopt historical correct predictions to obtain prototype µk ∈ R1×D of category k. µk is updated via exponential moving average (EMA) (Ye et al., 2019; Wu et al., 2018):
µk = ρµk + (1− ρ) ∑|E|
i=1 δ(yi = k) · ei 1 + ∑|E| i=1 δ(yi = k) , (2)
where ρ is the learning rate to update the prototypes and δ(·) is the delta function. For convenience, we define Sk is a set of samples belonging to class k in E , i.e., Sk = {ei |ei ∈ E & yi = k}.
We further define sik = µ̃kẽi as the similarity between the normalized vectors µ̃k and ẽi in the unit hypersphere space, i.e., µ̃k =
µk ||µk||2 and
ẽk = ek
||ek||2 . Therefore, the prototype calibration loss (PCL) with temperature τp used to alleviate sampling bias is represented as:
Lpi = − log esik/τp
esik/τp + ∑
l ̸=k e sil/τp
. (3)
PCL calibrates the hyper-spherical representation of ẽi by bringing ẽi closer to µ̃k and pushing it apart from other prototypes, which injects rich label-semantic priors into the model.

heading: 3.3 Fine-tuning with Error-sensitive Prototype Calibration
text: In fine-tuning, we aggregate the features of manual and ASR transcripts into a set S = {ei}2Ni=1.
and we perform self-supervised contrastive learning (Khosla et al., 2020) in the same metric space:
Lc = − 1|S|
|S|∑
i
|S|∑ j ̸=i δij log es(ei,ej)/τc ∑|S| k ̸=i e s(ei,ek)/τc ,
(4) where yi is the label of ei, and τ is the temperature coefficient, δij = δ(yi = yj) is the delta function..
Error-sensitive Prototype Calibration Loss We approach correct and erroneous predictions with different strategies, making the model focus on explicit erroneous semantics. Following Wang et al. (2022), we firstly calculate the average cosine similarity between all misclassified samples and anchor ei via the correction matrix:
ζi =    1 nfk ∑ j∈Sfk (1− s(ẽi, ẽj)) if Sfk ̸= ∅; 0 otherwise,
(5) where Sfk is a set of misclassified texts in Sk, nfk = |S f k |. The misclassified texts are pulled to the anchor ei through minimizing ζi, thereby achieving remediation of erroneous intention predictions. In our work, ζi is a formal summary of the intention of misclassification. Motivated by Wang et al. (2022), we formulate the error-sensitive prototype calibration loss (ES-PCL) as:
Lei = − log esik/τe−σ(1−pik)ζi
esik/τe−σ(1−pik)ζi + ∑
l ̸=k e sil/τe
,
(6)
where pik denotes the probability P (yi = k|ei) obtained by the softmax operation; σ is the tradeoff hyper-parameter and τe is the temperature.

heading: 3.4 Decoupling Learning for Fine-tuning
text: We train manual and ASR transcripts through teacher θt and student θs models respectively. Following Chang and Chen (2022); Cheng et al. (2023a), we adopt the strategies of self-distillation and mutual distillation to achieve bidirectional consistency in both time and model dimensions:
Ldt = 1
N
N∑
i
KLτd(p t−1 i ||pti),
Lds = 1
N
N∑
i
KLτd(q t−1 i ||qti),
Lm = 1 N
N∑
i
JSτm(p t i||qti),
(7)
where pti and q t i denote the probability output via teacher and student models at t-th stage; KLτd and JSτm are KL and JS divergence with different temperatures. Therefore, the total distillation loss is
Ld = Ldt + Lds + Lm. (8) Adaptive Decoupling Training In order to achieve balanced training of the teacher-student model, we need to find a dynamic indicator to measure the sensitivity between the teacher and student branches during the training phase. Since teacher and student models are jointly optimized through gradient backpropagation, the magnitude of the gradients reflects the sensitivity of the two branches to input data. Motivated by this, we propose a novel gradient-sensitive EMA algorithm to update models in an asymmetric manner adaptively:
θt = ρoθt + (1− ρo)| Sp Sq |θs, (9)
where ρo is the balanced hyper-parameter. We utilize the average L2-norm of gradients to measure the significance of the models θt and θs:
Sp = || ∂Lpce ∂ep ||2, Sq = || ∂Lqce ∂eq ||2, (10)
where Lpce and Lqce are the cross-entropy loss calculated by using pti and q t i :
Lpce = − N∑
i=1
ypi log p t i, Lqce = −
N∑
i=1
yqi log q t i .
(11)
Then, we can obtain the total predicted loss as:
Lce = Lpce + Lqce. (12)

heading: 3.5 Training Objective and Analysis
text: Pre-training Following Chang and Chen (2022); Cheng et al. (2023a), we adopt an MLM loss Lmlm as the regularization term. And the total pre-training loss is :
Lpt = Ls + λ 1 |P| ∑
i∈P Lpi + ηLmlm, (13)
where λ and η are the trade-off hyper-parameters. Fine-tuning The complete fine-tuning loss is:
Lft = Lce + αLc + β 1 |S| ∑
i∈S Lei + γLd, (14)
where α, β and γ are the trade-off hyperparameters.

heading: 3.6 Theoretical Analysis
text: We analyze the effectiveness of our method on ASR robustness from the perspectives of gradient and decision, respectively:
Gradient perspective Similar to other prototype learning methods (Chen et al., 2022; Sharma et al., 2023; Wang et al., 2022; Li et al., 2020), we analyze the effectiveness of Lp and Le on bias and error calibration from a gradient perspective. Since Lp is a lower bound of Le, we take Le as an example to elaborate the effectiveness of prototype calibration. We can calculate the gradient of Lei w.r.t feature ei as follows:
∂Lei ∂ei =
( esik/τe∑ l′ e sil′/τe − 1) µ̃k τe + ∑
l ̸=k
esil/τe∑ l′ e sil′/τe µ̃l τe
︸ ︷︷ ︸ bias calibration + ( esik/τe∑ l′ e sil′/τe − 1) · (σ(1− pik) nfk ∑
j∈Sfk
ej)
︸ ︷︷ ︸ error calibration
.
(15)
We can observe that ∂Lei/∂ei can be formulated into two terms: (1) The bias calibration achieves intra- and inter-class gradient compensation to alleviate feature deterioration caused by sampling bias. This item directly calibrates the gradient by making ei close the intra-class prototype µ̃k and away
from inter-class prototypes µ̃l. (2) The error calibration provides gradient penalties for erroneous predictions set Sfk . This item adds a penalty for incorrect predictions ej , making the weight of incorrect predictions greater than that of the correct predictions.
Decision Perspective Following the analysis of convex outer adversarial polytope (Kolter and Wong, 2017), we first define the representation of polytope space and the average spatial overlap coefficient as follows: Definition 1. We mathematically define a representation polytope space under an ASR noise ϵ with p norm bounded by δ for input samples x through the pre-trained model RoBERTa θ :
Sδ(x; θ) = {RoBERTa(x+ ϵ) s.t., ||ϵ||p ≤ δ}. (16)
Definition 2. The space overlap between two samples xi and xj belonging to different categories yi and yj can be defined as the set of intersections between the respective polytope space:
SOδ(xi, xj) = Sδ(xi; θ) ⋂
Sδ(xj ; θ). (17) Then, the averaged spatial overlap coefficients for all samples can be defined as:
ASOδ = 1 M(M − 1) M∑
i=1
M∑
j=1,yj ̸=yi SOδ(xi, xj),
(18) where M denotes the number of all samples.
Based on this, we argue that our prototype loss can improve robustness as follows: Claim 1. (The existence of robust classifiers.) When we employ a classifier that maintains a margin m between the two closest samples belonging to different classes:
m > max x,y∈Sδ(xi;θ) d(x, y), (19)
where d(·, ·) denotes the distance between x and y, we can obtain a decision boundary with guaranteed robustness against a bounded noisy ϵ. Claim 2. (ES-PCL improves robustness by reducing ASOδ.) Based on Claim 1, we can obtain a classifier that is robust to ASR errors by changing the spatial overlap coefficient of the convex polytope. Our ES-PCL in Eq. 6 can reduce the average spatial overlap coefficient ASOδ by pushing out inter-class prototypes and pulling intra-class prototypes, thereby improving the robustness to ASR noise on the whole dataset.
Claims 1 and 2 reveal that the robustness of the model is directly related to the overlap coefficient of the representation space between the perturbed samples and other categories of samples. Our prototype-based method implicitly obtains a convex class-specific classification area for each category and uses prototypes as the center of polytope space to reduce overlap volume. Therefore, our method is more robust to bounded perturbations of samples while ensuring classification accuracy.

heading: 4 Experiment
text: Datasets and Metric Following previous work (Chang and Chen, 2022; Cheng et al., 2023b,a), we conduct all the experiments on three publicly available benchmark datasets1: SLURP (Bastianelli et al., 2020; Chang and Chen, 2022), ATIS (Hemphill et al., 1990) and TREC6 (Li and Roth, 2002). The statistics of the three datasets included are shown in Table 2. SLURP consists of several (scenario, action) pairs and the prediction is considered correct only when both the scenario and action are predicted correctly. ATIS and TREC6 are two public SLU datasets for flight reservation and question classification, respectively. Similar to (Chang and Chen, 2022; Cheng et al., 2023a), we utilize the synthesized text provided by PhonemeBERT (Sundararaman et al., 2021). The ASR transcripts are obtained by Google Web API. We use accuracy as the metric on ATIS and TREC6 and joint accuracy on SLURP for intent classification. Training Settings and Baselines For fair comparison with (Chang and Chen, 2022; Cheng et al., 2023a), we pre-train PCAD for 10K steps with a batch size 128 on each dataset, and finetune the whole model up to 10 epochs with a batch size of 256. We adopt a early-stop strategy and utilize Adam (Kingma and Ba, 2015) as optimizer with β1 = 0.9, β2= 0.98. The mask ratio of MLM is set to 0.15. The temperature coefficients τs, τp, τc, τp, τd and τm are set to 0.2, 0.1, 0.2, 0.1, 5 and 1, respectively. The hyper-parameter σ in Eq. 6 is set to 0.5. ρ in Eq. 2 is set to 0.99. λ and η in Eq. 13 are empirically set to 0.5 and 1. α, β, and γ in Eq. 14 are empirically set to 0.5, 0.5 and 1, respectively. The balanced hyper-parameter ρo is set to 0.99. All reported accuracy scores of our method in this paper are averaged over 5 runs. All experiments are
1SLURP is available at https://github.com/MiuLab/ SpokenCSE, and ATIS and TREC6 are available at https: //github.com/Observeai-Research/Phoneme-BERT.
conducted on 8 NVIDIA RTX3090 GPUs. We compare our model with five state-of-the-art baselines, including RoBERTa (Liu et al., 2019), PhonemeBERT (Sundararaman et al., 2021), SimCSE (Gao et al., 2021), SpokenCSE (Chang and Chen, 2022) and ML-LMCL (Cheng et al., 2023a).

heading: 4.1 Main Results
text: We conduct a series of quantitative experiments on three datasets, i.e., SLURP ATIS and TREC6, to compare our method with baselines. Table 1 and Table 3 show the experimental results of different models without and with clean transcripts during the fine-tuning stage, respectively. From the results, we have the following observations:
(1) Our PCAD (Full) consistently outperforms all baselines on all tasks and datasets. Without manual transcripts during fine-tuning, our PCAD achieves significant improvements in the accuracy of 2.06%, 1.12%, and 2.01% on SLURP, ATIS, and TREC6 compared to the best baseline (i.e., MLLMCL), even surpassing the ML-LMCL with clean transcripts in fine-tuning. PCAD also achieves significant improvements with manual transcripts in fine-tuning, demonstrating the effectiveness of our
calibration and decoupling strategies. (2) Our PCAD (Full) achieves significant performance improvement on the SLURP dataset. The task on SLURP is more challenging, which requires the correct classification of scenarios and action intentions simultaneously. Our method has greatly improved the joint accuracy of intention pairs, achieving the best joint accuracy (90.58% and 91.89%) on SLURP without and with manual transcripts in fine-tuning. The result further indicates the significant advantages of our method in improving accuracy and robustness in SLU tasks with ASR noise.

heading: 4.2 Components Analysis
text: Effects of the prototype-based loss One of the core contributions of our PCAD is the prototypebased calibration loss, which alleviates bias during the pre-training phase and reduces erroneous predictions in fine-tuning. As shown in Table 1 and Table 3, we conduct a series of experiments to evaluate the effectiveness of PCL and ES-PCL. It can be observed that PCL and ES-PCL contribute to the performance positively during pre-training and fine-tuning, respectively. Removing any component results in a significant decrease in model performance, e.g., reducing joint accuracy by 1.43% and 1.82% on SLURP in Table 1 without PCL and ES-PCL, respectively. The results show that our calibration strategy for pre-training and fine-tuning can achieve dual improvements in the accuracy and robustness of SLU with ASR noise.
Effects of GS-EMA To verify the effectiveness of adaptive-balanced decoupling learning, we remove the GS-EMA strategy to conduct experi-
ments as shown in Table 1 and Table 3. We can observe that when there is no clean transcript for fine-tuning (i.e., ASR hypotheses are fed into both teacher and student branches), accuracy drops by 0.65%, 0.56% and 0.98% on SLURP, ATIS and TREC6, respectively. In addition, the contribution of GS-EMA components to performance is significantly improved when both clean and noisy transcripts are available, i.e., accuracy drops by 1.69%, 0.81% and 1.53% on SLURP, ATIS and TREC6. The results prove that the GS-EMA strategy can indeed promote the teacher model to effectively extract knowledge from the student model to enhance ASR noise robustness. We further replace GS-EMA with a regular EMA algorithm for experiments, as shown in Table 4. We can observe that GS-EMA achieves more significant performance
than the vanilla EMA, which proves that our adaptive balancing algorithm based on gradient sensitivity is effective in improving the performance under noisy settings.
Analysis of the prototype As shown in Table 5, we conduct an experiment to evaluate the effects of different types of prototypes. We use learnable parameters, all predictions, and correct predictions as prototypes, respectively. We observe that the best accuracy can be obtained when we leverage correct predictions as prototypes. We suppose the reason for this is that correct predictions avoid injecting erroneous semantics compared to other prototypes. In addition, the learnable prototypes use a set of learnable parameters as prototypes, which ignores category label information and obtains the lowest performance.

heading: 4.3 Robustness analysis
text: Following Chang and Chen (2022), we divide the test set into four subsets based on the word error rate (WER) of ASR transcripts to evaluate the robustness of our method under different noise levels in Table 6. It can be observed that: (1) Our method outperforms all previous methods under all noise levels on SLURP, i.e., under different WER intervals; (2) When inputting clean transcripts (i.e., WER=0), our model improves by 0.68% in accuracy compared to SpokenCSE, which indicates that PCAD also has great potential in the original SLU task without ASR errors; (3) As the noise level increases, the performance improvement achieved by our model becomes more significant. In the case of the highest noise, our method improves performance by 5.13% in accuracy compared to SpokenCSE. This verifies that our strategy has re-
silience to higher ASR noise and can significantly improve the ASR robustness of the model.

heading: 4.4 Qualitative analysis
text: To exhibit the improvement of intra-class consistency and inter-class discrimination of the feature representation by our PCAD, we randomly select 4096 samples on the SLURP test set and visualize their intermediate features on SpokenCSE and our PCAD through the T-SNE toolkit (van der Maaten and Hinton, 2008). As shown in Figure 3, we can observe that PCAD can better achieve the separation of inter-class features and the aggregation of intra-class features, avoiding feature aliasing.

heading: 5 Conclusion
text: In this paper, we introduce prototype-based losses for pre-training and fine-tuning, namely PCL and ES-PCL, and analyze their remedial effects on sampling bias and semantic errors from a theoretical perspective. It is the first time to improve ASR ro-
bustness from the perspective of prototype calibration. In addition, in order to achieve high-quality decoupling training, we propose a novel update algorithm based on gradient saliency for adaptive balance. Extensive experiments on three datasets demonstrate that PCAD outperforms the state-ofthe-art approaches by a large margin.
Limitations
This article proposes a framework to improve the robustness of SLU models in practical scenarios. Our framework can accurately calibrate ASR errors and improve the performance of pipeline SLU models in practical scenarios. However, the training of our proposed model relies on aligning clean manual transcripts with ASR transcripts, which may not be easily obtained in specific scenarios. Obtaining clean artificial transcripts is time-consuming and labor-intensive. Therefore, exploring how to reduce the proportion of manual clean transcripts or train models only using ASR noise transcripts is more valuable for practical scenarios, which also is an important direction for future work.
Ethics Statement
We conduct all experiments on the public datasets, which do not contain any offensive content or information with negative social impact. The focus of our paper is to improve the noise robustness of the SLU model and our model does not have uncontrollable outputs. Therefore, we ensure that our paper complies with ethical review guidelines.

heading: Acknowledgements
text: We would like to thank all reviewers for their insightful comments. This paper was partially supported by NSFC (No: 62176008). Special acknowledgements are given to AOTO-PKUSZ Joint Research Center for its support.

heading: Future Work
text: Therefore, exploring how to reduce the proportion of manual clean transcripts or train models only using ASR noise transcripts is more valuable for practical scenarios, which also is an important direction for future work