Enhancing Code-Switching for Cross-lingual SLU: A Unified View of Semantic and Grammatical Coherence
Authors: Zhihong Zhu
Xuxin Cheng
Zhiqi Huang
Dongsheng Chen
Yuexian Zou
Venue: EMNLP 2023 Main (2023)
Keywords: Cross-lingual SLU
Semantic Coherence
Grammatical Coherence

Abstract: Despite the success of spoken language understanding (SLU) in high-resource languages, achieving similar performance in low-resource settings, such as zero-shot scenarios, remains challenging due to limited labeled training data. To improve zero-shot cross-lingual SLU, recent studies have explored code-switched sentences containing tokens from multiple languages. However, vanilla code-switched sentences often lack semantic and grammatical coherence. We ascribe this lack to two issues: (1) randomly replacing code-switched tokens with equal probability and (2) disregarding token-level dependency within each language. To tackle these issues, in this paper, we propose a novel method termed SoGo, for zero-shot cross-lingual SLU. First, we use a saliency-based substitution approach to extract keywords as substitution options. Then, we introduce a novel token-level alignment strategy that considers the similarity between the context and the code-switched tokens, ensuring grammatical coherence in code-switched sentences. Extensive experiments and analyses demonstrate the superior performance of SoGo across nine languages on MultiATIS++.

Enhancing Code-Switching for Cross-lingual SLU:
A Unified View of Semantic and Grammatical Coherence
Zhihong Zhu' Xuxin Cheng Zhiqi Huang
Dongsheng Chen Yuexian Zou"
School of ECE, Peking University, China (zhihongzhu, chengxx, chends]estu pku. edu. cn {zhiqihuang, zouyx)ēpku. edu. cn
Abstract
cross-lingual SLU (Deshpande et al., 2022; Cheng et al., 2023b; Zhu et al., 2023a). Therein, Qin et al. (2020) extended this concept to a code-switched setting, aligning the source language with multi- ple target languages. Qin et al. (2022) incorporate contrastive learning to achieve fine-grained cross- lingual transfer. Based on this, Liang et al. (2022) further propose multi-level contrastive learning framework for explicit alignment of utterance-slot- word structure in cross- pooss-lnngual SLU.
Despite the success of spoken language derstanding (SLU) in high-resource languages, achieving similar performance in low-resource settings, such as zero-shot scenarios, remains challenging due to limited labeled training data. To improve zero-shot cross moos-llingual SLU, re- cent studies have explored sode-swithhed sen- tences containing tokens from multiple lan- guages. However, vanilla code-switched sen- tences often lack semantic and grammatical coherence. We ascribe this lack to two is- sues: (I) randomly replacing code-switched tokens with equal probability and (2) disregard- ing token-level dependency within each lan- guage. To tackle these issues, in this paper, we propose novel method termed SoGo, for zero-shot cross-lingual SLU. First, we use saliency-based substitution approach to extract keywords as substitution options. Then, we introduce novel token-level alignment strat- egy that considers the similarity between the context and the code-switched tokens, ensur- ing grammatical coherence in code-switched sentences. Extensive experiments and analy- ses demonstrate the superior performance of SoGo across nine languages MultiATIS++.
However, most mentinne models adopt code-switching methods, which often lack both mantic coherence and grammatical coherence. This can be attributed two main issues: (1) Current approaches overlook the importance of individual words a sentence, as they are randomly replaced with equal probability, potentially introducing un- necessary translation burden and noise that affects semantic coherence. (2) Code-switched sentences comprising tokens from multiple languages may lack grammatical coherence, token-level depen- dencies within each language are disregarded.
To address these issues, we propose novel framework dubbed SoGo for enhancing code- switching. Concretely, first employ a saliency- based substitution approach identify keywords with high saliency scores. These keywords are then replaced with their target language equiva- lents using bilingual dictionaries, to generate code- switched sentences. Notably, unlike the approach by Liu et al. (2020), our SoGo framework requires only single training process for multiple target languages. Then, we devise novel token-level alignment strategy to bridge the gap between origi- nal context and code-switched tokens. By mapping code-switched tokens back to the original context. SoGo minimizes the impact of token substitutions on token-level coherence, thus maintaining consis- tent contextual space within each language.
1 Introduction
Acting as the interface between users and machines, spoken language understanding (SLU) is crucial in task-oriented dialogue systems (Qin al., 2021; Chen al., 2022; Zhu al., 2023b; Cheng al.. 2023c). While joint training models have led to sig- nificant advancements, most existing models rely on anmotated training data, limiting their scalability to low-resource languages. To this end, zero-shot cross-lingual SLU gains increasing attention.
Existing works have shown the effectiveness of Multilingual BERT (mBERT) in multilingual corpus pre-training, enabling promising zero-shot
EEuual contribution. "Corresponding author.
Experimental results on MultiATIS++ (Xu et al., 2020) demonstrate that employing code-switching

---

Global Saliency Score
Cas Dictionary
expersivee
1g
Mean
Mean
cheapeai 1.47
theapest 26:49
lowest
mBERT
Encoder Layer 12
Encoder Layer 12
MBERT
Document Frequency
shared
Locca Saliency Score
Encoder Layer
Encoder Layer
like
funny
movie
Token-lavel Alignment
English Training Corpus
much
Encoder Layer
Encoder Layer
american airlines...
Id 2: could you tell me the cheapest fore
[ou5]
funmy movie [SEP)
[SSP)
like
funny
movie
Gradiant-based Substituttinn
(8)
(b)
Figure 1: (a) Our proposed gradient-base based substitution in SoGo. The code-switched dictionary comprised salient words has the capacity to more accurately characterize the common semantic space and facilitate the acquisition of semantic connections among different languages by the model. (b) The main architecture of SoGo. We perform token-level alignment in lower six layers of mBERT to enhance the grammatical coherence of the code-switched sentence.
with SoGo leads to significant performance im- provement compared baseline models. Further- more, demonstrate the compatibility of SoGo with other existing SLU structures.
as our sentence encoder. Formally, the input of mBERT is formulated as follows:
x' (ICIS), af....,,,
2 [SFP]),
(2)
where [CL.S] denotes the special symbol for rep- resenting the whole sequence, and [SEP] can be used for separating non-consecutive token quences (Devlin et al., 2019).
2 Method
We describe problem formulation ($2.1) and back- bone model ($2.2) for zero-shot cross-lingual SLU first, before describing SoGo ($2.3). The main architecture of SoGo is illustrated in Figure 1.
Intent Detection. We feed the sentence represen- tation Vess generated from mBERT into a classifica- tion layer to obtain of of softmax (W' h'ass + b'), where W and b' are tunable parameters.
2.1 Problem Formulation
Given each input sentence Zegt in target language, zero-shot cross-lingual SLU means the SLU model is trained in source language dataset S, e.g., (English), and directly applied other target lan- guages datasets T, egg., (Chinese, Japanese):
Slot Filling. We the hidden state predict each slot: of softmax(WSK,, bo), where I denotes the representation of word a/-- Note that mBERT tokenizes words into subwords (Wang et al., 2019), so we adopt the first sub-token's rep- resentation as the whole word representation.
(oige: (gg)) = (Figt),
(1)
where f(-) is the trained model. df represents an intent label and oiy represents slot sequence. Note that T may consist of multiple lgl languages.
Training and Inference. Following previous works, the loss function for intent detection (Ln) and slot filling (Ls) are optimized jointty by cor- responding cross-entropy loss. Finally, the total training loss is the weighted sum two losses:
2.2 Backbone Model
For each input sentence x, follow Qin al. (2022); Liang al. (2022) utilize bilingual dic- tionaries (Lample et al., 2018) generate its corre- sponding multi-lingual one x'. Following previous studies, we select mBERT (Devlin et al., 2019)
C= AsLs.
where A and À denote two hyper-parameters. During inference, we do not use any code- switched sentences.

---

2.3 SoGo
hidden states and original hidden states. The simi- larity scores are then used as weights to aggregate the embedding of original hidden states, reflecting the calculated potential for substituted hidden states in the code-switched sequence. To be specific, for cach hidden state h in h', the similarity score is calculated as Vaswani et al. (2017):
Saliency-ber based Substitution. Intuitively, each word in a sentence contributes differently, and we refer to words with sigmificant impact as keywords. Given a vocabulary set V containing v words, we can identify a salient subset of keywords KC' S V generate code-switched sentences. In our work, gradient-based saliency scores (Li and Yu, 2015; Denil et al., 2014; Arras al., 2019) are used to select keywords, which has been proven effective in many other tasks (Yang et al., 2022; Lai et al., 2021; Lei et al., 2023). Methodologically, let Vn denotes the n-th sentence from S, L.. is the loss between model's prediction of slot filling the and the golden label Yn- For each token win E In, the saliency score is defined as:
score, = -
(6)
The final potential of the code-switched token K2 is the weighted sum of its corresponding h:
exp(score(,.) ai Ev exp(score)
h1 = Danr
-
ans he-
(7) Note that we only employ the alignment in lower six layers to better align the representations for switched tokens due to their superior syntax- capturing ability (Jawahar et al., 2019; Rogers et al., 2020). Eventually, the loss function for each sub- task L. (+ E [4, SJ) in Equation 3 is rewritten as:
Sax (win)) -V
V hu fyo hims
(4)
where hi, represents the embedding of Win. Note that mBERT tokenizes words into subwords, so we calculate the average of the subword saliency scores for each word to obtain the final score. In this manner, the gradient signifies the degree to which a word embedding influences the final deci- sion, while the input considers both the sign and magnitude of the input (Shrikumar et al., 2017). Equation 4 assesses the local significance of objective is
C. L(h", w²) + C(h", ,v0
(8)
in which y" denotes golden label of each task. When s denotes I, we employ mean pooling strat- egy representations of all tokens instead of using cLs and 1 for intent detection, considering the close relationship of the two tasks.
token within sentence. However, contrust global keyword subset K. from S. Fol- lowing Yuan al. (2020), we aggregate all saliency scores for the token wis occurring in S. We then multiply these scores with the inverse document frequency (IDF) (Robertson, 2004) of the token wi:
Experiments
3.1 Experimental Setting
Dataset and Metrics. We conduct our experi- ments on the latest multilingual benchmark dataset MultiATIS++ (Xu et al., 2020),* which consists of nine languages: English (en), Spanish (es), Por- tuguese (pt), German (de), French (fr), Chinese (zh), Japanese (ja), Hindi (hi), and Turkish (tr). The details of MultiATIS++
(w)) = log [[CS Enc Sx (w;), (5) in which N is the total number of words in S. The IDF term balances word frequency and saliency scores by assigning words with high document fre- quency a lower weight and vice versa.
shown in Appendix A. Following Li al. (2021), we evaluate the per- formance of intent prediction using accuracy, slot filling using Fl score, and the sentence- se- mantic frame parsing using overall accuracy.
Finally, top-k salient words are chosen to com- pose the code-switched dictionary K. We opt for the dictionary IC as a replacement strategy instead of random word substitution in sentences.
Implementation Details. We adopt two baseline models as the backbone, i.e., CoSDA (Qin et al., 2020) and GL-CLEF (Qin et al., 2022). Follow- ing previous works, the random rate a for token substitution is set to 0.2. Adam W (Loshchilov and
Token-level Alignment. To further align the space of all hidden states in the code-switched sen- tence, inspired by attention-bace based alignment meth- ods Zhu et al. (2023c); Feng et al. (2022), we map the code-switched sentence to the original sentence by calculating the similarity between substituted
/ttps://giiee thub. com/ comm/amzzon- wttp:///oooooiiieeeem/iieeiettee

---

Model
en
de
es
hi
ja
pt
tr
zh
Avg.
mBERT (Devlin et al., 2019)
98.54
95.40
96.30
94.31
82.41
76.18
94.95
75.10
82.53
88.42
Intent Ace
Z (Chen et al., 2019)
98.54
90.48
93.28
94.51
77.15
76.59
94.62
73.29
84.55
87.00
CoSDA (Qin et al., 2020)
95.74
94.06
92.29
77.04
82.75
73.25
93.05
80.42
78.95
87.32
SoGo Cos (Ours)
98.54
96.97
98.12
96.33
82.76
76.41
97.18
82.96
85.77
90,56
GL-CLEFF (Qin al., 2022)
98.77
97.53
97.05
97.72
86.00
82.84
96.08
83.92
87.6
91.95
SoGo (Ours)
98.89
98.45
98.15
97.74
83.87
84.75
97.73
85.53
89.10
92.69
LAI-MCL (Liang al., 2022)
98.77
98.10
98.10
98.77
84.54
81.86
97.09
85.45
89.03
92.41
m (Devlin et al., 2019)
95.11
80.11
78.22
82.25
26.71
25.40
72.37
41.49
53.22
61.66
ZSJoint (Chen et al., 2019)
95.20
74.79
76.52
74.25
52.73
70.10
72,56
29.66
66.91
68.08
Slot FI
CoSDA (Qin et al., 2020)
92.29
81.37
76.94
79.36
64.06
66.62
75.05
48.77
77.32
73.47
SoGo Cos (Ours)
95.46
84.12
83.84
83.46
57.63
65.78
80.27
55.13
79.56
76.14
GL-CLEF (Qin et al., 2022)
95.39
86.30
85,22
84.31
70.34
73.12
81.83
65.85
77.61
80,00
SoGo GL (Ours)
95.42
87.46
87.01
84.45
74.25
76.69
83,91
67.04
78.53
81.64
LAJ-MCL (I.ang et al., 2022)
96.02
86.59
83.03
82.11
61.04
68.52
81,49
65.20
82.00
78.23
m (Devlin et al., 2019)
87.12
52.69
$2.02
37.29
4.92
7.11
43.49
4.33
18.58
36,29
Overall Ace
2 (Chen et al., 2019)
87.23
41.43
44.46
43.67
16.01
33.59
43.90
1.12
30.80
38.02
CoSDA 9 (Qin et al., 2020)
77.04
57.06
46.62
50.06
26.20
28.89
48.77
15.24
46.36
44.03
SoGo cos (Ours)
88.35
61.43
58.30
56.37
19.70
27.65
60.43
19.30
50.17
49.08
OL-CLEF2 (Qin et al., 2022)
88.02
66.03
59.53
57.02
34.83
41.42
60.43
28.95
50.62
54.09
SoGo GL (Ours) LAI-MCL (Liang et al., 2022)
90.54
72.26
61.05
57.88
39.90
46.95
64.23
29.14
51.31
57.02
89.81
67.75
59.13
57.56
23.29
29.34
61.93
28.95
54.76
52.50
Table 1: Major results (%) on MultiATIS++. Higher is better in all columns. Results with V are taken from Qin et al. (2022), and results with o are taken from Liang et al. (2022). SoGo cos and SoGo GL can be directly compared to CoSDA and GL-CLEF. Note that LAI-MCI. is not open-source.
Hutter, 2019) is used to train SoGo with learn- ing rate of 5e-6. The coefficients in Equation are A, and Äg, where the best ones are selected by searching a combination with the following ranges: (0.8, 0.9, 1.0, 1.1, 1.2]. All experiments are con- ducted single TESLA-VI00. The experimen- tal results of our method are averaged over five runs with different random seed to ensure the final reported results are statistically stable.
SoGo, referred to as SoGo Cos and SoGO GL to compare with LAJ-MCL. Remarkably, SoGo GL outperformed the competitive LAJ-MCL in overall accuracy across 8/9 languages. This highlights the significant potential of our SoGo when combined with existing SLU models.
3.3 Further Analysis
Ablation Study. We conduct an ablation study to explore the effect each component in SoGo. The results are illustrated in Figure 2, which show that each proposed component (i.e., saliency-based substitution and token-level alignment) contributes to the performance positively. It is obvious that the proposed saliency-based substitution has a strong ability to pick out important words boost perfor- mance. Moreover, our token-level alignment has shown greater relative importance of the switched tokens than the vanilla code-switching method. It indicates that such substituted tokens significantly contribute to SoGo's 's prediction.
3.2 Major Results
From the major results in Table 1, we have the fol- lowing observations: (1) SoGo Cos outperforms CoSDA significantll. This is because that code- switched sentences generated by performing salient word substitution and token-level aligmment main- tain better syntactic and semantic consistency than those generated by random substitution. (2) Both SoGO Cos and SoGo GL beat their counterparts with 5.05% and 2.93% average improvement in overall accuracy, respectively. This demonstrates that SoGo can be integrated into existing SLU models, orthogonal of other structures. Notably, the gain in slot filling is more significant than in intent detection, further validating the effective- ness of loken-level alignment. (3) As LAJ-MCL is not open-source, source, we implemented two versions of
Analysis of Saliency-based Substitution. To plore the potential of saliency-based substitution. we perform experiments on SoGo Cos differ- ent strategies with respect to different keyword re- placement rate (cf. Figure 3, where denotes

---

Irtent Ace
Sioeft
Ace
switching method entirely separates the substituted tokens apart from the original context.
22.6
67.40
17.00
switched tokens
(V
original tokens
565
SuGocs
Position
Figure 2: Ablation experiments. "SbS" and "TIA" de- note saliency-basee based substitution and token-level align- ment, respectively.
-10
the number of keywords, and v denotes the total number of words.). We find the performance of randomly selecting keywords significantly declines å decreases, whereas our saliency-based substi- tution continues to perform well even with only 1% keywords. This is because saliency-based sub- stitution prioritizes the most indicative keywords for code-switching. As increases, the additional keywords become less indicative and might even have negative effect on the model's 's performance.
CoSDA
Ours
Figure 4: Token-Level density of SoGO Cos during training procedure.
Conclusion
We proposed a novel semantics-coherent and grammar-cgherent method (SoGo) to enhance code-switching for zero-shot cross-lingual SLU. Specifically, we utilized saliency- based substitu- tion to replace keywords for better semantic coher- ence in the shared space. Besides, we introduced an alignment strategy map the code-switched tokens to original contexts, which solves the grammatical incoherence of code-switching. Experiments on MultiATIS++ showed that code-switching based methods with SoGo obtained competitive perfor- mance compared to counterpart baselines.
48
Accuracy(55)
47
46
All words considered
keywords
45
Saliency
44
Random
42
10%
100%
Number of Keywords
Limitations
Figure 3: Overall Accuracy of SoGo cos in different replacement rate of keywords &. Random denotes se- lecting keywords randomly, while Saliency denotes selecting keywords via saliency-based substitution.
By applying saliency-based substitution and token- level alignment, SoGo achieves significant im- provement the benchmark datasets. Neverthe- less, we summarize two limitations for further dis- cussion and investigation by other researchers:
Analysis of Token-level Alignment. To verify whether token-level alignment preserves a context space to keep grammatical coherence, we plot the degree of dispersion among tokens within a random sentence in Figure 4. Concretely, we retrieve the to- ken embedding from intermediate (sixth) layers for both switched and original tokens and utilize top features calculated from PCA (Abdi and Williams, 2010). From Figure we can conclude that SoGo Cos has shown more compact space for the origi- nal tokens, where switched tokens are also inside the context space. In contrast, the vanilla code-
(1) Like other code-switching methods, the per- formance of the model still relies heavily on the correctness of the bilingual dictionary (Fazili and Jyothi, 2022: Whitehouse et al., 2022). In the future, we continue to enhance the performance of zero-shot cross-lingual SLU by refining code- switching techniques (Cheng et al., 2023a).
(2) SoGo should further leverage language- independent information (Yu et al., 2021), such as dependency relations and POS tags, to ensure syntactic invariance for lingsalingual transfers.

---

Acknowledgements
Barah Fazili and Preethi Jyothi. 2022. Aligning multilin- gual embeddings for improved code-switched natural language understanding. In COLING.
We thank all anonymous reviewers for their con- structive comments. This paper was partially sup- ported by Shenzhen Science & Technology Re- search Program HWWW222201231668077777 2020081411530100) and NSFC (No: 621760088
Yukun Feng, Feng Li, and Philipp Koehn. 2022. To- ward the limitation of code-switching in cross-lingual transfer. In EMNLP.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? ACL.
References
Siyu Lai, Hui Huang, Dong Jing, Yufeng Chen, Jinan Xu, and Jian Liu. 2021. Saliency-basee multi-view mixed language training for zero-shot cross-lingual classification. In EMNLP Findings.
Hervé Abdi and Lynne J Williams. 2010. Principal component analysis. Wiley interdisciplinary reviews: computational statisties.
Leila Arras, Ahmed Osman, Klaus-Robert Müller, and Wojeiech Samek. 2019. Evaluating recurrent neural network explanations. In ACL Workshop.
Guillaume Lample, Alexis Conneau, Marc' Aurelio Ran- zato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In ICLR.
Dongsheng Chen, Zhiqi Huang, and Yuexian Zou. 2022. Leveraging bilinear attention to improve spoken lan- guage understanding. In ICASSP.
Xiaochun Lei, Xiang Cai, Linjun Lu, Zihang Cui, and Zetao Jiang. 2023. Su2ge-net: a salieney-based ap- proach for noe-siecciic class foreground segmenta- tion. Scientific Reports.
Qian Chen, Zhu Zhuo, and Wen Wang. 2019. BERT for joint intent classification and slot filling. CoRR.
Guanbin Li and Yizhou Yu. 2015, Visual saliency based multiscale deep features. In CVPR.
Xuxin Cheng, Bowen Cao. Qichen Ye, Zhihong Zhu, Hongxiang Li. and Yuexian Zou. 2023a. MI-Imel: Mutual learning and large-margin contrastive learn- ing for improving asr robustness in spoken language understanding. In Proc. of ACL Findings.
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021. MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark. EACL.
Xuxin Cheng, Wanshi Xu, Zhihong Zhu, Hongxiang Li, and Yuexian Zou, 2023b. Towards spoken lan- guage understanding via multi-level multi-graines contrastive learning. CIKM.
Shining Liang, Linjun Shou, Jian Pei, Ming Gong, Wanli Zuo, Xianglin Zuo, and Daxin Jiang. 2022. Label-aware multi-level contrastive learning for cross-lingual spoken language understanding. EMNLP.
Xuxin Cheng, Zhihong Zhu, Bowen Cao, Qichen Ye, and Yuexian Zou. 2023c. MRRL: Modifying the reference via reinforcement learning for non- autoregressive joint multiple intent detection and slot filling. In Proc. of EMNLP Findings.
Zihan Liu, Genta Indra Winata, Zhaojiang Lin, Peng Xu, and Pascale Fung. 2020. Attention-informed mixed-language training for zero-shot rings-lingual
task-oriented dialogue systems. In AAAI.
Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. In NeurIPS.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. ICLR.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Ad- ina Williams, Samuel Bowman. Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross- lingual sentence representations. In EMNLP.
Libo Qin, Qiguang Chen, Tianbao Xie, Qixin Li, Jian- Guang Lou, Wanxiang Che, and Min-Yen Kan. 2022. Gl-clef: A global-local contrastive learning frame- work for cross- lingull spoken language understand- ing. ACL.
Misha Denil, Alban Demiraj, and Nando de Freitas. 2014. Extraction of salient sentences from labelled documents. CoRR.
Libo Qin, Minheng Ni. Yue Zhang, and Wanxiang Che. 2020. Cosda-ml: Multi-lingual code-switching data augmentation for zero-shot cross-lingual NI.P. In IJCAI.
Ameet Deshpande, Partha Talukdar, and Karthik Narasimhan. 2022. When is BERT multilingual? isolating crucial ingredients for cross-lingue transfer. In AACL.
Libo Qin, Tianbao Xie, Wanxiang Che, and Ting Liu. 2021. A survey on spoken language understanding: Recent advances and new frontiers. In ISCAI.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In AACI.
Stephen Robertson. 2004. Understanding inverse docu- ment frequency: on theoretical arguments for IDF. Documentation.

---

Anna Rogers, Olga Kovaleva, and Anna Rumshisky, 2020. A primer in bertology: What we know about how BERT works. Trans. Assoc. Comput. Linguis- tics.
A
Dataset
MixATIS++ is an extension of Multilingual ATIS (Table 2). Human-translated data for six languages including Spanish (es), German (de), Chinese (zh), Japanese (ja), Portuguese (pt), French (fr) are added to Multilingual ATIS which initially has Hindi (hi) and Turkish (tr). There are 4,478 utterances in the train set, 500 in the valid set, and 893 in the test set, with 18 intents and 84 slots for each language.
Avanti Shrikumar, Peyton Greenside, and Anshul Kun- daje. 2017. Learning important features through propagating activation differences. In JCML.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NeurlPS.
Utterances
Intent types
Slot
Language
Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019. Cross-lingual BERT transformation for zero-shot dependency parsing. In EMNLP.
#Train
#Valid
#Test
types
hi
1,440
160
578
60
715
17
71
others
4,488
893
84
Chenxi Whitehouse, Fenia Christopoulou, and Igna- cio lacobacci. 2022. Entitycs: Improving zero-shot cross-lingual transfer with entity-centric code switch- ing. In EMNLP Findings.
Table 2: Statistics of MultiATIS++.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-linge cross-lingual effectiveness of bert. In EMNLP.
B Experiment Details
Following the zero-shot setting, we fine-tune the model on en training set and use en validation set for the hyper-parameters search. The best model checkpoint is decided by the overall accuracy on validation set. Our code is based on Py PoTch and Transformers?.
Weijia Xu, Batool Haider, and Saab Mansour. 2020. End-to-end slot alignment and recognition for cross- lingual NLU. In EMNLP.
Yi Yang, Yueyuan Zheng, Didan Deng, Jindi Zhang, Yongxiang Huang, Yumeng Yang, Janet H Hsiao, and Caleb Chen Cao. 2022. Hsi: Human saliency imitator for benchmarking saliency-based model explanations. AAAI
C Baselines
We compare our model to the following baselines.
Sicheng Yu, Hao Zhang. Yulei Niu, Qianru Sun, and Jing Jiang, 2021. COSY: counterfactual syntax for cross-lingual understanding. In ACL.
mBERT. mBERT follows the same model archi- tecture as BERT (Devlin et al., 2019), but instead of training only on monolingual English data, it is trained on the Wikipedia pages of 104 languages with a shared word piece vocabulary, allowing the model to share embeddings across languages.
Michelle Yuan, Mozhi Zhang, Benjamin Van Durme, Leah Findlater, and Jordan L. Boyd-Graber. 2020. Interactive refinement of cross-lingual word embed- dings. In EMNLP.
Zhihong Zhu, Xuxin Cheng, Dongsheng Chen, Zhiqi Huang, Hongxiang Li, and Yuexian Zou, 2023a. Mix before Align: Towards Zero-shot Tross-lingual Sen- timent Analysis via Soft-Mix and Multi- View Learn- ing. In Interspeech.
ZSJoint. Chen et al. (2019) propose a zero-shot SLU model, which is trained on the en training set and directly applied to the test sets of target languages.
Zhihong Zhu, Xuxin Cheng, Zhiqi Huang, Dongsheng Chen, and Yuexian Zou. 2023b. Towards unified spoken language understanding decoding via label- aware compact linguistics representations. In Proc. of ACL Findings.
SoSDA. Qin et al. (2020) propose data augmen- tation framework to generate multi-lingual code- switching data to fine-tune mBERT, which encour- ages the model to align representations from the source and multiple target languages.
Zhihong Zhu, Weiyuan Xu, Xuxin Cheng, Tengtao Song, and Yuexian Zou. 2023c. A dynamic graph interactive framework with label-semantic injection for spoken language understanding. In ICASSP.
h httt::///gtttb.c.c////eiiiaaaaae htmm:hugiingface/trrnsffoo hhttps: thub. com/ wtt.cooooeeeeeeeocoooooooooooooe master/multill inguull,

---

Qin al. (2022) introduce con- trastive learning framework explicitly align rep- resentations across languages for zero-shot cross- lingual SLU.
GL-CLEF.
LAJ-MCL.
Liang et al. (2022) proposes multi- level contrastive learning framework for zero-shot cross- lingull SLU.
D
Generalizability of SoGo
We investigate the potential of the proposed model for other tasks through an additional set of experi- ments. Following Qin et al. (2020), we conduct ad- ditional experiments for natural language inference using XNLI (Conneau et al., 2018), which includes 15 languages. We directly input pairs of sentences into the mBERT encoder, and task-specific clas- sification layer is employed for classification. The models are evaluated based on classification accu- racy (ACC), and the results are presented in Table 3.
Model
Avg.
(Conneau and Lample, 2019)
75.3
SoGo (Ours)
77.4
CoSDA mBERT-bbasd (Wu and Dredze, 2019)
SoGo (Ours)
71.9
Table 3: Natural language inference experiments.


Reviews:
[{"review_id": "oeIR2iMBRr", "reviewer": "Reviewer_ncFV", "summary": "", "questions": "- How should we decide the hyperparameter of replacement rate? As depicted in Fig 2, the performance fairly depends on the replacement ratio.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R1:**\nThank you for your professional review! Saliency methods are commonly employed for model interpretability, while weighted token alignment has also been explored in tasks like NER and POS tagging [1]. However, there remains a gap in related literature on cross-lingual dialogue. Our aim is to intricately devise two techniques for enhancing code-switch augmentation to address two key issues in dialogue understanding tasks, with the intention of extending their applicability to a broader spectrum of dialogue scenarios, including Dialogue State Tracking and Sentiment Classification, etc. If you are interested, please see Reviewer #nHnV for preliminary experiments on more tasks like NLI. **We promise to include the missing related references to the best of our knowledge, and further clarify the connection and difference in the final version**.\n\n**R2:**\nThanks for your valuable suggestions! As indicated in Line #298-304 in Limitation, we **have pointed out** the limitations of the proposed method that rely on dictionaries. This limitation is inherent in all similar code-switching methods, and it is not specific to the technique employed in our proposed method. This is not the focus of this work.\nThe suggestion of employing unsupervised dictionary generation, such as fast_align, is valuable and will be considered in our future work.\n\n**R3:**\nThank you for your insightful references! As indicated in Line #69-71, [2] computes attention scores for pairs of source and target languages individually, leading to different models for each target language. In contrast, our proposed method employs a uniform saliency score to map multiple languages into a consistent space, and thus only requires a single training process for multiple target languages. To thoroughly address your concerns, we have implemented adjustments to the proposed method, replacing the saliency component with an attention-based approach. The additional results on CoSDA are presented in the Table below (averaged by 3 runs).\n| Overall Acc |  Avg.  | \n|:------ |:---------------------:|\n| Attention-based |            47.86          | \n| Saliency-based | 49.08 |\n\nBased on the outcomes, we intuitively infer that the saliency-based method offers improved interpretability for code word replacement, as it can effectively decide the more salient words in the dialogue context. \n\nHere, we want to further discuss the differences between the two: Since attention mechanisms boosted performance on many current NLP tasks [3], using attention weight as the explanation of model predictions is a general approach for many models [4, 5]. However, some recent works [6, 7] cast doubt on attention's interpretability. Besides, [8] claimed that saliency methods are more applicable to model explanations. Therefore, we propose saliency-based substitution for cross-lingual SLU task.\n\nAs you suggested, we consider the discussion of this aspect to be crucial, and we will incorporate it into the main text in the final version to clarify the effectiveness of our proposed method.\n\n**Q1:** \nThank you for your concern! In order to provide further validation for the model's generalizability, we conducted additional experiments on MTOP [9]. These results also show notable and consistent improvements with our proposed method.\n\n| Intent Acc |  Avg.  | \n|:------ |:---------------------:|\n| CoSDA |            90.72          | \n|&nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **92.48**\n| LAJ |         91.04         | \n| &nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **93.30** |\n| **Sloft F1** |  **Avg.**  | \n| CoSDA |            73.34          | \n|&nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **75.63**\n| LAJ |         74.50         | \n| &nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **77.74** |\n| **Overall Acc** |  **Avg.**  | \n| CoSDA |            58.77          | \n|&nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **61.52**\n| LAJ |        60.11         | \n| &nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **64.80** |\n\nDue to time constraints, we conducted tests using replacement rates of [20%, 30%, 40%]. Interestingly, MTOP exhibited a similar performance trend to MixATIS++. Therefore, we empirically conclude that an approximately 40% replacement rate serves as the optimal hyper-parameter in most scenarios. We will report the complete hyper-parameter experiment results in the final version.\n\n**On reproducibility:**\nFollowing the common practice, the source code of this paper will be released upon publication.\n\n**References**\n\n[1] Feng Y et al. Toward the Limitation of Code-Switching in Cross-Lingual Transfer. EMNLP 2022\n\n[2] Liu Z et al. Attention-informed mixed-language training for zero-shot cross-lingual task-oriented dialogue systems. AAAI 2020\n\n[3] Bahdanau D et al. Neural machine translation by jointly learning to align and translate. arXiv 2014\n\n[4] Lin Z et al. A structured self-attentive sentence embedding. arXiv 2017\n\n[5] Ghaeini R et al. Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference. EMNLP 2018\n\n[6] Serrano S et al. Is Attention Interpretable? ACL 2019\n\n[7] Wiegreffe S et al. Attention is not not Explanation. EMNLP 2019\n\n[8] Bastings J et al. The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP  2020\n\n[9] Li H et al. MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark. arXiv 2020\n\nWe appreciate if you would consider increasing your score in light of our response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method outperforms existing methods\n- The ablation studies shows the effectiveness of each proposal. PCA visualization is intriguing also.", "reasons_to_reject": "[Half of proposal is same with existing work]\n- The second proposal (token-level alignment) is exactly the same with the method from [Feng et al.](https://aclanthology.org/2022.emnlp-main.400/), but never mentioned in the manuscript.\n\n[More evaluation needed]\n- Since the method depends on the quality of dictionaries, the method may not be efficiently applied to low-resourced languages, which is the goal of the paper (line 3,34). What if the quality of dictionary is low? For example, if the number of word pairs is low, even if we select top words based on saliency, we may not be able to substitute the words. Using unsupervised dictionary generation (e.g. fast_align) could be more supportive\n- The principle of Saliency-based Substitution is basically same with [Liu et al.](https://ojs.aaai.org/index.php/AAAI/article/view/6362), who selects the top-k important words based on the attention values. Comparing the selection method must make this submission stronger.", "questions_to_authors": "- How should we decide the hyperparameter of replacement rate? As depicted in Fig 2, the performance fairly depends on the replacement ratio.", "paper_topic_and_main_contributions": "This paper proposes a better code-switching method for SLU methods. The authors propose how to select words critical for code-switching, and how to consider the contextual information (semantics and grammar).", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "X"}, "extra_text": {"Missing_References": "- Y. Feng, F. Li, and P. Koehn, “Toward the Limitation of Code-Switching in Cross-Lingual Transfer,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Feb. 2022, pp. 5966–5971. Accessed: Feb. 20, 2023. [Online]. Available: https://aclanthology.org/2022.emnlp-main.400"}}, {"review_id": "bvBLO5rRt4", "reviewer": "Reviewer_YoWR", "summary": "", "questions": "Question A: in the saliency-based substitution part, why words with high document frequency has lower weight? Won't this mitigate the saliency of those words? \n\nQuestion B: in the saliency-based substitution part, are the saliency scores normalized over a sentence before they were added?\n\nQuestion C: It is hard to relate the figure 4(b) to model part in section 3, like what is Lsf and Lid in the figure. Shouldn't them be Ls and Li? There is no illustration in the figure caption or main paper\n\nQuestion D: in the implementation part, why the random rate is set to 0.2? You just randomly chose 0.2? \n\nQuestion E: in the table 1, in the Slot F1 section en column, the LAJ-MCL section  has the highest score, why SoGo score is bolded? Same thing in the overcall acc zh column.\n\nQuestion F: in the analysis of saliency-based substitution part, do you use only one sentence to plot? Cause it is said \"within a random sentence\". It seems that the data points in the figure are quite little. Is the plot based on multiple sentences or only one? If it is based on multiple sentences, what is the number of sentences you used. If it is based on one sentence, isn't it arbitrary to has this conclusion? Figure 3 needs more illustration and analysis.\n\nQuestion G: no need to answer, just a suggestion: put the full name of SoGo in the beginning, not in the conclusion", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R1:**\nThank you for your professional review. Saliency methods are commonly employed for model interpretability, while weighted token alignment has also been explored in tasks like NER and POS tagging. However, there remains a gap in related literature on cross-lingual dialogue. Our aim is to intricately devise two techniques for enhancing code-switch augmentation to address two key issues in dialogue understanding tasks, with the intention of extending their applicability to a broader spectrum of dialogue scenarios, including Dialogue State Tracking and Sentiment Classification, etc. If you are interested, please see Reviewer *#nHnV* for preliminary experiments on more tasks like NLI.\n\n\n**R2:**\nThank you for your valuable suggestion! Figure 3 serves as a case visualization to demonstrate grammatical coherence. Since grammar is difficult to quantify, we employ the token density in the embedding space for illustration. In the final version, we plan to include three additional examples to further illustrate the effectiveness of our method (*Due to website restrictions, pictures cannot be uploaded*).\n\n\n**R3:**\nWe would like to respectfully clarify the results and substance of our paper. To reduce the influence of the randomness caused by the token substitution, all results reported in the paper are averaged by 5 runs. Regarding the main table, we have restructured it to enhance clarity. The results show that our proposed SoGo achieves significant and consistent performance gains over the two baseline models.\n\n|  **Intent Acc**  |  **Avg.**   | \n|:------ |:---------------------:|\n| CoSDA | 87.32 | \n|&nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **90.56** ($\\uparrow$3.24)  | \n| GL-CLeF |         91.95         | \n| &nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **92.69** ($\\uparrow$0.74) |\n| **Sloft F1** |  **Avg.**  | \n| CoSDA |            73.47          | \n|&nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **76.14** ($\\uparrow$2.67)\n| GL-CLeF |         80.00         | \n| &nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **81.64** ($\\uparrow$1.64) |\n| **Overall Acc** |  **Avg.**  | \n| CoSDA |            44.03          | \n|&nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **49.08** ($\\uparrow$5.05)\n| GL-CLeF |        52.50         | \n| &nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **57.02** ($\\uparrow$4.52) |\n\n\n**QA:**\nThe IDF term balances word frequency and saliency scores by assigning words with high document frequency a lower weight and vice versa. It is necessary because some irrelevant stop words (e.g., \"of\" and \"a\") have high total saliency scores, for they appear in the sentence many times.\n\n**QB:**\nThank you for your thorough consideration! Yes, the saliency scores are normalized to ensure training stability. \n\n\n**QC:**\nThank you for your attentive review! Yes, Lsf and Lid in Figure 4 correspond to Ls and Li in Equation 8. We apologize for the inconvenience caused by the figure version update issue. In the final version, we will make the necessary adjustments to relocate Figure 4 to the main text, thus enhancing the overall clarity of the method.\n\n\n**QD:**\nFollowing [1, 2], the random rate for token substitution is set to 0.2 for a fair comparison.\n\n**QE:**\nApologies for the mistakes! We will correct the incorrect bold formatting in the final version.\n\n**QF:**\nThank you for this concern! Note that Figure 3 has been indeed visualized for a single sentence. We observe that our model achieves a more cohesive space for token sequences, where the switched tokens are well-integrated within the context. In contrast, the previous CoSDA method separates the substituted tokens from the original context, resulting in clear distinctions between tokens from different languages. Our approach maintains a consistent embedding space throughout training, fostering effective cross-lingual representations. In the final version, we plan to include three additional examples to further illustrate the effectiveness of our method (*Due to website restrictions, pictures cannot be uploaded*).\n\n**On typo and presentation suggestion:**\nThank you for your careful reading! We will fix them in the final version.\n\n**References**\n\n[1] Qin L et al. GL-CLeF: A Global-Local Contrastive Learning Framework for Cross-lingual Spoken Language Understanding. ACL 2022\n\n[2] Qin L et al. Cosda-ml: Multi-lingual code-switching data augmentation for zero-shot cross-lingual nlp. IJCAI 2020\n\nWe hope that you may consider reevaluating the paper in light of these clarifications and additional experiments that we have performed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Main strengths: This paper proposed two solutions: saliency-based substitution to find important words for replacement in a sentence; token-level alignment to improve token-level dependencies and grammatical coherence. This paper discovered the possible problems in SLU, employed and modified existing techniques to improve the model. The results show that their new method performs better than previous models generally. In their ablation study, they also showed that either solution solely can improve the accuracy. \n\nThe main benefit of this paper is that both methods are easy to understand and reproduce. Also the results show that the methods are quite promising.", "reasons_to_reject": "1. Saliency-based method and token-alignment are not new, and the method part lacks illustration (more details in 'Questions' part)\n2. The author designed the novel method based on the observation of previous models lacking grammatical coherence and token-level dependencies. However, how to know that the accuracy improvement has direct relation to those, especially the grammatical coherence. The token level alignment might improve grammatical coherence and token-level dependencies but not necessary. It would be better if authors studied some examples from vanilla models output and new model output, so as to see if the accuracy are really improved because of coherence and dependencies. \n3. The improvement is not significant. Compared to GL-CLEF and LAJ-MCL (other two models mentioned in the table), almost half the results (14 out of 30) have less than 0.5% accuracy/F1 increment.", "questions_to_authors": "Question A: in the saliency-based substitution part, why words with high document frequency has lower weight? Won't this mitigate the saliency of those words? \n\nQuestion B: in the saliency-based substitution part, are the saliency scores normalized over a sentence before they were added?\n\nQuestion C: It is hard to relate the figure 4(b) to model part in section 3, like what is Lsf and Lid in the figure. Shouldn't them be Ls and Li? There is no illustration in the figure caption or main paper\n\nQuestion D: in the implementation part, why the random rate is set to 0.2? You just randomly chose 0.2? \n\nQuestion E: in the table 1, in the Slot F1 section en column, the LAJ-MCL section  has the highest score, why SoGo score is bolded? Same thing in the overcall acc zh column.\n\nQuestion F: in the analysis of saliency-based substitution part, do you use only one sentence to plot? Cause it is said \"within a random sentence\". It seems that the data points in the figure are quite little. Is the plot based on multiple sentences or only one? If it is based on multiple sentences, what is the number of sentences you used. If it is based on one sentence, isn't it arbitrary to has this conclusion? Figure 3 needs more illustration and analysis.\n\nQuestion G: no need to answer, just a suggestion: put the full name of SoGo in the beginning, not in the conclusion", "paper_topic_and_main_contributions": "This paper proposes a novel framework, SoGo, for zero-shot cross-lingual spoken language understanding to improve the semantic and grammar coherence. Previous approaches using code-switched sentences have two main problems: (1) randomly replace the word with equal probability, however those words have different levels of importance to sentences. (2) lack grammatical coherence and token-level dependencies are disregarded. \n\nThe paper makes two main contributions to solve the problems: (1) use saliency-based substitution to choose the top-k words from sentences and gather all the salient words into a salient subset K. (2) after replace the salient words, use token-level alignment to match the code-switched sentence to the original sentence,  so as to improve the token-level dependencies and grammatical coherence. The experiment results show that those two methods both increase the accuracy of intent detections and slot filling compared with vanilla models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "line 137 denotes->denote\nline 123 are optimized -> is optimized"}, "extra_text": {}}, {"review_id": "X9mXtV4aXz", "reviewer": "Reviewer_uvK4", "summary": "", "questions": "Is the algorithm applied solely on the training data from MultiAtis++? \nIf not, which data are you using? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R1:**\nThank you for your valuable feedback! We will carefully review the method section and make necessary modifications in the final version to enhance clarity. Additionally, we plan to relocate the figure of the main framework to the main text, providing readers with a better understanding of the method.\n\n**R2:**\nThank you for your valuable guidance! Drawing upon the literature and benchmarks you've provided, we conducted additional experiments on MTOP [2]. The results indicate that our proposed method SoGo exhibits robust generalization performance and obtains consistent improvements on the MTOP dataset. We will add these parts to the Appendix in the final version.\n| Intent Acc |  Avg.  | \n|:------ |:---------------------:|\n| CoSDA |            90.7          | \n|&nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **92.4**\n| LAJ |         91.0         | \n| &nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **93.3** |\n| **Sloft F1** |  **Avg.**  | \n| CoSDA |            73.3          | \n|&nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **75.6**\n| LAJ |         74.5         | \n| &nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **77.7** |\n| **Overall Acc** |  **Avg.**  | \n| CoSDA |            58.8          | \n|&nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **61.5**\n| LAJ |        60.1         | \n| &nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **64.8** |\n\n**Q1:**\nYes, we conduct cross-lingual tasks in the zero-shot setting, in which only English labeled sentences with code-switching augmentation are used for training, and evaluation is performed in all other languages. Note that the English labeled sentences used for training are only from MixATIS++, and the code-switching augmentation does not extend beyond its training set.\n\n**On references:**\nThank you for your valuable references! Due to time constraints, we will report the complete experimental results of these two datasets [1, 2] in the final version. \n\n**References**\n\n[1] Schuster S et al. Cross-lingual transfer learning for multilingual task oriented dialog. NAACL 2019\n\n[2] Li H et al. MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark. arXiv 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Solid work, well justified, simple and straightforward.\n\nNumbers hold.\n\nEasy to reproduce. \n\nPotentially applicable to other scenarios\n", "reasons_to_reject": "The method is not completely clear\n\nThe method could be used on other datasets.\n", "questions_to_authors": "Is the algorithm applied solely on the training data from MultiAtis++? \nIf not, which data are you using? ", "paper_topic_and_main_contributions": "This work presents a zero shot approach for spoken language understanding. Authors propose to train a model using code-switched sentences where the words to be code switched are chosen based on their saliency. In opposition to previous approaches (ie. Fazili and Jyothi, 2022; Whitehouse et al., 2022), where a bilingual dictionary is required, this approach scans all the vocabulary to identify a salient subset of keywords to generate code-switched sentences. To prevent changing the semantics of the resulting sentence, authors propose a Token-level Alignment that checks the similarity between hidden states of the original and the code switched sentence and considered as weight to be aggregated to the embedding of the original sentence.\nThe approach is checked on the MultiAtis++ dataset compared with other baselines such as  CoSDA (Qin et al., 2020) and GL-CLEF (Qin et al., 2022) as well as baseline mBERT, backbone of the presented model. Results show a consistent gain performance.\nAblation experiments are carried out\tespecially checking if the replacement based on saliency is bringing gains compared to replacing random words.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "\nCross-lingual Transfer Learning for Multilingual Task Oriented Dialog (Schuster et al 2019)\n\nMTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark (Li et al. 2021)\n"}, "extra_text": {}}, {"review_id": "ecIh7ONHXK", "reviewer": "Reviewer_nHnV", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R1:**\nThanks for your valuable suggestion! Following [1], we conduct additional experiments for **natural language inference**. We use XNLI [2], which covers 15 languages for natural language inference. We feed a pair of sentences directly into the mBERT encoder and a task-specific classification layer is used for classification. Models are evaluated by the classification accuracy (ACC), and the results are shown in the table below.\n| Models |  Avg.  | \n|:------ |:---------------------:|\n| CoSDA-XLM_based |            75.3          | \n|&nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **77.4**\n| CoSDA-mBERT_based |         69.7         | \n| &nbsp;&nbsp;&nbsp;&nbsp; w/ SoGo | **71.9** |\n\nWe will include the detailed experimental settings as well as the additional results in the final version.\n\n**R2:**\nThank you for your insightful suggestion! Figure 3 serves as a case visualization to demonstrate grammatical coherence. Since grammar is difficult to quantify, we employ the token density in the embedding space for illustration. In the final version, we plan to include three additional examples to further illustrate the effectiveness of our method (Due to website restrictions, pictures cannot be uploaded). Due to time constraints, we will include three more case studies in the final version to visually demonstrate the effectiveness of our proposed method.\n\n**On Typos Grammar Style And Presentation Improvements:**\nThank you for your careful reading! We will fix them in the final version.\n\n**References**\n\n[1] Qin L et al. Cosda-ml: Multi-lingual code-switching data augmentation for zero-shot cross-lingual nlp. IJCAI 2020\n\n[2] Conneau A et al. XNLI: Evaluating Cross-lingual Sentence Representations. EMNLP 2018", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel and well-written approach: The usage of task-level saliency for finding substitution words and hidden-state-based word alignments are both used in a novel way for generating code-switched sentences. The methods are also very well-explained despite the short space of 4 pages. \n- Generalizable approaches to other tasks: Their work can have a larger impact as code-switched sentences are used in various different tasks and pre-training as well. So, it might be of good interest to the larger community. \n- Good experimentation and analysis: They correctly validate their results on three tasks of SLU and compare them with baseline models. They further provide a good ablation study along with many analyses. Their experimentation looks solid to me.", "reasons_to_reject": "- Another task: Nitpicking, but it would be nice to see a small experiment if this code-switched sentence generation technique works for any other tasks as well. It would also make the technique more generalizable. \n- Lack of qualitative analysis: Great analysis done throughout the paper. But I would like to see actual qualitative examples of how the proposed technique is different and better than previous baselines. Overall, it would build a stronger intuition for the reader.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper works on improving spoken language understanding (SLU). More specifically, it aims at exploring the usage of code-switched sentences for improving SLU in the zero-shot cross-lingual setting. This paper proposes better generation techniques for code-switched sentences through better word replacement and novel token-level alignment strategies. Models trained using their proposed method SoGo show superior performance across nine languages on the MultiATIS++ benchmark.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The small font in figures: It’s understandable that smaller space means graphs need to be squeezed. But currently, the font is too small for readers to view and understand these graphs/figures. \n\n- L281: Spelling error. gramma -> grammar\n\n- Formatting of captions in Tables and Figures is not consistent. The caption is on the top of Table 1 while it’s on the bottom for the figures. Better to keep it at the bottom I think."}}]

Author rebuttal:

Meta-review: Based on the reviews, the paper proposed a better method for code-switching in spoken language understanding (SLU). The main contributions include the selection of critical words for code-switching and the consideration of contextual information. Reviewers appreciated the outperformance of the proposed method compared to existing methods and the effectiveness shown in the ablation studies. However, there were concerns about the lack of novelty in one of the proposals and the dependency on the quality of dictionaries, which may limit the applicability to low-resourced languages. 

On the positive side, the reviewers mentioned that the paper is easy to understand and reproduce, and the results are promising. They also found the proposed techniques to be well-explained and the experimentation and analysis to be solid. However, they suggested conducting more qualitative analysis to provide concrete examples of the improvements. They highlighted the potential generalizability of the proposed techniques to other tasks.

To summarize, the work is considered to be of strong soundness and it provides sufficient support for its claims and arguments. There is generally strong excitement about the paper, as it contributes to the understanding of code-switching in SLU and offers potential improvements. However, there are concerns about the lack of novelty in one aspect of the work and the limited applicability to low-resourced languages. The reviewers suggest including more qualitative analysis and exploring the generalizability of the proposed techniques.
Decision: Accept-Main