Cyclical Contrastive Learning Based on Geodesic for Zero-shot Cross-lingual Spoken Language Understanding (2024)

Abstract: Owing to the scarcity of labeled training data, Spoken language understanding (SLU) is still a challenging task in low-resource languages. Therefore, zero-shot cross-lingual SLU attracts more and more attention. Contrastive learning is widely applied to explicitly align representations of similar sentences across different languages. However, the vanilla contrastive learning method may face two problems in zero-shot cross-lingual SLU: (1) the consistency between different languages is neglected; (2) each utterance has two different kinds of SLU labels, i.e. slot and intent, the utterances with one different label are also pushed away without any discrimination, which limits the performance. In this paper, we propose Cyclical Contrastive Learning based on Geodesic (CCLG), which introduces cyclical contrastive learning to achieve the consistency between the different languages and adopts geodesic to measure the similarity to construct the positive pairs and negative pairs. Experimental results demonstrate that our proposed framework achieves the new state-of-theart performance on MultiATIS++ and MTOP datasets, and the model analysis further verifies that CCLG can effectively transfer knowledge between different languages.

heading: 1 Introduction
text: Spoken Language Understanding (SLU) holds the central position in the task-oriented dialogue systems (Tur and De Mori, 2011; Qin et al., 2019; Xu et al., 2021; Zhu et al., 2023c, 2024a,b). The primary objective of SLU task is to comprehend and extract relevant information from user utterances. This capability enables the system to discern the user’s current objective and generate appropriate responses. SLU comprises two critical sub-tasks: intent detection, which focuses on identifying users’ intentions, and slot filling, which entails extracting semantic elements from user queries (Chen et al.,
* Corresponding author.
2022; Zhou et al., 2022; Huang et al., 2023; Cheng et al., 2023a,c; Zhu et al., 2023b).
However, the effectiveness of traditional SLU models is intrinsically linked to the availability of extensive annotated data, which poses challenges in the scalability. This challenge is particularly evident in the case of low-resource languages, where the lack of substantial labeled datasets exacerbates scalability issues, hindering the seamless deployment and advancement of SLU models. With the demand for language processing solutions extending across various diverse linguistic landscapes, the necessity for scalable SLU models that can operate effectively in resource-constrained environments becomes increasingly critical.
To tackle these constraints, the concept of zeroshot cross-lingual SLU generalization has emerged as a central focus of interest and investigation. Recently, mBERT (Devlin et al., 2019) has demonstrated significant advancements in zero-shot crosslingual SLU. Building upon this work, Liu et al. (2020) first introduces an attention-informed mixedlanguage training approach for cross-lingual SLU. In addition, the exploration of multilingual codeswitched settings has been extended by Qin et al. (2020a), which entails aligning a source language with target languages. GL-CLEF (Qin et al., 2022) employs contrastive learning, leveraging bilingual dictionaries to construct multilingual views of the same utterance, then encouraging their representations to be more similar than those negative example pairs. LAJ-MCL (Liang et al., 2022) proposes to model the utterance-slot-word structure using a multi-level contrastive learning framework to facilitate explicit alignment, further enhancing performance. FC-MTLF (Cheng et al., 2023b) points out the deficiencies in conventional code-switching methods (Qin et al., 2020a) and introduces an auxiliary multilingual neural machine translation task to facilitate knowledge transfer across different languages. Although existing zero-shot cross-lingual
SLU methods have made promising strides by contrastive learning, we identify two main issues:
(1) The consistency between the different languages is neglected. Although the code-switching method has been applied to construct positive samples in contrastive learning, we find that the consistency between different languages has not been effectively established. Specifically, the distances between the corresponding samples in different languages are inconsistent, which affects the transfer of knowledge across different languages.
(2) The utterances with one different label are also pushed away without discrimination. Traditional contrastive learning methods utilize codeswitching to construct the positive samples and negative samples, bringing tokens with the same label and intent label closer together while pushing other the tokens away. However, this can result in a side effect where tokens with only one different label (slot or intent) can be also indiscriminately pushed away, which undoubtedly hampers the representation modeling of contrastive learning, leading to the suboptimal performance.
In this paper, we propose Cyclical Contrastive Learning based on Geodesic (CCLG) to solve these two problems. For the first problem, we introduce two consistency losses, including the cross-lingual consistency loss and the intra-lingual consistency loss, aiming to boost the consistency between different languages. For the second problem, we abandon the previous approach of directly employing code-switching to construct positive samples and negative samples in contrastive learning. Instead, we utilize geodesic to reconstruct positive and negative samples and employ geodesic-based similarity instead of the traditional similarity metrics, thereby facilitating the learning of representations.
We conduct experiments on MultiATIS++ (Xu et al., 2020) and MTOP (Li et al., 2021), covering nine and six different languages, respectively. The experimental results show that our framework can outperform previous cross-lingual SLU baselines. Further model analysis also indicates that our method can transfer knowledge from high-resource languages to low-resource languages. In summary, our work makes three-fold contributions:
• We use cyclical contrastive learning to achieve consistency between different languages.
• We apply geodesic to construct positive and negative samples in contrastive learning, leading to improved representations of tokens.
• Experiment results show that our framework achieves the new state-of-the-art performance on MultiATIS++ and MTOP datasets.

heading: 2 Related Works
text: The related works are introduced from zero-shot cross-lingual SLU and contrastive learning.

heading: 2.1 Zero-shot Cross-lingual SLU
text: Traditional SLU usually focuses on languages with abundant resources, which limits their widespread use. This limitation has sparked growing interest in a novel approach known as zero-shot cross-lingual SLU. The essence of success in this approach lies in tapping into the linguistic insights present in languages with ample resources. By doing so, it opens up exciting possibilities for overcoming challenges posed by limited data in cross-lingual scenarios. Moreover, it extends the reach of SLU to languages that have been previously overlooked, thereby contributing to a more inclusive and adaptable framework in the field of multilingualism.
In recent years, with the popularity of pre-trained models (Xin et al., 2022; Xin and Zou, 2023; Xin et al., 2023a,b; Yang et al., 2023, 2024a; Dong et al., 2023; Yang et al., 2024b; Hu et al., 2024; Wu et al., 2023; Shen et al., 2023; Feng et al., 2019; Dong et al., 2022), many cross-lingual embeddings, such as mBERT (Devlin et al., 2019), have shown promising results. Liu et al. (2020) propose codemixing to construct training sentences containing both the source and target phrases, implicitly finetuning mBERT. Building upon it, Qin et al. (2020a) proposes multilingual code-switching data augmentation to better align the source language with all target languages. Additionally, van der Goot et al. (2021) suggests three non-English auxiliary tasks to boost cross-lingual transfer. More recently, SOGO (Zhu et al., 2023a) highlights the limitations of the conventional code-switching method and proposes a saliency-based substitution approach for extracting keywords as substitutions. In our method, we use cyclical contrastive learning based on geodesic to further transfer the knowledge from the source language to the target language.

heading: 2.2 Contrastive Learning
text: As attention mechanisms have become increasingly popular, exploring how to obtain the better representations is a highly worthwhile topic of investigation (Yin et al., 2023; Wei et al., 2023a,b, 2024;
𝑜! Decoder NMT Module ℎ! ℎ" ℎ# ℎ$%&ℎ'($ 𝑜!) 𝑜") 𝑜#)
Zhang et al., 2024a,b; Cao et al., 2021, 2022; Jin et al., 2023; Chen et al., 2024). Contrastive learning aims to learn representations of examples via minimizing the distance between positive pairs and maximizing the distance between negative pairs (Saunshi et al., 2019; Chuang et al., 2020; Liu et al., 2022), a concept initially proposed in the field of computer vision (Chopra et al., 2005; Chen et al., 2020; Wang and Liu, 2021). In natural language processing, contrastive learning is utilized for learning the sentence embeddings (Giorgi et al., 2021; Yan et al., 2021), translation tasks (Pan et al., 2021; Ye et al., 2022), and summarization (Wang et al., 2021; Cao and Wang, 2021). Owing to its strong capability in achieving alignment across different languages, contrastive learning has also been used in zero-shot cross-lingual SLU (Liang et al., 2022; Qin et al., 2022). However, we find two main issues with directly using vanilla conservative learning in cross-lingual SLU. As a result, we propose cyclical contrastive learning based on geodesic to tackle these two issues in this paper.

heading: 3 Background
text: SLU comprises two core subtasks, including intent detection and slot filling. Given the input utterance x = (x1, x2, . . . , xn), where n denotes the length of x, intent detection is treated as a classification task, producing the intent label oI , and slot filling is a sequence labeling task, mapping each utterance x to a slot output sequence oS = (oS1 , o S 2 , . . . , o S n). Due to the intrinsic correlation between intent detection and slot filling, it is common to train a unified SLU model capable of jointly handling both tasks. Zero-shot cross-lingual SLU task involves training an SLU model on a high-resource source language, such as English, and seamlessly using it on a low-resource target language, such as Hindi.
In this scenario, when presented with an instance xtarget in the target language, the trained model f can directly generate predictions for both intent and slot values in the target language:(
oItarget,o S target ) = f (xtarget) (1)
where f denotes the trained model and target denotes the target language.

heading: 4 Method
text: In this section, we first introduce the Generic SLU Module (Sec. 4.1) and the previous paradigm of utilizing contrastive learning to enhance zero-shot cross-lingual SLU (Sec. 4.2). Then, we introduce the components of our proposed approach, including Cyclical Contrastive Learning (Sec. 4.3) and Geodesic (Sec. 4.4). Finally, we introduce the final Training Objective (Sec. 4.5). The overview of our approach is demonstrated in Figure 1.

heading: 4.1 Generic SLU Module
text: Given the input sentence x = (x1, x2, ..., xn), the construction of the input utterance is based on each input utterance by incorporating the specific tokens x = ([CLS], x1, x2, ..., xn, [SEP]). Following Qin et al. (2020a), code-switching is applied to leverage the bilingual dictionaries (Lample et al., 2018) in generating multi-lingual code-switched data as the input. The representation of the utterance, denoted as H = (hCLS, h1, . . . , hn, hSEP), is obtained by utilizing the pre-trained mBERT (Devlin et al., 2019). The predicted intent oI and the predicted slot oSt are formulated as follows, respectively:
oI = softmax ( W IhCLS + b I )
(2) oSt = softmax ( W Sht + b S ) (3)
where W I , W S , bI , and bS are trainable parameters, ht is the first sub-token representation of xt.

heading: 4.2 Previous Contrastive Paradigm
text: Contrastive learning has been applied in zero-shot cross-lingual SLU (Qin et al., 2022; Liang et al., 2022). In general, previous methods aim to bring tokens and the corresponding code-switched tokens (positive pairs) closer together while pushing apart tokens and the non-corresponding tokens (negative pairs). And the previous contrastive loss LCL can be formulated as follows:
LICL = − N∑ j=1 log s(hjCLS,h j+ CLS)∑B
hjCLS ̸=h j′ CLS
s(hjCLS,h j′ CLS) (4)
LSCL = − 1
n N∑ j=1 n∑ i=1 log s(hji ,h j+ i )∑B
hji ̸=h j′ i
s(hji ,h j′ i ) (5)
LCL = LICL + LSCL (6)
where s(·) denotes the cosine similarity function, h+CLS denotes the positive sample of hCLS, h + i denotes the positive sample of hi, B denotes the minibatch of original and code-switched tokens, and N denotes the total number of utterences.

heading: 4.3 Cyclical Contrastive Learning
text: Inspired by previous work (Goel et al., 2022), we introduce two additional consistency losses to improve the consistency between different languages, including the cross-lingual consistency loss and the intra-lingual consistency loss.
The cross-lingual consistency loss LCCCL is utilized to reduce the discrepancy in similarity scores between the representations of all mismatched pairs of original tokens and code-switched tokens, which could be formulated as follows:
LCCCL = 1
N N∑ j=1 N∑ i=1 (⟨Hj ,Hi⟩ − ⟨Hi,Hj⟩)2 (7)
where ⟨·, ·⟩ denotes the inner product function, and H denotes the representation of the corresponding code-switched utterance.
The intra-lingual consistency loss LICCL is employed to reduce the discrepancy in the similarity scores between the representations of all the original token pairs and corresponding code-switched token pairs, which could be formulated as follows:
LICCL = 1
N N∑ j=1 N∑ i=1 (⟨Hj ,Hi⟩ − ⟨Hi,Hj⟩)2 (8)
The final cyclical contrastive learning loss LCCL is the sum of LCCCL and LICCL:
LCCL = LCCCL + LICCL (9)

heading: 4.4 Geodesic
text: In the previous contrastive paradigm, only the tokens with the same two labels, including intent and slot, are regarded as the positive pairs. Therefore, the tokens with only one different label (slot or intent) are also pushed apart without discrimination, which limits the overall performance. To solve this problem, we leverage geodesic to discriminate positive pairs in contrastive learning (Li et al., 2023).
The representations of tokens are often embedded within a high-dimensional manifold, and our objective is to gauge the geodesic distance between two points along this manifold. However, calculating the precise geodesic distance proves challenging in the absence of some knowledge regarding the manifold’s structure (Kimmel and Sethian, 1998). To address this, we resort to leveraging the K-NN graph (Cover and Hart, 1967) as an approximation to the manifold structure (Surazhsky et al., 2005; Chowdhury et al., 2022). Within this graph, each token hi constitutes a node, and connections are established between nodes such that each node links to at most k other nodes of the graph.
Specifically, a directed edge is established from the node hi to node hj if hj is one of the k nearest neighbors of hi. The weight of each edge d(hi,hj) is defined utilizing the cosine similarity:
d(hi,hj) = 1− s(hi,hj) (10)
Finally, we employ the shortest path algorithm Dijkstra (Dijkstra, 1959) to compute the length of the shortest path between the two token representations along the obtained weighted directed graph, serving as the final geodesic distance G(hi,hj).
For a token hi, we define the k tokens with the closest geodesic distance from the code-switched tokens as its positive samples Pi:
Pi = { pki } = arg topkG(hi,hj)
k
(11)
In vanilla contrastive learning, for the negative samples with only one different label and the samples with two different labels, the push operation for all negative samples is indistinguishable, which clearly undermines the model to learn the correct representations. To solve this issue, we leverage the geodesic distance to push negative samples away. The similarity SG(hi,hj) between different tokens could be formulated as follows:
SG(hi,hj) = exp(hih ⊤ j · log
1
exp(G(hi,hj) + 1) )
(12)
By considering the relationships between negative samples while maximizing the mutual information, we believe SG(hi,hj) is more beneficial than the conventional similarity function. The geodesicbased contrastive learning loss LGCL are as follows:
LIGCL = − N∑ j=1 log
∑ pkCLS∈PCLS
exp(hjCLS,p k CLS)∑B
hjCLS ̸=h j′ CLS
SG(h j CLS,h
j′ CLS)
(13)
LSGCL = − 1
n N∑ j=1 n∑ i=1 log
∑ pki ∈Pi
exp(hji ,p k i )∑B
hji ̸=h j′ i
SG(h j i ,h
j′ i )
(14)
LGCL = LIGCL + LSGCL (15)

heading: 4.5 Training Objective
text: Following previous work (Qin et al., 2020b, 2022), the intent detection objective LI and the slot filling objective LS are computed as follows:
LI = − nI∑ i=1 ŷIi log ( oIi ) (16)
LS = − n∑
j=1 nS∑ i=1 ŷi,Sj log ( oi,Sj ) (17)
where ŷIi denotes the gold intent label, ŷ i,S j denotes the gold slot label for the j-th token, nI denotes the number of gold intent labels, and nS denotes the number of gold slot labels.
The final training objective L is as follows:
L = αLI + (1− α)LS + λLCCL + γLGCL (18)
where α and λ are two hyper-parameters.

heading: 5 Experiments

heading: 5.1 Datasets and Metrics
text: We primarily conduct our experiments on two public cross-lingual SLU benchmark datasets, including the MultiATIS++ (Xu et al., 2020) dataset and the MTOP (Li et al., 2021) dataset.
MultiATIS++1 dataset is the broadened version of the Multilingual ATIS (Upadhyay et al., 2018) dataset, whose statistics are shown in Table 1. This extension includes human-translated data for an additional six languages: Spanish (es), German (de), Chinese (zh), Japanese (ja), Portuguese (pt), and
1https://github.com/amazon-science/multiatis
French (fr), complementing the original languages, Hindi (hi) and Turkish (tr). The dataset comprises 4,478 utterances in the training set, 500 in the validation set, and 893 in the test set, with a total of 18 intents and 84 slots for each language.
MTOP2 is compiled from interactions between humans and assistant systems, with statistics presented in Table 2. MTOP comprises over 100,000 human-translated utterances in six languages (English (en), German (de), Spanish (es), French (fr), Thai (th), and Hindi (hi)). For the fair comparison, we follow Liang et al. (2022) to utilize the flat version, divided into 70:10:20 percentage splits for the training set, validation set, and test set.
Consistent with prior research (Qin et al., 2022; Zhu et al., 2023a; Cheng et al., 2023d, 2024), accuracy serves as the main metric for evaluating intent detection, and the F1 score is applied to assess the slot filling performance. In addition, overall accuracy is utilized for sentence-level semantic frame parsing evaluation, which is more important.

heading: 5.2 Implementation Details
text: Following Qin et al. (2022), we utilize the base case of the multilingual BERT (mBERT)3(Devlin et al., 2019), featuring N = 12 attention heads and M = 12 transformer blocks. The learning rate is set to 5× 10−7 and the total batch size is equal to 16. During the whole training process, the value of
2https://fb.me/mtop_dataset 3https://github.com/google-research/bert/blob/
master/multilingual.md
label smoothing is set to 0.1, and the dropout rate is set to 0.1. We train our model for 40 epochs, and to avoid overfitting, our training will early-stop if the loss on the development set does not decrease by 10 epochs. We apply Adam optimizer (Kingma and Ba, 2015) with β1 = 0.9, β2 = 0.98, and 4k warmup updates to optimize parameters. Following the zero-shot setting, we choose the model with the highest overall accuracy based on the English development set and subsequently evaluate the model
on test datasets. For all hyper-parameters, we perform several experiments and select the values with the best performance, where α is set to 0.9, λ is set to 0.5, γ is set to 1, and k is set to 5. All the experiments are conducted on an NVIDIA A100 GPU. Our code is based on PyTorch (Paszke et al., 2019) and Transformers4(Wolf et al., 2020) framework.

heading: 5.3 Baselines
text: We compare our proposed CCLG with the following strong cross-lingual SLU baselines:
(1) ZSJoint: We have re-implemented the zeroshot joint model (Chen et al., 2019) (referred to as ZSJoint), trained on the English training set, and directly applied it to the test sets of target languages.
(2) CoSDA: Qin et al. (2021) introduces a novel dynamic code-switching method involving random multilingual token-level replacement. To ensure a fair comparison, we utilize both the English training data and code-switching data for fine-tuning.
(3) GL-CLEF: Qin et al. (2022) proposes a strong global-local contrastive learning framework for ex-
4https://github.com/huggingface/transformers
plicit alignment, achieving the different granularity alignments, including sentence-level local intent alignment, token-level local slot alignment, and the semantic-level global intent-slot alignment.
(4) LAJ-MCL: Liang et al. (2022) introduces a multi-level contrastive learning framework, which is designed for zero-shot cross-lingual SLU.
(5) DiffSLU: Mao and Zhang (2023) introduces a diffusion model and also utilizes knowledge distillation for zero-shot cross-lingual SLU, achieving mutual guidance between intent and slots.
(6) SOGO: Zhu et al. (2023a) further proposes a semantics-coherent and grammar-coherent method to boost the code-switching method for zero-shot cross-lingual SLU, achieving higher performance.
(7) FC-MTLF: Cheng et al. (2023b) introduces a framework for cross-lingual SLU, utilizing codeswitching for coarse-grained alignment and applying machine translation for fine-grained alignment.

heading: 5.4 Main Results
text: The results on MultiATIS++ are shown in Table 3 and the results on MTOP are listed in Table 4. From them, we observe that our method achieves a relative enhancement of 1.83% in average overall accuracy over the previous state-of-the-art model. This notable improvement could be attributed to our innovative approach based on the cyclical contrastive learning method utilizing geodesic techniques. By ensuring consistency across diverse languages and reconstructing both positive and negative samples by applying geodesic methods, our method excels in achieving superior overall accuracy.
Additionally, CCLG demonstrates notable and consistent advancements across all subtasks, particularly showcasing significant improvements in low-resource languages compared to high-resource
languages. The success of CCLG in low-resource languages aligns with the original goal of the zeroshot cross-lingual SLU task, which aims to address the challenges in languages with limited training data. This outcome underscores the effectiveness of our method in transferring knowledge successfully from source languages to target languages.

heading: 5.5 Ablation Study
text: To validate the advantages of CCLG from different perspectives, we conduct several ablation studies on the MixATIS++ dataset, the results of which are demonstrated in Table 5.

heading: 5.5.1 Effect of Cyclical Contrastive Learning
text: CCLG makes a pivotal contribution through its innovative cyclical contrastive learning, strategically achieving consistency across different languages. To meticulously evaluate the impact of this module, we conduct an ablation study by excluding LCCL in Eq. 18, as denoted by "w/o Cyclical Contrastive Learning" in Table 5. The discernible degradation in performance emerges across all metrics for every language when the cyclical contrastive learning module is omitted. We contend that this observed improvement stems from the module’s capability to model the consistency between different languages, particularly beneficial for low-resource languages facing the data scarcity challenges.

heading: 5.5.2 Effect of Geodesic
text: To bolster the effectiveness of geodesic, we conduct an ablation study by excluding LGCL in Eq. 18. This configuration is denoted as "w/o Geodesic" in Table 5. Significantly, our findings reveal a decline in performance across all the metrics for each language, underscoring the importance of geodesic in
constructing positive and negative samples in contrastive learning. This ensures a robust and reliable model performance in real-world applications.

heading: 5.6 Case Study
text: As illustrated in Table 6, we present a case study in English and German to validate the advancements of our model compared to previous zero-shot crosslingual SLU methods. These results reveal notable distinctions in the performance of GL-CLEF, FCMTLF, and our proposed CCLG.
In the case of English, all these models correctly predict the intent. However, as the linguistic complexity increases in German, errors become more pronounced in both GL-CLEF and FC-MTLF, while CCLG maintains correct predictions. It exemplifies the robustness and cross-lingual generalizability of CCLG, outperforming its counterparts in accurately predicting intents across diverse languages, without succumbing to increased linguistic complexity, thereby enhancing overall performance.
In terms of slot filling accuracy, GL-CLEF and FC-MTLF show several errors in English, whereas CCLG maintains accuracy. Moving to German, the errors in GL-CLEF and FC-MTLF become more pronounced, while CCLG continues to maintain a high performance. This observed trend highlights the robust nature of CCLG, showcasing its consistent superiority in accurately predicting slots.

heading: 6 Conclusion
text: In this paper, we propose a novel framework CCLG for zero-shot cross-lingual spoken language understanding (SLU), which utilizes cyclical contrastive learning to achieve the consistency across different
languages and applies geodesic to construct positive samples and negative samples in contrastive learning. Experiments on the MultiATIS++ dataset and the MTOP dataset show that our CCLG outperforms the previous best model and achieves a new state-of-the-art performance. Further analysis also demonstrates that our method could indeed transfer knowledge between different languages effectively.
Limitations
While our method has achieved state-of-the-art performance by modifying the traditional contrastive paradigm, we recognize the potential for the further enhancements through the incorporation of the external knowledge. Considering the recent successes observed with LLMs, we anticipate that harnessing LLMs could yield additional improvements in our model’s performance. Exploring the integration of LLMs into our framework represents a promising avenue. We leave this aspect for future work.
Ethics Statement
We conducted all experiments using publicly available datasets that are free from offensive content or information with negative social impact. Hence, we took measures to ensure that our paper adheres to recent ethical review guidelines. By prioritizing the ethical considerations, our aim is to contribute responsibly to the advancement of NLP technology.

heading: Acknowledgements
text: This paper was partially supported by NSFC (No: 62176008). We thank all the anonymous reviewers for their insightful comments.

heading: Future Work
text: We leave this aspect for future work