Towards Zero-Shot Cross-Lingual Slot Filling and Intent Classification with Cross-Lingual Data Augmentation
Authors: Genta Indra Winata, Andrea Madotto, Pascale Fung (The Hong Kong University of Science and Technology, Salesforce Research Asia, 2021)
Topic: Chatbots and Dialogue Systems

Abstract
Spoken Language Understanding (SLU) in task-oriented dialogue systems involves both intent classification (IC) and slot filling (SF) tasks. The de facto method for zero-shot cross-lingual SLU consists of fine-tuning a pretrained multilingual model on English labeled data before evaluating the model on unseen languages. However, recent studies show that adding a second pretraining stage (continued pretraining) can improve performance in certain settings. This paper investigates the effectiveness of continued pretraining on unlabeled spoken language data for zero-shot cross-lingual SLU. We demonstrate that this relatively simple approach benefits either SF and IC task across 8 target languages, especially the ones written in Latin script. We also find that discrepancy between languages used during pretraining and fine-tuning may introduce training instability, which can be alleviated through code-switching.

Introduction
In task-oriented dialogue systems, a Spoken Language Understanding (SLU) component typically involves intent classification (IC) and slot filling (SF) <cit.> tasks. For example, in “Show me the fares for Delta flights from Dallas to San Francisco“, the intent is asking an airfare and its corresponding slots are Delta (airline-name), Dallas (city-origin), and San Francisco (city-destination). Scaling SLU models to other languages is still challenging, especially when there is limited or no labeled data available in the target language <cit.>. To approach this problem, previous work studies IC and SF tasks in a zero-shot cross-lingual setting <cit.>, where it is assumed that a labeled dataset is only available for a high resource language (e.g., English). With the rise of pretrained multilingual language models (LMs) <cit.> the most common approach is by fine-tuning the pretrained multilingual model on the English labeled data, and then evaluate the model directly on the target language data that are not seen during fine-tuning. While direct fine-tuning serves as a strong baseline, pretrained LMs are not necessarily universal and they may need domain-specific adaptation. Recent works have shown that adding a second pretraining stage (or continued pretraining) before fine-tuning can give positive impact on the model performance <cit.>. During continued pretraining, we continue training the pretrained language model using a domain-specific or task-specific unlabeled dataset, with the same masked language model objective. This stage is useful to alleviate the domain mismatch between the original pretraining and the target task data. By continued pretraining on domain specific unlabeled data, the model acquires prior knowledge which is expected to be helpful in the fine-tuning stage. This approach has shown promising results on text classification, typically on English. However, it remains unclear whether it is applicable in the context of zero-shot cross-lingual SLU. The overall stages of zero-shot cross lingual SLU using a pretrained multilingual model. The standard approach follows the stages marked with blue arrows (direct fine-tuning). We investigate the effectiveness of adding a continued pretraining stage (red dashed box) in the overall pipeline. In contrast to previous work which has mostly focused on English classification tasks, we investigate the effectiveness of continued pretraining for zero-shot cross-lingual SLU tasks on eight target languages. Our study reveals that the existing continued pretraining method <cit.>, that is successful in English text classification tasks, does not always generalize to the context of zero-shot cross-lingual SLU. We focus on the following research questions: (Q1) Is continued pretraining effective for zero-shot cross-lingual SLU tasks? Our experiments on the MultiATIS++ dataset <cit.> reveal that incorporating continued pretraining on intermediate English data can improve performance over direct fine-tuning for all languages, on zero-shot SLU. The performance gain is especially evident for languages with Latin script writing system. The benefit of continued pretraining diminishes as we inject cross-lingual supervision in the fine-tuning stage, even with simple data augmentation through code-switching. (Q2) What are the factors that influence the effectiveness of the continued pretraining stage? Using the target language for continued pretraining before fine-tuning on English can be detrimental to the overall performance. However, this can be largely alleviated by code-switching the fine-tuning data. We also observe that performance improvement are not obtained by merely adding more continued pretraining data; higher domain similarity between the continued pretraining data and the fine-tuning data is indeed more important.

Conclusion
We systematically study the effectiveness of continued pre-training of a multilingual model on intermediate English unlabeled spoken language data for zero-shot cross-lingual tasks, namely intent classification and slot filling, on 8 languages. Our results show that the domain knowledge learned in English is transferable to other languages. The gain from continued pre-training diminishes as we inject cross-lingual supervision in the fine-tuning stage. There are several factors that influence the effectiveness of the continued pre-training: (i) Using different language between pre-training and fine-tuning can hamper performance and introduce instability in the model training, which can be alleviated with code switching. (ii) Domain similarity is important. The more similar – in terms of data distribution – the intermediate data to the target dataset yields better performance.