Venue: EMNLP 2020

# End-to-End Slot Alignment and Recognition for Cross-Lingual NLU

Weijia XuUniversity of Maryland weijia@cs.umd.edu&Batool HaiderAmazon AI bhaider@amazon.com &Saab MansourAmazon AI saabm@amazon.com

Work done at Amazon AI.

###### Abstract

Natural language understanding (NLU) in the context of goal-oriented dialog systems typically includes intent classification and slot labeling tasks. Existing methods to expand an NLU system to new languages use machine translation with slot label projection from source to the translated utterances, and thus are sensitive to projection errors. In this work, we propose a novel end-to-end model that learns to align and predict target slot labels jointly for cross-lingual transfer. We introduce MultiATIS++, a new multilingual NLU corpus that extends the Multilingual ATIS corpus to nine languages across four language families, and evaluate our method using the corpus. Results show that our method outperforms a simple label projection method using fast-align on most languages, and achieves competitive performance to the more complex, state-of-the-art projection method with only half of the training time. We release our MultiATIS++ corpus to the community to continue future research on cross-lingual NLU.

## 1 Introduction

As a crucial component of goal oriented dialogue systems, natural language understanding (NLU) is responsible for parsing an utterance into a semantic frame to identify the user's need. These semantic frames are structured by what the user intends to do (the _intent_) and the arguments of the intent (the _slots_) [20]. Given the English example in Figure 1, we identify the intent of the utterance as "flight" and label the slots to extract the departure city and airline name. Intent detection can be modeled as a sentence classification task where each utterance is labeled with an intent \(y_{I}\). Slot filling is typically modeled as a sequence labeling task where given the utterance \(\mathbf{x}_{1...n}\), each word \(x_{i}\) is labeled with a slot \(y_{i}\).

Despite the high accuracy achieved by neural models on intent detection and slot filling [7, 35], training such models on a new language requires additional efforts to collect large amounts of training data. One would consider transfer learning from high-resource to low-resource languages to minimize the efforts of data collection and annotation. However, currently available multilingual NLU datasets [4, 6] only support three languages distributed in two language families, which hinders the study of cross-lingual transfer across a broad spectrum of language distances. In this paper, we release a new multilingual NLU corpus that contains training, development, and test data for six new languages in addition to the three languages in the Multilingual ATIS corpus [4]. The resulting corpus, namely MultiATIS++, consists in total of 37,084 training examples and 7,859 test examples covering nine languages in four language families.

Using our corpus, we explore the use of multilingual BERT encoder [4], machine translation (MT), and label projection methods for multilingual training and cross-lingual transfer. Furthermore, we propose an end-to-end model for joint slot label alignment and recognition, so that it no longer relies on slot label projections using external word alignment tools [18, 6] or engineered features [1, 10], which may not generalize well to low-resource languages. Our model performs soft label alignment using an attention module which is trained jointly with other model components on intent classification, slot filling, and an augmented reconstruction objective designed to improve the soft label alignment.

Experimental results show that our method uses the same amount of training time as a simple label projection approach using fast-align, while achievi

ing significantly higher slot F1 on most languages. Furthermore, our method achieves competitive performance to the more complex, state-of-the-art label projection method that uses linguistic features to improve projection quality, while using half of the training time. Finally, our results show different trends when comparing various cross-lingual transfer methods on different languages, which emphasizes the need to evaluate cross-lingual transfer methods on a diverse set of languages to fully illustrate the strengths and weaknesses of each method.

## 2 Related Work

Cross-lingual transfer learning has been studied on a variety of sequence tagging tasks including part-of-speech tagging Yarowsky et al. (2001); Tackstrom et al. (2013); Plank and Agic (2018), named entity recognition Zirikly and Hagiwara (2015); Tsai et al. (2016); Xie et al. (2018), and natural language understanding He et al. (2013); Upadhyay et al. (2018); Schuster et al. (2019). Existing methods can be roughly categorized into two categories: transfer through multilingual models and transfer through machine translation.

Transfer via Multilingual ModelsFor closely related languages with similar alphabets, it is beneficial to train a multilingual model with shared character encoder to learn common character-based features Yang et al. (2017); Lin et al. (2018). However, such techniques are less effective when applied to dissimilar languages that lack common lexical features. Chen et al. (2018, 2019) focus on the multisource transfer scenario and apply adversarial training to extract language-invariant features shared by source languages. Recent advances on cross-lingual representations have enabled transfer between dissimilar languages. Representations from multilingual neural machine translation (NMT) encoders have been shown to be effective for cross-lingual text classification Eriguchi et al. (2018); Yu et al. (2018); Singla et al. (2018). In this work, we use multilingual BERT Devlin et al. (2019), an unsupervised cross-lingual language model trained on monolingual texts from a wide range of languages and has been shown to provide powerful sentence representations that lead to promising performance for zero-resource cross-lingual language understanding tasks Pires et al. (2019). We leave the use of other recently proposed cross-lingual language models such as XLM-R Conneau et al. (2020) to future work.

Transfer via Machine Translationrequires translating the source language training data into the target language or translating the target language test data into the source language. Despite its empirical success on cross-lingual text classification tasks Wan (2009), it faces a challenging problem on the sequence tagging tasks: labels on the source language sentences need to be projected to the translated sentences. Most of the prior work relies on unsupervised word alignment from statistical MT Yarowsky et al. (2001); Shah et al. (2010); Ni et al. (2017) or attention weights from NMT models Schuster et al. (2019). Other heuristic approaches include matching tokens based on their surface forms Feng et al. (2004); Samy et al. (2005); Ehrmann et al. (2011) and more complex projection approaches that combine linguistic features with information from the MT systems Jain et al. (2019). By contrast, our method does not rely on external word alignment or linguistic features, but models label projection through an attention module that can be jointly trained with other model components on the machine translated data.

## 3 Data

One of the most popular datasets for multilingual NLU is the ATIS dataset Price (1990) and its multilingual extension Upadhyay et al. (2018). The ATIS dataset is created by asking each participant to interact with an agent (who has access to a database) to solve a given air travel planning problem. Upadhyay et al. (2018) extend the English ATIS to Hindi and Turkish by manually translating and annotating a subset of the training and test data via crowdsourcing.1

Footnote 1: [https://catalog.ldc.upenn.edu/LDC2019T04](https://catalog.ldc.upenn.edu/LDC2019T04)

To facilitate study on cross-lingual transfer across a broader spectrum of language distances, we create the MultiATIS++ corpus by extending both the training and test set of the English ATIS corpus to six additional languages.2 The resulting corpus covers nine languages in four different language families including Indo-European English, Spanish, German, French, Portuguese, and Hindi), Sino-Tibetan (Chinese), Japonic (Japanese), and Altaic (Turkish).

Footnote 2: [https://github.com/amazon-research/multiatis](https://github.com/amazon-research/multiatis)

For each new language, we hire professional native translators to translate the English utterances and annotate the slots at the same time. When 

translating, we ask the translators to preserve the spoken modality phenomena (e.g. hesitations and word repetitions) and style (e.g. degree of formality) of the original English sentences, so that it is closer to the real scenaria. To get the slot labels for the translated utterances, we ask the translators to tag the segments in the translated utterances that are aligned to the corresponding English segments. Finally, we tokenize the translated utterances and BIO tag the tokens. For quality control, we ask a third-party to perform several rounds of qualification checks until no issues are reported.

We show an English training example and its translated versions in the other eight languages in Figure 1 and report the data statistics in Table 1. We split the English training and development sets randomly and keep the same split for all the other languages except for Hindi and Turkish from the Multilingual ATIS corpus. Note that the Hindi and Turkish portions of the data are smaller than the other languages, covering only a subset of the intent and slot types.

## 4 Cross-Lingual NLU

### Joint Intent Detection and Slot Filling

Following Liu and Lane (2016), we model intent detection and slot filling jointly. We add a special classification token \(x_{0}\) at the beginning of the input sequence \(\boldsymbol{x}=(x_{1},x_{2},...,x_{T})\) of length \(T\) following Devlin et al. (2019). Next, an encoder \(\Theta_{enc}\) is used to produce a sequence of contextualized representations \(\boldsymbol{h}_{0...T}\) given the input sequence

\[\boldsymbol{h}_{0...T}=\Theta_{enc}(x_{0},x_{1},...,x_{T})\]

Figure 1: An English training example and its translated versions in the MultiATIS++ corpus. The English utterance is manually translated to the other eight languages including Spanish (ES), Portuguese (PT), German (DE), French (FR), Chinese (ZH), Japanese (JA), Hindi (HI), and Turkish (TR). For each language, we show the utterance followed by the slot labels in the BIO format. The intent of the utterances is the _flight_ intent.

 

For intent detection, we take the representation \({\mathbf{h}_{0}}\) corresponding to \(x_{0}\) as the sequence representation and apply a linear transformation and a softmax function to predict the intent probability

\[p_{\text{intent}}(\cdot|{\mathbf{x}})=\text{softmax}({\mathbf{W}^{I}}{\mathbf{ h}_{0}}+{\mathbf{b}^{I}})\]

For slot filling, we compute the probability for each slot using the representations \({\mathbf{h}_{1...T}}\)

\[p_{\text{slot}_{i}}(\cdot|{\mathbf{x}})=\text{softmax}({\mathbf{W}^{S}}{\mathbf{h}_{i}}+{\bm {b}^{S}})\]

We explore two different encoder models:

* **LSTM:** We use the concatenation of the forward and backward hidden states of a bidirectional LSTM Schuster and Paliwal (1997) as the encoder representations. We initialize the encoder and embeddings randomly.
* **Multilingual BERT:** We use the multilingual BERT Devlin et al. (2019) pre-trained in an unsupervised way on the concatenation of monolingual corpora from 104 languages. We take the hidden states from the top layer as the encoder representations and fine-tune the model on the NLU data.

### Problems in Slot Label Projection

Past work has shown the effectiveness of using MT systems to boost the performance of cross-lingual NLU Schuster et al. (2019). More specifically, one first translates the English training data to the target language using an MT system, and then projects the slot labels from English to the target language. Prior work projects the slot labels using word alignments from statistical MT models Yarowsky et al. (2001) or attention weights from neural MT models Schuster et al. (2019). The final performance on the target language highly depends on the quality of the slot projection. Jain et al. (2019) show that improving the quality of projection leads to significant improvements in the target performance on cross-lingual named entity recognition. However, the improvements come at the cost of much more complex and expensive projection process using engineered features.

To address the above issues, we propose a soft-alignment model that performs end-to-end slot alignment and recognition using an additional attention module (Figure 2), so that it requires no external slot projection process. Furthermore, we show that the soft slot alignment can be strengthened by building it on top of strong encoder representations from multilingual BERT.

### End-to-End Slot Alignment and Recognition via Attention

Given a source utterance \({\mathbf{s}_{1...S}}\) of length \(S\) and its translation \({\mathbf{t}_{1...T}}\) of length \(T\) in the target language, the model learns to predict the target slot labels and soft-align it with the source labels via attention. First, it encodes the source utterance into a sequence of embeddings \({\mathbf{e}_{1...S}^{(src)}}\) and then encodes

Figure 2: Architecture of our soft-alignment model for end-to-end slot alignment and recognition. The model is trained without external label projection: it learns to soft-align the representations of the target utterance to the source slot labels and predict the intent and slot labels jointly. The dotted line denotes the path during inference, where we directly connect the encoder module to the intent and slot classification layer to make predictions on the target utterance.

 

the translation \({\mathbf{t}_{0...T}}\) (\(t_{0}\) is inserted as the classification token) into a sequence of contextualized representations \({\mathbf{h}^{(tgt)}_{0...T}}=\Theta_{enc}({\mathbf{t}_{0...T}})\), where \(\Theta_{enc}\) represents the encoder. For intent classification, we assume that the translated utterance has the same intent as the source utterance. Thus we compute the intent probabilities using the representation \({\mathbf{h}^{(tgt)}_{0}}\)

\[p_{\text{intent}}(\cdot|{\mathbf{t}})=\text{softmax}({\mathbf{W}^{I}}{\mathbf{h}^{(tgt)}_{0 }}+{\mathbf{b}^{I}})\]

and the intent classification loss given the intent label \(y^{(src)}_{I}\) on the source utterance

\[\mathcal{L}_{\text{intent}}=-\log p_{\text{intent}}(y^{(src)}_{I}|{\mathbf{t}})\]

For slot filling, we introduce an attention module to connect the source slot labels \({\mathbf{y}^{(src)}_{1...S}}\) with the target sequence \({\mathbf{t}_{1...T}}\). First, we compute the hidden state at each source position as a weighted average of the target representations

\[{\mathbf{z}_{i}}=\sum_{j=1}^{T}a_{ij}{\mathbf{h}^{(tgt)}_{j}}\]

where \({\mathbf{z}_{i}}\) is the hidden state at source position \(i\), and \(a_{ij}\) is the attention weights between the source word \(s_{i}\) and translation word \(t_{j}\). To compute the weights \(a_{ij}\), we first linearly project the query vector \({\mathbf{e}^{(src)}_{i}}\) and the key vector \({\mathbf{h}^{(tgt)}_{j}}\) with learnable parameters to \(d\) dimensions. We then perform the scaled dot-product attention on the projected query and key vectors

\[{\mathbf{a}_{i}}=\text{softmax}\left(\frac{({\mathbf{e}^{(src)}_{i}}{\mathbf{W}^{Q}})({ \mathbf{h}^{(tgt)}}{\mathbf{W}^{K}})^{\mathsf{T}}}{\sqrt{d}\tau}\right)\]

where the projections \({\mathbf{W}^{Q}}\) and \({\mathbf{W}^{K}}\) are parameter matrices, and \(\tau\) is a hyperparameter that controls the temperature of the softmax function.

Next, we compute the slot probabilities at the source position \(i\) using the hidden state \({\mathbf{z}_{i}}\)

\[p_{\text{slot}_{i}}(\cdot|s_{i},{\mathbf{t}})=\text{softmax}({\mathbf{W}^{S}}{\mathbf{z}_ {i}}+{\mathbf{b}^{S}})\]

and the slot filling loss given the slot labels \(y^{(src)}_{1...S}\) on the source utterance

\[\mathcal{L}_{\text{slot}}=-\sum_{i=1}^{S}\log p_{\text{slot}_{i}}(y^{(src)}_{ i}|s_{i},{\mathbf{t}})\]

In addition, to improve the attention module to better align the source and target utterances, we add a reconstruction module consisting of a position-wise feed-forward and a linear output layer3 to recover the source utterance using the attention outputs. We compute the probability distribution over the source vocabulary at position \(i\) as

Footnote 3: We tie the output weights \({\mathbf{W}^{R}}\) with BERT embeddings.

\[\begin{split}& p_{\text{rec}_{i}}(\cdot|s_{i},{\mathbf{t}})=\text{ softmax}({\mathbf{W}^{R}}{\tilde{\mathbf{z}}_{i}}+{\mathbf{b}^{R}})\\ &{\tilde{\mathbf{z}}_{i}}=\text{FeedForward}({\mathbf{z}_{i}})\end{split} \tag{1}\]

and the reconstruction loss as

\[\mathcal{L}_{\text{rec}}=-\sum_{i=1}^{S}\log p_{\text{rec}_{i}}(s_{i}|s_{i},{ \mathbf{t}}) \tag{2}\]

The final training loss is \(\mathcal{L}=\mathcal{L}_{\text{inert}}+\mathcal{L}_{\text{slot}}+\mathcal{L}_ {\text{rec}}\).

Empirically, we find it beneficial to train the model jointly on the machine translated target data using the objective \(\mathcal{L}\) and the source data using the supervised objective.

The attention and reconstruction modules are only used during training. During inference, we directly feed the encoder representations \({\mathbf{h}^{(tgt)}_{0...T}}\) of the target language utterance to the intent and slot classification layers

\[\begin{split}& p_{\text{intent}}(\cdot|{\mathbf{t}})=\text{ softmax}({\mathbf{W}^{I}}{\mathbf{h}^{(tgt)}_{0}}+{\mathbf{b}^{I}})\\ & p_{\text{slot}_{*}}(\cdot|{\mathbf{t}})=\text{softmax}({\mathbf{W}^{S}} {\mathbf{h}^{(tgt)}_{i}}+{\mathbf{b}^{S}})\end{split} \tag{3}\]

## 5 Multilingual NLU

In our first set of experiments, we explore using pretrained multilingual BERT encoder for multilingual NLU. We compare the following training strategies to leverage the full supervised training data:

* **Target only:** Train each model on the target language training data using the standard supervised objective.
* **Multilingual:** Train a model on the concatenation of training data from all languages using the standard supervised objective.

SetupWe train the models using the Adam optimizer Kingma and Ba (2015) for 20 epochs and select the model that performs the best on the development set (details in Appendix A). Following Goo et al. (2018), we use intent accuracy and slot F1 as evaluation metrics.

 

ResultsTable 2 shows the results using full training data and the supervised objective. First, we compare LSTM and BERT models trained on the target language data only. Multilingual BERT encoder brings significant4 improvements of 1-6% on intent accuracy and 1-11% on slot F1. The largest improvements are on the two low-resource languages: Hindi and Turkish. Multilingual training on all languages brings further improvements on Hindi and Turkish: it improves intent accuracy by 2-5% and slot F1 by 4-11% for both LSTM and BERT models.

Footnote 4: All mentions of statistical significance are based on paired Student’s t-test with \(p<0.05\).

Comparison with SOTAOn English ATIS, Qin et al. (2019) report 97.5% intent accuracy and 96.1% slot F1 when using BERT with their proposed stack-propagation architecture. This is comparable to our _target only with BERT_ scores in Table 2. On multilingual ATIS, Upadhyay et al. (2018) report slot F1 of 80.6% on Hindi and 78.9% on Turkish using bilingual training. Our multilingual BERT model achieves higher F1 by +6.1% on Hindi and +7.1% on Turkish.

## 6 Cross-Lingual Transfer

In this section, we compare the following methods for cross-lingual transfer where we only use the English training data and a small amount (fewshot) or no (zero-shot) training data from the target language:

* **No MT:** Train the models only on the English training data without machine translating them to the target language.
* **MT+fast-align:** Use MT systems to translate the English data to the target language and project the slot labels using word alignment from _fast-align_.5

Footnote 5: [https://github.com/clab/fast_align](https://github.com/clab/fast_align)
* **MT+TMP:** Use MT to translate the English data to the target language and the TranslateMatch-Project method Jain et al. (2019) to project the slot labels.
* **MT+soft-align:** Use MT to translate the English data to the target language and our soft-alignment method described in Section 4.3.

For all three MT-based approaches, we use AWS Translate for automatic translation and perform multilingual training on the English and machine translated data. We adopt the same setup as the previous section, except that we select the model at the last epoch as we assume no access to the development set from the target language in this setting. For the attention module in our soft-alignment model, we set the temperature \(\tau=0.1\).

### Zero-Shot Results

Table 3 shows the results on zero-shot cross-lingual transfer. First, we study the impact of using multilingual BERT for MT-based transfer approaches. For _MT+fast-align_, BERT boosts the performance over LSTM by large margins: 14-32% on intent accuracy and 29-61% on slot F1 on all languages except for Turkish - a dissimilar language to English. For both LSTM and BERT models, using _MT+fast-align_ brings significant improvements over their counterparts without MT on intent

\begin{table}
\begin{tabular}{l l r r r r r r r r r} \hline \hline
**Intent acc.** & & en & es & de & zh & ja & pt & fr & hi & tr \\ \hline \multirow{3}{*}{Target only} & LSTM & 96.08 & 93.04 & 94.02 & 92.50 & 91.18 & 92.70 & 94.71 & 84.46 & 81.12 \\  & BERT & **97.20** & **96.44** & **96.73** & **95.52** & 95.54 & **96.71** & **97.38** & 90.50 & 87.10 \\ \cline{1-1}  & LSTM & 95.45 & 94.09 & 95.05 & 93.42 & 92.90 & 94.02 & 94.80 & 87.79 & 85.43 \\ \cline{1-1}  & BERT & **97.20** & **96.77** & **96.86** & **95.54** & **96.44** & 96.48 & **97.24** & **92.70** & **92.20** \\ \hline \hline
**Slot F1** & & en & es & de & zh & ja & pt & fr & hi & tr \\ \hline \multirow{3}{*}{Target only} & LSTM & 94.71 & 75.89 & 91.44 & 90.84 & 88.80 & 88.43 & 85.93 & 74.93 & 64.43 \\  & BERT & 95.57 & 86.58 & **94.98** & **93.52** & 91.40 & 91.35 & 89.14 & 82.36 & 75.21 \\ \cline{1-1}  & LSTM & 94.75 & 84.11 & 92.00 & 90.76 & 88.55 & 88.79 & 87.96 & 77.34 & 77.25 \\ \cline{1-1}  & BERT & **95.90** & **87.95** & **95.00** & **93.67** & **92.04** & **91.96** & **90.39** & **86.73** & **86.04** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results on MultiATIS++ using full training data and the standard supervised objective averaged over 5 runs. The _Target only_ models are trained only on the target language training data. The _Multilingual_ models are trained on the concatenation of training data from all languages.

 

accuracy - improvements of 13-33% when using LSTM and 1-24% when using BERT. However, we observe different trends on slot F1 for different languages. For example, when using BERT, _MT+fastalign_ improves slot F1 by 20-44% on Chinese, Japanese, and Hindi over BERT without MT, but hurts by around 6% on French.6 This is possibly because that the mBERT representations of French are of high quality, which leads to relatively high slot F1 without MT, thus adding more noisy data via _MT+fast-align_ does more harm than good. This indicates that, while training directly on the target language data is beneficial especially for languages dissimilar to English, the noisy projection of the slot labels could also bring harm to the model.

Footnote 6: The difference is significant with \(p<0.05\).

Next, we compare our soft-alignment method with other MT-based approaches. Our method outperforms _MT+fast-align_ on five of the eight languages and is more robust across languages - it achieves consistent improvements over BERT fine-tuned without MT on both intent accuracy and slot F1, while _MT+fast-align_ leads to a degradation on French and Turkish. Furthermore, we compare our method with _MT+TMP_, a strong label projection baseline that combines MT and linguistic features to improve the quality of label projection. Results show that our method uses only half the training time as _MT+TMP_ (Table 4), while achieving competitive performance (Table 3): on intent accuracy, our method achieves on par or higher scores on all languages. On slot F1, our method performs on par or better than _MT+TMP_ on four of the eight languages and achieves close performance on the remaining languages.

Finally, these results emphasize the need to evaluate cross-lingual transfer methods on a diverse set of languages to fully illustrate the strengths and

\begin{table}
\begin{tabular}{l l r} \hline \hline  & \multicolumn{3}{c}{Time (mins)} \\ \hline No MT & LSTM & 18 \\  & BERT & 65 \\  & LSTM & 232 \\  & BERT & 317 \\  & LSTM & 626 \\  & BERT & 719 \\  & BERT & 352 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Total training time of each method on all languages for the zero-shot transfer experiments (including the time for machine translation, label projection, and model training). _MT+TMP_ requires a time-consuming label projection process and thus takes double the time as our soft-alignment method.

\begin{table}
\begin{tabular}{l c r r r r r r r r} \hline \hline
**Intent acc.** & es & de & zh & ja & pt & fr & hi & tr \\ \hline \multirow{3}{*}{No MT} & LSTM & 64.82 & 64.77 & 59.69 & 65.40 & 65.15 & 69.92 & 60.11 & 63.64 \\  & BERT & 96.35 & 95.27 & 86.27 & 79.42 & 94.96 & 95.92 & 80.96 & 69.59 \\  & LSTM & 95.36 & 94.02 & 92.70 & 77.96 & 94.20 & 94.76 & 88.71 & 87.72 \\  & BERT & **97.02** & **96.77** & **96.10** & **88.82** & 96.55 & 96.89 & **93.12** & **93.77** \\  & LSTM & 95.32 & 93.03 & 90.41 & 84.43 & 94.36 & 94.15 & 88.67 & 87.47 \\  & BERT & **97.00** & 96.01 & 95.16 & **88.51** & 96.46 & 97.04 & 92.41 & **93.74** \\
**MT+soft-align** & BERT & **97.20** & **96.66** & **95.99** & **88.33** & **96.78** & **97.49** & 92.81 & **93.71** \\ \hline
**Slot F1** & & es & de & zh & ja & pt & fr & hi & tr \\ \hline \multirow{3}{*}{No MT} & LSTM & 27.98 & 32.96 & 1.60 & 2.71 & 25.52 & 29.70 & 2.26 & 26.56 \\  & BERT & 74.98 & 82.61 & 62.27 & 35.75 & 74.05 & 75.71 & 31.21 & 23.75 \\  & LSTM & 76.30 & 83.83 & 78.61 & 70.23 & 76.28 & 64.37 & 60.02 & 21.53 \\  & BERT & 79.18 & 87.21 & 81.82 & 79.53 & 78.26 & 70.18 & 69.42 & 23.61 \\  & LSTM & 79.19 & 84.99 & 82.84 & 71.98 & 79.44 & 77.52 & 67.70 & 40.14 \\  & BERT & **83.98** & 87.54 & **85.05** & **82.60** & **81.73** & **79.80** & 77.24 & 44.80 \\
**MT+soft-align** & BERT & 76.42 & **89.00** & 83.25 & 79.10 & 76.30 & **79.64** & **78.56** & **61.70** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Zero-shot results on MultiATIS++ averaged over 5 runs. The _No MT_ rows are models trained only on the English data. The _MT+fast-align_ rows correspond to models trained on the English and machine translated data with automatically projected slot labels using _fast-align_, and _MT+TMP_ correspond to Translate-Match-Project Jain et al. [20]. The _MT+soft-align_ row is the model trained on the English and machine translated data using our soft-alignment method.

 

weaknesses of each method and to avoid overly strong claims. In our experiments, we find that although _MT+TMP_ achieves higher slot F1 than our method on some languages, our method is more robust across languages and outperforms _MT+TMP_ by a large margin (+16.9%) on Turkish. To further explain the large gap, we measure the slot projection accuracy of _fast-align_ and _TMP_ compared against the annotated data. We find that _fast-align_ obtains extremely low projection accuracy (20%) on Turkish, and _TMP_ improves it to 39%, which is still low compared to that on other languages (above 70%). The low projection accuracy on Turkish can be attributed to the morphological difference between Turkish and English, which makes it difficult to hard-align each Turkish word to a single English word.

### Learning Curves

Figure 3 shows the few-shot transfer results where we add a small amount of labeled data from the target language.7 We select French as a similar language to English, and Chinese as a dissimilar language. We find that _BERT+NoMT_ obtains promising results in the few-shot setting with only several hundred training examples in the target language - it achieves comparable or even higher scores than the best MT-based approach on French, but still lags behind on Chinese by around 5% on intent accuracy. In addition, results

\begin{table}
\begin{tabular}{l l} \hline \hline
**Intent acc.** & average \\ \hline MT+soft-align & 94.87 \\ w/o reconstruction loss & 94.95 \\ w/o joint training on source & 94.64 \\ \hline
**Slot F1** & average \\ \hline MT+soft-align & 80.00 \\ w/o reconstruction loss & 76.42 \\ w/o joint training on source & 71.62 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Ablation results for zero-shot transfer learning. Scores are averaged over Spanish, German, Chinese, Japanese, Portuguese, French, Hindi, and Turkish. Ablating the reconstruction loss or joint training on the source language data using supervised objective leads to a major drop on slot F1, while the impact on intent accuracy is small.

Figure 3: Results for cross-lingual transfer given various sizes of the labeled data from the target language in addition to the English and machine translated data used in zero-shot experiments. We report scores averaged over 5 runs using target training data selected with different random seeds. Results show the effectiveness of our soft-alignment method as compared to the best projection-based method (_BERT+MT+TMP_) in few-shot transfer where only a few hundred target examples are available.

 

show the effectiveness of our method in few-shot transfer: our method obtains comparable (on French) or higher (on Chinese) slot F1 than the best projection-based method (_BERT+MT+TMP_) given a few hundred target examples, which suggests that, even with a less noisy label projector, the projection errors may still hinder the model from best exploiting the small amount of target language data especially on languages dissimilar to English.

### Ablation Study

We evaluate the impact of different components in our soft-alignment model. Table 5 shows that both the reconstruction loss and joint training on the source data using supervised objective are beneficial - slot F1 drops by 3.6% when ablating the reconstruction loss and by 8.4% when ablating joint training on the source, while both have little impact on intent accuracy.

## 7 Conclusion

We introduce MultiATIS++, a multilingual NLU corpus that extends the Multilingual ATIS corpus to nine languages across four language families. We use our corpus to evaluate various cross-lingual transfer methods including the use of multilingual BERT encoder, machine translation, and label projection. We further introduce a novel end-to-end model for joint slot label alignment and recognition that requires no external label projection. Experiments show that multilingual BERT brings substantial improvements on multilingual training and cross-lingual transfer tasks. Furthermore, our model outperforms the simple projection baseline using fast-align on most languages, and achieves competitive performance to the state-of-the-art label projection approach with only half of the training time. We release our MultiATIS++ corpus to facilitate future research on cross-lingual NLU to bridge the gap between cross-lingual transfer and supervised methods.

## Acknowledgments

We thank Tamim Swaid and Olga Pospelova at the Amazon AI Data team for collecting the data, the anonymous reviewers, Eleftheria Briakou, Marine Carpuat, Pranav Goel, Pedro Rodriguez, the CLIP lab at UMD, and the Amazon AI team for their valuable feedback.

## References

* Chen et al. (2019) Xilun Chen, Ahmed Hassan Awadallah, Hany Hassan, Wei Wang, and Claire Cardie. 2019. Multi-source cross-lingual model transfer: Learning what to share. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3098-3112. Association for Computational Linguistics.
* Chen et al. (2018) Xilinu Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. 2018. Adversarial deep averaging networks for cross-lingual sentiment classification. _Transactions of the Association for Computational Linguistics_, 6:557-570.
* Conneau et al. (2020) Alexis Conneau, Kartiaky Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186. Association for Computational Linguistics.
* Ehrmann et al. (2011) Maud Ehrmann, Marco Turchi, and Ralf Steinberger. 2011. Building a multilingual named entity-annotated corpus using annotation projection. In _Proceedings of the International Conference Recent Advances in Natural Language Processing 2011_, pages 118-124. Association for Computational Linguistics.
* Eriguchi et al. (2018) Akiko Eriguchi, Melvin Johnson, Orhan Firat, Hideto Kazawa, and Wolfgang Macherey. 2018. Zero-shot cross-lingual classification using multilingual neural machine translation. _CoRR_, abs/1809.04686.
* Feng et al. (2004) Donghui Feng, Yajuan Lv, and Ming Zhou. 2004. A new approach for English-Chinese named entity alignment. In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_, pages 372-379. Association for Computational Linguistics.
* Guo et al. (2018) Chih-Wen Gao, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung-Chieh Chen, Kong-Wei Hsu, and Yun-Nung Chen. 2018. Slot-gated modeling for joint slot filling and intent prediction. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 753-757. Association for Computational Linguistics 

Xiaodong He, Li Deng, Dilek Hakkani-Tur, and Gokhan Tur. 2013. Multi-style adaptive training for robust cross-lingual spoken language understanding. In _2013 IEEE International Conference on Acoustics, Speech and Signal Processing_, pages 8342-8346. IEEE.
* Jain et al. (2019) Alankar Jain, Bhargavi Paranjape, and Zachary C Lipton. 2019. Entity projection via machine translation for cross-lingual NER. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 1083-1092. Association for Computational Linguistics.
* Kingma and Ba (2015) Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _International Conference on Learning Representations (ICLR)_.
* Lin et al. (2018) Ying Lin, Shengqi Yang, Veselin Stoyanov, and Heng Ji. 2018. A multi-lingual multi-task architecture for low-resource sequence labeling. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 799-809. Association for Computational Linguistics.
* Liu and Lane (2016) Bing Liu and Ian Lane. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling. _Interspeech 2016_, pages 685-689.
* Mayhew et al. (2017) Stephen Mayhew, Chen-Tse Tsai, and Dan Roth. 2017. Cheap translation for cross-lingual named entity recognition. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2536-2545. Association for Computational Linguistics.
* Ni et al. (2017) Jian Ni, Georgiana Dinu, and Radu Florian. 2017. Weakly supervised cross-lingual named entity recognition via effective annotation and representation projection. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1470-1480. Association for Computational Linguistics.
* Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4996-5001, Florence, Italy. Association for Computational Linguistics.
* Plank and Agic (2018) Barbara Plank and Zeljko Agic. 2018. Distant supervision from disparate sources for low-resource part-of-speech tagging. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 614-620. Association for Computational Linguistics.
* Price (1990) P. J. Price. 1990. Evaluation of spoken language systems: the ATIS domain. In _Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990_.
* Qin et al. (2019) Libo Qin, Wanxiang Che, Yangming Li, Haoyang Wen, and Ting Liu. 2019. A stack-propagation framework with token-level intent detection for spoken language understanding. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2078-2087, Hong Kong, China. Association for Computational Linguistics.
* Samy et al. (2005) Doaa Samy, Antonio Moreno, and Jose M Guirao. 2005. A proposal for an arabic named entity tagger leveraging a parallel corpus. In _International Conference RANLP, Borovets, Bulgaria_, pages 459-465.
* Schuster and Paliwal (1997) Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. _IEEE Transactions on Signal Processing_, 45(11):2673-2681.
* Schuster et al. (2019) Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019. Cross-lingual transfer learning for multilingual task oriented dialog. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 3795-3805. Association for Computational Linguistics.
* Shah et al. (2010) Rushin Shah, Bo Lin, Anatole Gershman, and Robert Frederking. 2010. Synergy: a named entity recognition system for resource-scarce languages such as swahili using online machine translation. In _Proceedings of the Second Workshop on African Language Technology (AfLaT 2010)_, pages 21-26.
* Singla et al. (2018) Karan Singla, Dogan Can, and Shrikanth Narayanan. 2018. A multi-task approach to learning multilingual representations. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 214-220. Association for Computational Linguistics.
* Tackstrom et al. (2013) Oscar Tackstrom, Dipanjan Das, Slav Petrov, Ryan McDonald, and Joakim Nivre. 2013. Token and type constraints for cross-lingual part-of-speech tagging. _Transactions of the Association for Computational Linguistics_, 1:1-12.
* Tsai et al. (2016) Chen-Tse Tsai, Stephen Mayhew, and Dan Roth. 2016. Cross-lingual named entity recognition via wikification. In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_, pages 219-228. Association for Computational Linguistics.
* Tur et al. (2010) Gokhan Tur, Dilek Hakkani-Tur, and Larry Heck. 2010. What is left to be understood in ATIS? In _2010 IEEE Spoken Language Technology Workshop_, pages 19-24. IEEE.
* Upadhyay et al. (2018) Shyam Upadhyay, Manaal Faruqui, Gokhan Tur, Dilek Hakkani-Tur, and Larry Heck. 2018. (almost) zero-shot cross-lingual spoken language understanding. In _Proceedings of the IEEE ICASSP_ 

Xiaojun Wan. 2009. Co-training for cross-lingual sentiment classification. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 235-243. Association for Computational Linguistics.
* Xie et al. (2018) Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A. Smith, and Jaime Carbonell. 2018. Neural cross-lingual named entity recognition with minimal resources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 369-379. Association for Computational Linguistics.
* Yang et al. (2017) Zhilin Yang, Ruslan Salakhutdinov, and William W. Cohen. 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. In International Conference on Learning Representations.
* Yarowsky et al. (2001) David Yarowsky, Grace Ngai, and Richard Wientowski. 2001. Inducing multilingual text analysis tools via robust projection across aligned corpora. In Proceedings of the First International Conference on Human Language Technology Research.
* Yu et al. (2018) Katherine Yu, Haoran Li, and Barlas Oguz. 2018. Multilingual seq2seq training with similarity loss for cross-lingual document classification. In Proceedings of The Third Workshop on Representation Learning for NLP, pp. 175-179. Association for Computational Linguistics.
* Zirikly and Hagiwara (2015) Ayah Zirikly and Masato Hagiwara. 2015. Cross-lingual transfer of named entity recognizers without parallel corpora. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 390-396. Association for Computational Linguistics.

 

Model and Training Details

We train all models on 4 NVIDIA V100 Tensor Core GPUs. For both the multilingual NLU and cross-lingual transfer experiments, we train the models using the Adam optimizer Kingma and Ba (2015) for 20 epochs. We set the initial learning rate to \(10^{-3}\) for the LSTM model and \(10^{-5}\) for the BERT model. The LSTM model has embeddings of size 256 and 128 hidden units. We add dropout of 0.1 to the embeddings and encoder hidden states. Both LSTM and BERT models use the WordPiece tokenization model from Devlin et al. (2019). Table 6 shows the total number of parameters for each model, and Table 7 shows the training time used for each method in the multilingual NLU experiments.

## Appendix B Evaluation

We evaluate all NLU models using intent accuracy and slot F1. Before computing slot F1, we merge all slots that are segmented during preprocessing to match with the original slot segments. We use the script conlleval.pl8 to compute slot F1.

Footnote 8: [http://deeplearning.net/tutorial/code/conlleval.pl](http://deeplearning.net/tutorial/code/conlleval.pl)

## Appendix C Validation Performance

We report the average intent accuracy and slot F1 on the development sets in the multilingual NLU experiments in Table 8. For zero-shot experiments, we select the model at the last epoch as we assume no access to the development sets from the target language in this setting.