Transfer-Free Data-Efficient Multilingual Slot Labeling
Authors: Evgeniia Razumovskaia
Ivan Vulić
Anna Korhonen
Venue: EMNLP 2023 Main (2023)
Keywords: Dialogue NLU
Multilingual
Slot Labeling
Data Efficient

Abstract: Slot labeling (SL) is a core component of task-oriented dialogue (TOD) systems, where slots and corresponding values are usually language-, task- and domain-specific. Therefore, extending the system to any new language-domain-task configuration requires (re)running an expensive and resource-intensive data annotation process. To mitigate the inherent data scarcity issue, current research on multilingual ToD assumes that sufficient English-language annotated data are always available for particular tasks and domains, and thus operates in a standard cross-lingual transfer setup. In this work, we depart from this often unrealistic assumption. We examine challenging scenarios where such transfer-enabling English annotated data cannot be guaranteed, and focus on bootstrapping multilingual data-efficient slot labelers in transfer-free scenarios directly in the target languages without any English-ready data. We propose a two-stage slot labeling approach (termed TWOSL) which transforms standard multilingual sentence encoders into effective slot labelers. In Stage 1, relying on SL-adapted contrastive learning with only a handful of SL-annotated examples, we turn sentence encoders into task-specific span encoders. In Stage 2, we recast SL from a token classification into a simpler, less data-intensive span classification task.
Our results on two standard multilingual TOD datasets and across diverse languages confirm the effectiveness and robustness of TWOSL. It is especially effective for the most challenging transfer-free few-shot setups, paving the way for quick and data-efficient bootstrapping of multilingual slot labelers for TOD.

Transfer-Free Data-Efficient Multilingual Slot Labeling
Evgeniia Razumovskaia Ivan Vulié Anna Korhonen
Language Technology Lab, University of Cambridge (er563, iv250, alk23)@cam. ac. uk
Abstract
to Milan for tomorrow", the airline booking sys- tem should match the values "Chicago", "Milan", and "tomorrow" " with the slots departure_cine arrival_city, and date, respectively.
Slot labeling (SI.) is core component of task- oriented dialogue (ToD) systems, where slots and corresponding values are usually language- task- and domain-specific. Therefore, extend- ing the system to any new language-domain- task configuration requires (re)running an ex- pensive and issturce-nttessive data annotation process. To mitigate the inherent data scarcity issue, current research on multilingual ToD as- sumes that sufficient English- languugh anno- tated data are always available for particular tasks and domains, and thus operates in stan- dard cross- loooull transfer setup. In this work, we depart from this often unrealistic assump- tion. We examine challenging scenarios where such transfer- eeaallingg English annotated data cannot be guaranteed, and focus on bootstrap- ping multilingual data-efficient slot labelers in transfer-free scenarios directly in the target lan- guages without any English-ready data. We pro- pose a two-stage slot labeling approach (termed TWOSL) which transforms standard multilin- gual sentence encoders into effective slot la- belers. In Stage 1, relying on SL-adapted con- trastive learning with only a handful of SL- annotated examples, into task-specific span encoders. In Stage 2, we recast SL from a token classification into a sim- pler, less data-intensive span classification task. Our results on two standard multilingual ToD datasets and across diverse languages confirm the effectiveness and robustness of TWOSL. It is especially effective for the most challenging transfer-fi free few-shot setups. paving the way for quick and data efficient bootstrapping of multilingual slot labelers for ToD.
Building ToD systems which support new do- mains, tasks, and also languages is challenging, ex- pensive and come-consuming: it requires large anno- tated datasets for model training and development, where such data scarce for many domains, tasks, and most importantly languages (Razumovskaia et al., 2022a). The current approach mitigate the issue the standard cross-lingual transfer. The main 'transfer' assumption is that a suitable large English annotated dataset is always available for a particular task and domain: (i) the systems are then trained on the English data and then directly de- ployed to the target language (i.e., zero-shot trans- fer), (ii) further adapted to the target language relying on small set of target language examples (Xu al., 2020; Razumovskaia al., 2022b) which are combined with the large English dataset (i.e.. few-shot transfer). However, this assumption might often be unrealistic in the context of ToD due to large number of potential tasks and domains that should be supported by ToD systems (Casanueva et al., 2022). Furthermore, the standard assumption implicitly grounds any progress of ToD in other languages the English language, hindering any system construction initiatives focused directly on the target languages (Ruder et al., 2022).
turn sentence encoders
Therefore, this work we depart from this often unrealistic assumption, and propose to focus on transfer-free scenarios for SL instead. Here, the system should learn the task in particular domain directly from limited resources in the target lan- guage, assuming that any English data cannot be guaranteed. This setup naturally calls for construct- ing versatile multilingual data-efficient method that leverages searce annotated data as effectively as possible and should thus be especially applicable to low-resource languages (Joshi et al., 2020).
1 Introduction and Motivation
Slot labeling (SL) is crucial natural language un- derstanding (NLU) component for task-oriented dialogue (ToD) systems (Tur and De Mori, 2011) 1). It aims to identify slot values in a user utterance and fill the slots with the identified values. For instance, given the user utterance "Tickets from Chicago

---

Putting this challenging setup into focus, we thus propose novel two-stage slot-labeling approach, dubbed TWOSL. TWOSL recasts the SL task into a span classification task within its two respective stages. In Stage 1, a multilingual general-purpose sentence encoder is fine-tuned via contrastive learn- ing (CL), tailoring the CL objective towards SL- based span classification; the main assumption is that representations of phrases with the same slot type should obtain similar representations in the specialised encoder space. CL. allows for more ef- ficient use of scarce training resources (Fang et al., 2020; Su et al., 2021; Rethmeier and Augenstein, 2021). Foreshadowing, it manages to separate the now-specialised SL.-based encoder space into slot- type specialised subspaces, as illustrated later in Figure 2. These SL-aware encodings more in- terpretable and allow for easier classification into slot types in Stage 2, using simple MLP classifiers.
level semantic "knowledge" (Liu et al., 2021; Vulić et al., 2022). In general, we hope that this work will inspire and pave the way for further research in the challenging transfer-free few-shot setups for multilingual SL as well as for other NLP tasks. The code for TWOSL will be available online.
2 Related Work
Multilingual Slot Labeling. Recently, the SL task in multilingual contexts has largely benefited from the development of multilingually pretrained lan- guage models (PLMs) such as mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020). These models are typically used for zero-shot or few-shot multilingual transfer (Xu et al., 2020; Krone et al., 2020; Cattan et al., 2021). Further. the representational power of the large multilin- gual PLMs for cross-lingual transfer has been fur- ther refined through adversarial training with latent variables (Liu al., 2019) and multitask training (van der Goot et al., 2021).
We evaluate TWOSL in transfer-free scenarios on two standard multilingual SL benchmarks: Multi- ATIS++ (Xu et al., 2020) and xSID (van der Goot et al., 2021), which in combination cover 13 ty- pologically diverse target languages. Our results indicate that TWOSL yields large and consistent improvements 1) across different languages, 2) in different training set size setups, and also 3) with different input multilingual encoders. The gains especially large in extremely low-resource setups. For instance, on MultiATIS++, with only 200 train- ing examples in the target languages, we observe an improvement in average F1 scores from 49.1 with- out the use of TWOSL to 66.8 with TWOSL, relying on the same multilingual sentence encoder. Similar gains were observed on xSID, and also with other training set sizes. We also report large gains over fine-tuning XLM-R for SL framed as the standard token classification task (e-g., from 50.6 66.8 on Multi ATIS++ and from 43.0 to 52.6 on xSID with 200 examples), validating our decision to recast the task in TWOSL as a span classification task.
Other effective methods for cross-lingual trans- fer are mansslatio-bbsedd where either the training data in the source language is translated into the target language or the evaluation data is translated into the source (translate-trait and translate-tesı, respectively; Schuster et al. (2019); Razumovskaia et al. (2022a)). The issues with these methods for SL are twofold. First, the translations might be of lower quality for low-resource languages or any lan- guage pair where large parallel datasets are lacking, Second, they involve the crucial label-projection step. which aligns the words in the translated ut- terances with the words in the source language. Therefore, (i) applying translation-basel methods to sequence labeling tasks such as SL is not straight- forward (Ponti et al., 2021), (ii) it increases the number of potential accumulated errors (Fei et al.. 2020), and (iii) requires powerful word alignment tools (Dou and Neubig, 2021).
Several methods were proposed to mitigate the issues arising from the label-projection step. Xu et al. (2020) propose to jointly train slot tagging and alignment algorithms. Gritta and Iacobacci (2021) and Gritta et al. (2022) fine-tune the mod- els for post-alignment, i.c., explicitly aligning the source and translated data for better cross-lingual dialogue NLU. These approaches still rely on the availability of parallel corpora which are not guar- anteed for low- fow-resource languages. Thus, alterna- tive approaches using code-switching (Qin et al.,
In summary, the results suggest the benefits of TWOSL for transfer-free multilingual slot labeling, especially in the low-resource setups when only several dozen examples are available in the target language: this holds promise to quicken SL de- velopment cycles in future work. The results also demonstrate that multilingual sentence encoders can be transformed into effective span encoders us- ing contrastive learning with a handful of examples. The CL procedure in TWOSL exposes their phrase-

---

Slgge
chitton
Tewtuee Mullingiaa Se Manoa Encocar
B Skrtilo Sixt Clossicaiin
Malh-raae Stet Type Class Roaton
wort from Shitagg Menday
wove from Chllage MACI
Morcay
ereoding
pars reguivve goins
Contrasivv Learning
Serterce Enooder
O London to
Tichea fuan ]/ASS( to Milan
Lordbnn
Milhingual Fartents
Figure 1: Illustration of the proposed TWOSL framework which turns general-purposs multilingual sentence encoders into efficient slot labelers via two stages. Stage 1: contrastive learning tailored towards encoding sub-sentence spans, Stage 2: slot classification in two steps, binary slot-span identification/fiteriin (Step 1. aiming to answer the question 'Is this span a value for any of the slot types?') and multi-class span-type classification (Step 2, aiming to answer the question 'What class is this span associated with?"). Ablation variants include: a) using off-the-shelf multilingual sentence encoders in Stage 2 without their CL-based fine-tuning in Stage 1; b) directly classifying slot spans without the binary filtering step (i.e., without Step in Stage 2).
2020; Krishnan et al., 2021) were proposed. All of the above methods assume the availability of an 'aid' for cross-lingual transfer such transla- tion model or bilingual lexicon; more importantly, they assume the existence of readily available task- annotated data in the source language.
scheme for sequence labeling (e.g,, the labels are O, B-SL,, I-ST1..... I-ST.N,). We also assume that M SL-annotated sentences are available in the target language as the only supervision signal.
The full two-stage TWOSL framework is illus- trated Figure 1, and we describe its two stages in what follows.
Data-Efficient Methods for Slot Labeling, One approach to improve few-shot generalisation in ToD systems pretrain the models in a way that is specifically tailored to conversational tasks. For instance, Con (Henderson and Vulié, 2021) fine-tunes only a subset of decoding layers on conversational data. QANLU (Namazifar et al., 2021) and QASL (Fuisz et al., 2022) use question- answering for data-efficient slot labeling in mono- lingual English-only setups by answering questions based on reduced training data.
3.1 Stage 1: Contrastive Learning for Span Classification
Stage 1 has been inspired by contrastive learning regimes which were proven especially effective in few-shot setups for cross-domain (Su et al., 2022: Meng et al., 2022; Ujiie et al., 2021) and cross- lingual transfer (Wang et al., 2021; Chen et al., 2022), well for task specialisation of general- purpose sentence encoders and PLMs for intent detection (Mehri and Eric, 2021: Vulić al., 2021). To the best of our knowledge, CL. has not been coupled with the ToD SL task before.
In addition, methods for zero-shot cross-lingual contrastive learning have been developed (Gritta et al., 2022; Gritta and lacobacci, 2021) which al- low for efficient use of available annotated data. Further, Qin et al. (2022) and Liang et al. (2022) code-switched examples to improve perfor- mance intent classification and slot labeling in zero-shot setting by training on code-switched examples on slot-value, sentence and word levels. Unlike prior work, TWOSL focuses on adapting multilingual sentence encoders to the SL task in transfer-free low-data setups.
Input Data Format for CL. First, we need reformat the input sentences into the format suit- able for CL. Given M annotated sentences, we transform each of them into M triples of the fol- lowing format: (smask. sp, L). Here, (i) smask is the original sentence s, but with word tokens comprising a particular slot value masked from the sentence; (ii) sp is that slot value span masked from the original sentence; (iii) L is the actual slot type associated with the span sp. Note that L can be one of the N₂ slot types from the slot set S or a special None value denoting that sp does not capture a proper slot value. One exam-
3 Methodology
Preliminaries. We assume the set of N, slot types S (SL,...., SLy,. } associated to an SL task. Each word token in the input sentence/sequence s wi, W2 ..., Wn should be assigned slot la- bel yi. where we assume a standard BIO tagging
For simplicity and clarity, we will focus on the standard BIO seheme, while the method is fully operational with other tagging schemes as well.

---

ple of such a triple is (Smost=Ich benötige einen Flug von [MASK] [MASK] nach Chicago, sp=New York, I=departure_citteee cytt). In another example, (Smask= mir die Preise von Boston nach Denver, sp=Zeige, -. Note that sp can span one or more words as in the examples above, which effectively means masking one or more words from the original sentence. We limit the length of sp to the maximum of maTspp consecutive words.
distance, formulated the cosine distance, be- tween representations of examples forming the pos- itive pairs, and 2) increase the distance between representations of examples forming the negative pairs. The online version of the loss, which typi- cally outperforms its standard variant (Reimers and Gurevych, 2019), focuses only on hard positive and hard negative examples: the distance is higher than the margin m for positive examples, and below for negative examples.
Positive and Negative Pairs for CL. The main idea behind CL in Stage is to adapt the input (multilingual) sentence encoder to the span classi- fication task by "teaching' it to encode sentences carrying the same slot types closer in its CL-rrefined semantic space. The pair p=(Smask: sp) is extracted from the corresponding tuple, and the encoding of the pair is a concatenation of encodings of smask and sp encoded separately by the sentence encoder. CL proceeds in a standard fashion relying on sets of positive and negative CL pairs. A positive pair (actually, 'a pair of pairs') is one where two pairs P; and Pj contain the same label L in their correspond- ing tuple. but only if L f None. A negative pair is one where two pairs p: and Py contain different labels L, and Lj in their tuples, but at least one of the labels not None.
3.2 Stage 2: Span Identification and Classification
The aim of Stage 2 is to identify and label the slot spans, relying on the embeddings produced by the encoders fine-tuned in the preceding Stage 1. In order to identify the slot spans, we must consider every possible subspan of the input sentence, which might slow down inference. Therefore, to boost inference speed, we divide Stage 2 into two steps. In Step 1, we perform a simple binary classification, aiming to detected whether a certain span is a slot value for any slot type from S. Effectively, for the input pair (8masks sp) the binary classifier returns (i.c., 'sp is some slot value') or O. The 0-examples for training are all subspans of the sentences which not associated with any slot type from S.
Following prior CL work (Vulié et al., 2021), each positive pair (p:. ) is associated with 2K negative pairs, where we randomly sample K neg- atives associated with pt and K negatives for Pj- Finally, for the special and most efficient CL setup where the ratio of positive and negative pairs is 1: 1, we first randomly sample the item from the positive pair (p.,p) Pý), and then randomly sample a single negative for the sampled p; or Pj-
Step 2 is multi-class span classification task, where we aim to predict the actual slot type from S for the input pair (Smask, sp). The binary filtering Step I allows us to remove all input pairs for which the Step I prediction is O, and we thus assign slot types only for the 1-predictions from Step 1. Put simply, Step I predicts if span covers any proper slot value, while Step 2 maps the slot value the actual slot type. We can directly proceed with Step without Step 1, but the training data then also has to contain all the examples with spans where L=None, see Figure again.
Online Contrastive Loss. Fine-tuning the input sentence encoder with the positive and negative pairs proceeds via a standard online contrustive loss. More formally:
The classifiers in both steps are implemented simple multi-layes perceptrons (MLP), and the in- put representation in both steps is the concatenation of the respective encodings for Smask and sp.
Countr (St, Sy. f) = 1y, v,lll((s))) f(s)))
+ +1 # yz] ma m [I[(s;) [+
where and 8j are two examples with labels yı and Uj- ff is the encoding function and is hy- perparameter defining the margin between samples of different classes.
Experimental Setup
Training Setup and Data. The standard few-shot setup multilingual contexts (Razumovskaia al., 2022a; Xu et al., 2020) assumes availability of a large annotated task-specific dataset in English, and a handful of labeled examples in the target language. However, as discussed in $1, this as-
Similarly to the original contrastive loss (Chopra et al., 2005), it aims at 1) reducing the semantic
*Put simply, we disallow pulling closer pairs which do not convey any useful slot-related semantic information.

---

Dataset
Domains
Slots
Languages
Examples per Lang
models are used in their 'base' variants, with 12 hidden-layers and encoding the sequences into 768- dimensional vectors. This means that the actual encodings of (Smask sp) pairs, which are fed to MLPs in Stage 2, are 1.536-dimensional; see $3.
MulriATIS++
ft, tr, hi
5,871
ar, da, de, de-st, id.
xSID
7
33
800
iL. ja. kk, st,
Table I: Multilingual SL. datasets in the experiments.
Hy perparaamees and Optimisation. We rely on sentence- transsocrsrssssormerss (SBERT) library (Reimers and Gurevych, 2019, 2020) for model checkpoints and contrastive learning in Stage 1. The models are fine-tuned for 10 epochs with batch size of 32 using the default hyperparameters in SBERT: c.g., the margin in the contrastive loss is fixed to m = 0.5. max is fixed to 5 as even the longest slot values very rarely exceed that span length. Unless stated otherwise, K- 1, that is, the ratio of positive-to- negative examples is 2, see $3.1.
sumption might not always hold. That is, the En- glish data might not be available for many target- language specific domains, especially since the an- notation for the SI, task is also considered more complex than for the intent detection task (van der Goot et al.. 2021; Xu et al., 2020; FitzGerald et al.. 2022). We thus focus on training and evaluation in these challenging transfer-free setups.
We run experiments on two standard multilin- gual SL datasets, simulating the transfer-free free se- tups: MultiATIS+ (Xu et al., 2020) and xSID (van der Goot et al., 2021). Their data statistics are provided in Table 1, with language codes in Ap- pendix A. For low-resource scenarios, we randomly sample M annotated sentences from the full train- ing data. Since xSID was originally intended only for testing zero-shot cross- -onsull transfer, we use its limited dev set for (sampling) training instances. A current limitation of TWOSL is that it leans on whitespace-based word token boundaries in the sentences: therefore, in this work we focus on subset of languages with that property, leaving fur- ther adaptation to other languages for future work.
In Stage 2. we train binary and multi-class MLPs with the following number of hidden layers and their size, respectively: [2,500, 1,500J and [3,600, 2,400, 800J, and ReLU as the non-linear activation. The Step binary classifier is trained for 30 epochs, while the Step 2 MLP is trained for 100 epochs. The goal in Step 1 is to ensure high recall (i.e., to avoid too aggressive filtering), which is why we opt for the earlier stopping. As a baseline, we fine-tune XLM-R for the token classification task, as the standard SL. task format (Xu et al., 2020; Razumovskaia et al., 2022b). Detailed training hyperparameters are provided in Appendix B. All results are averages across 5 random seeds.
Input Sentence Encoders. We experiment both with multilingual sentence encoders as well as gen- eral multilingual PLMs in order to (i) demonstrate the effectiveness of TWOSL irrespective of the un- derlying encoder, and to (ii) study the effect of pretraining task on the final performance. 1) XLM- R (Conneau et al., 2020) is multilingual PLM, pretrained with a large multilingual dataset in 100 languages via masked language modeling. 2) Mul- tilingual mpner (Song et al., 2020) is pretrained for paraphrase identification over 50 languages; the model was specifically pretrained in a contrastive fashion to effectively encode sentences. 3) We also run a subset of experiments with another state- of-the-art multilingual sentence encoder, LaBSE (Feng et al., 2022), to further verify that TWOSL be disentangled from the actual e All
Evaluation Metric. For direct comparability with standard token classification approaches we rely on token-level micro-Fi as the evaluation metric. For TWOSL this necessitates the reconstruction of the BlO-lleeled sequence Y from the predictions for the (mmask: sp, Lprea) tuples. For every sentence we first identify all the tuples (Smask- sp, Lpred) associated with s such that the predicted slot type Tpprdd # None. In the positions of sp are filled with B. complemented with the corresponding number |,yred if the length of sp > 1. Following that, the rest of the positions are set to the O label.
Results and Discussion
Before experimenting in the planned multilingual context, evaluate model in English, based on the standard ATIS dataset. The models are trained with the hyper-parameters described in $4. For English, we use LaBSE (Feng et al., 2022) as a se- quence encoder as it has demonstrated state-of-the- art results in prior experiments in dialogue-specific
"Prior work has demonstrated effectiiennes of models pre- trained for span encoding in few-shot settings (Henderson and Vulié, 2021: Coope et al., 2020). However, they are not directly comparahle with TwOSI. as they are hased on Finglssh- only encoders and are not publicly available.

---

AR
DE
DA
DE-ST
ID
KK
NL
SR
TR
AVG
50 training examples
XLM-R
2.2
10.4
1.3
12.5
0.7
5.5
4.2
XT.M-R-Sen Wontiw w/o CT.
12.7
23.9
22.5
27.2
23.3
26.9
20.1
29.5
22.7
25.7
23.5
XLM-R-Sea w/ CL
40.2
49.5
41.2
45.2
46.7
49.2
43.1
36.1
41.5
44.7
43.7
mpnet w/o CL
23.9
25.1
26.8
21.8
28.5
26.3
22.6
29.7
23.9
24.5
25.3
mppee w/ CL.
36.8
40.4
39.2
39.1
43.4
42.2
33.5
38.8
39.7
42.8
39.6
100 training examples
XLM-R
33.0
37.1
36.0
32.5
39.6
37.4
31.1
37.7
34.4
37.7
35.6
XLM R-SeenWooCC.
28.
33.2
30.9
29.2
30.6
36.3
29,4
4T.5
27D
34.2
$2.0
XI.M-R-Sem w/ CL.
39.0
46.2
42.9
45.0
51.4
36.4
41.6
52.1
35.7
50.4
44.1
mpnet w/o CL
34.3
37.1
36.3
31.2
37.6
38.6
30.0
37.1
33.7
32.8
34.9
mppet w/ CL
43.5
51.1
44.2
44.1
52.6
47.2
40.4
48.3
46.5
51.1
46.9
training examples
XLM-R
39.5
45.2
43.1
41.3
47.8
44.8
37.5
44.0
42.0
45.1
43.0
mpnet w/o CL
41.3
44.9
44.9
42.2
46.5
46.5
37.8
46.3
41.7
41.2
43.3
mpnet w/ CL
48.2
55.8
50.3
52.1
59.0
55.2
46.1
53.8
52.9
52.5
52.6
Table 2: Results xSID's test set using subsample of 50, 100, and 200 examples from its validation portion for training, with no English training examples. Micro P1 scores are reported. XLM-R refers to using the XLMM- PLMM for the standard token classification fine-tuning for SL. XLM-R-Sed denotes using XLM-R directly as a sentence encoder in the same fashion as mpnet. We provide standard deviation for results with mpnet as the sentence encoder in Appendix D, demonstrating statistical significance the improvements provided by contrastive learning.
Ireiaciity BXCE
TD
depart data day 3528
BITH SEA 1i2a
Langune
XLM
reretees CL
opoal
100
250
302
450
200
000
303
850
(a) before CL
(b) after CL
Figure 3: Slot Fi scores on MultiATIS++ across dif- ferent languages and data setups. The exact numerical scores are available in Appendix C.
Figure t-SNE plots (van der Maaten and Hinton, 2012) for annotated German examples from Multi- ATIS++'s test set. We show the examples for 8 slot types, demonstrating the effect of Contrastive Learning (CL) the final encodings. The encodings were cre- ated using (a) the original mppet encoder before Stage CL and (b) mpnet after CL-tuning in Stage 1. 800 annotated training examples were used for CI. K=1.
50
100
XLM-R
28.60
47.14
73.48
TWOSL: LaBSE w/o CL
41.88
$2.14
65.01
TWOSL: LaBSE w/ CL
64.92
76.17
82.51
Table 3: Results on English ATIS in few-shot setups with 50, 100, and 200 training examples.
tasks (Casanueva et al., 2022). The results in Ta- ble 3 demonstrate that TWOSL is effective in low- data English-only setups, with large gains atop such strong sentence encoder LaBSE, This indicates that TWOSL can be used bootstrap any project when only a handful of in-domain data points are available.
effect CL in Stage of TWOSL actually has on span encodings, and how it groups them over slot types. The aim of CL in Stage is exactly to make the representations associated with particular cluster into coherent groups and offer clearer sepa- ration of encodings across slot types. As proven previously for the intent detection task (Vulić al., 2021), such well-divided groups in the encoding space might facilitate learning classifiers on top of
Impact of Contrastive Learning in TWOSL on Slot Representations. Further, before delving deep into quantitative analyses, we investigate what

---

the fine-tuned encoders. As revealed by I-SNE plot (van der Maaten and Hinton, 2012) in Figure which shows the mpner-based encodings before and after Stage 1, exactly that effect is observed.
50
100
200
500
FR
DF
FR
DE
FR
DE
XLMER
82.4
74.5
83.4
75.5
87.9
78.3
89.8
85.1
impect win
56.3
55.2
62.9
57.7
63.8
71.2
70.2
TWTSEE imprct w/ CT.
82.1
75.2
7688
71.8
84.7
78.9
87.6
84.6
Namely, the non-tuned mpnet encoder already provides some separation of encodings into slot type-based clusters (Figure 2a), but the groupings are less clear and noisier. In contrast, in Figure 2b the examples are clustered tightly by slot type, with clear separations between different slot type-based clusters. This phenomenon is further corroborated by the automated Silhouettes cluster coherence met- ric (Rousseeuw, 1987): its values are 0.00 (before Stage 1) and a = 0.67 (after Stage 1). In sum, this qualitative analysis already suggests the effectiveness of CL for the creation of customised encodings that support the SL. task. We note that the same observations hold for all other languages well as for all other (data-leaner) training setups.
Table 4: Results on German and French in MultiATIS++ for standard few-shot setup where English annotated data is combined with a few target language examples.
the power of CL especially for such low-data se- tups. Finally, the results in Table 2 also hint that TWOSL works well with different encoders: it im- proves both mpnet and XLM-R as the underlying multilingual sentence encoders.
5.1 Ablations and Further Analyses
span classfficatoon-orinnte
TWOSL for Low-Resource Languages. TWOSL is proposed for extremely low-resource scenarios, when only a handful of examples in a target lan- guage are available. This is more likely to hap- pen for low-resource languages, which in addi- tion usually not represented enough in pre- training of PLMs (Conneau et al., 2020). To eval- uate TWOSL on low-resource languages, we ap- ply it to Named Entity Recognition (NER) task in several low-resource African languages. We focused on NER as i) similarly to slot labelling, NER is a sequence labelling task; ii) limited amno- tated data is available in low-resource languages for NER. Specifically, we use MasakhaNER (Adelani et al., 2021) dataset in our experiments focusing on Yoruba (yor) and Luo (luo). The experiments were conducted with 100 training examples, using XLM-R (base) as a sentence encoder. The rest of the setup was kept the same as described in $4.
Main Results. The results on xSID and Multi- ATIS+ are summarised Table 2 and Figure 3, respectively. The scores underline three impor- tant trends. First, TWOSL is much more powerful than the standard PLM-based (i.e., HLM-R-basel) token classification approach in very low-data se- tups, when only handful (e.g., 50-200) annotated examples in the target language are available as the only supervision. Second, running TWOSI. on top of general-purpose multilingual encoder such as mpnet yields large and consistent gains, and this is clearly visible across different target lan- guages in both datasets, and across different data setups. Third, while the token classification ap- proach is able to recover some performance gap as more annotated data become available (e.g., check Figure with 800 examples), TWOSL remains the approach in general.
TWOSI. has brought considerable improvements for both languages: from 14.46 to 45.38 F-1 for yor and from 16.56 to 36.4 36.46 F- for luo. These results further indicate the effectiveness of the method for fow-resource languages as well as for language- domain combinations with scarce resources.
A finer-grained inspection of the scores further reveals that for low-data setups, even when exactly the same model is used as the underlying encoder (i.e., XLM-R), TWOSI. offers large benefits over token classification with full XLM-R fine-tuning, see Table 2. The scores also suggest that the gap between TWOSI. and the baselines increases with the decrease of annotated data. The largest abso- lute and relative gains are in the 50-example setup. followed by the 100-example setup, etc.: e.g., on xSID, the average gain +9.5 Fl points with 200 training examples, while reaching up to +35.3 Fl points with 50 examples. This finding corroborates
TWOSL. in Standard Few-Shot Setups. TWOSL has been designed with a primary focus on transfer- free, extremely low-data setups. However, an- other natural question also concerns its applicabil- ity and effectiveness in the standard few-shot trans- fer setups, where we assume that large annotated dataset for the same task and domain is available in the source language: English. To this end, we run several experiments on MultiATIS++, with German and French as target languages, where we first fine-

---

with
0.383
Langunge
AR
10
Withott
0.500
ID
(Sa0)
KX
D1E
0.450
Num manmmple
906
0.423
100
50
01:775
AR
DA
DE
DE:ST
ID
IT
KK
NL
3R
TR
AVG
0.350
0.325
Number of oegative examples
Figure 4: Inference time per language on XSID with and without the binary filtering Step in Stage 2.
Figure 5: Impact of the number of negative examples per each positive example for the 50-example and 100- example setups. For clarity, the results are shown for subset of languages xSID, and the similar trends observed for other languages. Similar trends are observed on MultiATIS++, as shown in App. F.
tune the model on the full English training data, before running another fine-tuning step (Lauscher et al., 2020) on the M = 50, 100, 200, 500 exam- ples in the target language.
Overall, the results in Table 4 demonstrate that TWOSL maintains its competitive performance, although the token classification approach with XLM-R is stronger method overall in this setup. TWOSL is more competitive for French as the tar- get language. The importance of CL in Step I for TWOSL is pronounced also this more abundant data setup. We leave further exploration and adap- tation of TWOSL to transfer setups for future work.
50 examples
XLM-R-Seat CL
XI XLM---SAa w/ Cl.
40.2
41.2
46.7
49.2
43.1
41.5
44.7
43.8
opner wlo ipgeet CL
26.3
36.8
39.2
43.4
42.2
33.5
39.7
42
39,6
LaBSE with CT
21,2"
33.77
33.7
31.6
29.0
15.5
ECT
26.6
LaBSE w/ CI.
41.8
47.0
48.2
48.3
41.8
36.6
37.9
43.1
100 examples
XLM-R- Sent wio CL
28.1
30.9
30.6
36,3
29.4
27.0
34.2
30,9
XI M.LM-R-See Rent w/ Cl.
59.0
42.9
51.4
36.4
41.6
35.7
50.4
42.5
impect Cl
55.3"
36.3
37.6
300
mpnett w/ CL
43.5
44.2
52.6
41.2
40.4
46.5
$1.1
46.5
LaBSE wi CT.
40.7
37.9
Impact of Binary Filtering in Stage 2. In order to understand the benefit of Step (i.e., binary fil- tering) in Stage 2, compare the performance and inference time with and without that step. We focus the xSID dataset in the 200-example setup. The scores, summarised in Table 10 in Appendix E, demonstrate largely on-par performance between the two variants. The main benefit of using Step is thus its decrease of inference time, as reported in Figure 4, where inference was carried out on single NVIDIA Titan XP 12GB GPU. The fil- tering step. which relies on more compact and thus quicker classifier, greatly reduces the number of examples that have to undergo the final, more expensive slot type prediction (i.e., without filter- ing all the subspans of the user utterance must be processed) without harming the final performance.
LaBSE w/ CL
47.3
50.2
49.6
53.6
45.5
39.4
42.8
46.9
Table Results on xSID for sample of languages with different sentence encoders. XLM-R-Sae denotes using XIMMR as standard sentence encoder.
encoders. We compare XLM-R-Sent (i.e., XLM-R used a sentence encoder, mean- poonnnpooing subword embeddings), mpnet, and LaBSE on a representa- tive set of 7 target languages on xSID. In the major- ity of the experimental runs, LaBSE with TWOSL yields the highest absolute scores. This comes surprise LaBSE specifically customised to improve sentence encodings for low-resource languages and in low-resource setups (Feng et al., 2022). Interestingly, XLMMR performs the best in the 'lowest-data" 50-cxample sctup: we speculate this might be due to smaller model size, which makes it harder to overfit in extremely low-resource setups. Finally, the scores again verify the benefit of TWOSL when applied to any underlying encoder.
Different Multilingual Encoders. The results in Table 2 have already validated that TWOSL of- fers gains regardless of the chosen multilingual encoder (c.g.. XLM-R versus mpnet). However, the effectiveness of TWOSL in terms of absolute scores is naturally dependent on the underlying multilingual capabilities of the original multilin- gual encoder. We thus further analyse how the per- formance changes in the same setups with different
Number of Negative Pairs. The ratio of positive- to-negative examples, controlled by the hyperpa- rameter K, has a small impact on the overall per- formance, as shown in Figure We observe some slight performance gains when moving from 1 neg-

---

ative example to 2 (cf., the 50-example setup for AR and 100-example setup for ID xSID). In such cases, the increase in the number of negative pairs can act as data augmentation for the extreme low-resource scenarios. This hyper-parameter also impacts the trade-off between training time and the stability of results. With fewer negative examples, training is quicker, but the performance is less sta- ble: e.g., in the 50-example setup for German in MultiATIS++, the standard deviation is 7.45, a 2.36 and = 3.21 with 1,2 and 4 negatives per positive, respectively. Therefore, as stated in $4, we use the setup with negatives peaevves--er--pee per-positive in our experiments, indicating the good trade-off' between efficiency and stability.
scale fine-tuning of sentence encoders to turn them into universal span encoders which can then be used on sequence labelling tasks across languages and domains. TWOSL can be further extended to slot labelling with nested slots as well as to other 'non-Top non- sequence labelling tasks (e.g., NER) for which evaluation data exists for truly low-resource languages: c.g., on African languages (Adelani et al., 2021).
Limitations
TWOSI. relies on whitespace-basee word bound- aries. Thus, it only applicable languages which use spaces as word boundaries. We plan to extend and adapt the method to other languages, without this property, in our subsequent work. Additionally, the approach has been only tested on the languages which the large multilingual PLMs have seen dur- ing their pretraining. We plan to adapt and test the same approach unseen languages in the future.
6 Conclusion and Future Work
We proposed TWOSL, a two-stage slot labeling ap- proach which turns multilingual sentence encoders into slot labelers for lask-oriened dialogue (ToD), which was proven especially effective for slot label- ing in low-resource setups and languages. TWOSL was developed with the focus on transfer-free few- shot multilingual setups, where sufficient English- language annotated data are not readily available to enable standard cross-lingual transfer approaches. other words, the method has been created for bootstrapping a slot labeling system in a new lan- guage and/or domain when only a small set of an- notated examples is available. TWOSI. first converts multilingual sentence encoders into task-specific span encoders via contrastive learning. It then casts slot labeling into the span classification task sup- ported by the fine-tuned encoders from the previous stage. The method was evaluated on two standard multilingual ToD datasets, where we validated its strong performance across diverse languages and different training data setups.
As mentioned in $6, we opted for representative multilingual sentence encoders and components of contrastive learning that were proven to work well for other tasks in prior work (Reimers and Gurevych, 2020; Vulić et al., 2021) (e.g.. the choice of the contrastive loss, adopted hyper-parameters) while wider exploration of different setups and regimes in TWOSL'' 's Stage and Stage 2 might further improve performance and offer additional low-level insights.
The scope of our multilingual evaluation is also constrained by the current availability of multilin- gual evaluation resources for ToD NLU tasks. Finally, in order to unify the experimental proto- col across different languages, and for more com- prehensive coverage and languala compara- bility, we relied on multilingual encoders through- out the work. However, we stress that for the transfer-free scenarios, TWOSL is equally appli- cable to monolingual encoders for respective target languages, when such models exist, and this might yield increased absolute performance.
Due to its multi-comppoent nature, spectrum of extensions focused on its constituent compo- nents possible in future work, which includes other formulations of contrastive learning, tuning the models multilingually, mining (non-random) negative pairs and extending the method to cross- domain transfer learning for ToD (Majewska et al., 2022), especially for rare domains not covered by standard datasets. In this work, we have focused on the sample-efficient nature of TWOSL. We expect it to be complementary to modular and parameter- efficient techniques (Pfciffer et al., 2023). In the long run, we plan to use the method for large-
A Acknowledgments
The work was in part supported by Huawei re- search donation to the Language Technology Lab at the University of Cambridge. Ivan Vulié is also sup- ported by a personal Royal Society University Re- search Fellowship 'Inclusive and Sustainable Lan- guage Technology for a Truly Multilingual World' (no 221 137; 2022-).

---

References
cross- looss-lingual representation learning at scale. In Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
David Ifcoluwa Adelani, Jade Abbott, Graham Neu- big, Daniel D' souza, Julia Kreutzer, Constantine Lig- nos, Chester Palen-Michell Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Is- rael Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Bakatumba-Nabendee Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye. Jesujoba Alabi, Seid Muhie Yi- mam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Ver- rah Otiende. Iroro Orife. Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson. Mofetoluwa Adeyemi. Gerald Muriuki, Emmanuel Anebi, Chiamaka Chuk- wuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akin- ode, Deborah Nabagereka, Maurice Katusiime, Ayo- dele Awokoya, Mouhamadane MBOUP, Dibora Ge- breyohannes, Henok Tilaye, Kelechi Nwaike, De- gaga Wolde, Abdoulaye Faye, Blessing Sibanda, Ore- vaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thiemo Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Sa- lomey Osei 2021. MasakhaNER: Named entity recognition for African languages. Transactions of the Association for Computational Linguistics, 9:1 9:1116-1131.
Samuel Coope, Tyler Farghly, Daniela Gerz, Ivan Vulié, and Matthew Henderson. 2020. Span-ConveRT: Few- shot span extraction for dialog with pretrained conver- sational representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 107-121, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4188 Minneapolis, Minnesota. Association for Computational Linguistics.
Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics; Main Volume, pages 2112-2128, Online. Association for Computational Linguistics.
Inigo Casanueva, Ivan Vulić, Georgios Spithourakis, and Pawet Budzianowski. 2022. NLU++: A multi- label, slot-rich, generalisable dataset for natural lan- guage understanding in task-oriented dialogue. In Findings of the Association for Computational Lin- guistics: NAACL 2022, pages 1998-2013, Seattle, United States. Association for Computational Lin- guistics.
Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. 2020. Cert: Contrastive self-supervised learning for language understanding. ArXiv preprint, 12766.
Hao Fei, Meishan Zhang, and Donghong Ji. 2020. Cross-lingual semantic role labeling with high- quality translated training corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational L Linguistics, pages 7014-7026, On- line. Association for Computational Linguistics.
Oralie Cattan, Sophie Rosset, and Christophe Servan. 2021. On the cross-lingual transferability of multi- lingual prototypical models across NLU tasks. In Proceedings of the Isl Workshop on Meta Learning and Its Applications to Natural Language Processing. pages 36 36-43, Online. Association for Computational Linguistics.
Fangxiaoyu Feng. Yinfei Yang. Daniel Cer, Naveen Ari- vazhagan, and Wei Wang. 2022.
BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 878-891, Dublin, Ireland. Association for Computa- tional Linguistics.
Beiduo Chen, Wu Guo, Bin Gu, Quan Liu, and Yongchao Wang. 2022. Multi-level contrastive learn- ing for cross-lingual alignment. In IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022, pages 7947-7951. IEEE.
Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, et al. 2022. Massive: A Im-exameen example multilin- gual natural language understanding dataset with 51 typologically-divrse languages. ArXiv preprint, abs/2204.0880 08588..
Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005. Learning& a similarity metric discriminatively, with application to face verification. In 2005 JEEË Com- puler Society Conference on Computer Vision and Pattern Recognition (CVPR'05, volume I, pages 539-546. IEEE
Gabor Fuisz, Ivan Vulić, Samuel Gibbons, Inigo Casanueva, and Pawel Budzianowski.. 2022. Im- proved and efficient conversational slot labeling through question answering. ArXiv preprint, abs/2204.02123.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised

---

Milan Gritta, Ruoyu Hu, and Ignacio lacobacci. 2022. CrossAligner & co: Bero-shot transfer methods for task-oriented cross-lingual natural language under- standing. In Findings of the Association for Compu- lational Linguistics: ACL 2022, pages 4048-4061, Dublin, Ireland. Association for Computational Lin- guistics.
Fangyu Liu, Ivan Vulić, Anna Korhonen, and Nigel Collier. 2021. Fast, effective, and self-supervised: Transforming masked language models into universal lexical and sentence encoders. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1442-1459, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Milan Gritta and Ignacio lacobacci. 2021. XeroAlign:
Zero-shot cross-lingual transformer alignment. In Findings of the Association for Computational Lin-
Zihan Liu, Jamin Shin. Yan Xu, Genta Indra Winata, Peng Xu, Andrea Madotto, and Pascale Fung. 2019. Zero-shot cross-lingual dialogue systems with trans- ferable latent variables. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLI) pages 1297-1303, Hong Kong, China. Association for Computational Linguistics.
guistics: ACL-IJCNLP 2021. pages 371-381, Online. Association for Computational Linguisties.
Matthew Henderson and Ivan Vulić. 2021. ConVEx: Data-efficient and few-shot slot labeling. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3375-3389, Online. Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter, 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, IA, USA, May 6-9, 2019. OpenRe- view.net.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282-6293, Online. Association for Computational Linguisties.
Olga Majewska, Evgeniia Razumovskaia. Edoardo Maria Ponti, Ivan Vulic, and Anna Korhonen. 2022. Cross-lingual dialogue dataset creation via outline-basee generation. ArXiv preprint. abs/2201.13405
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings.
Shikib Mehri and Mihail Eric. 2021. Example-driven intent prediction with observers. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2979-2992. 2992, Online. Association for Computational Linguistics.
Jitin Krishman, Antonios Anastasopoulos, Hemant Puro- hit, and Huzefa Rangwala. 2021. Multilingual code- switching for zero-shot cross-lingual intent predic- tion and slot filling. In Proceedings of the 1st Work- shop on Multilingual Representation Learning, pages 211 211-223, Punta Cana, Dominican Republic. Associa- tion for Computational Linguistics.
Zaiqiao Meng, Fangyu Liu, Ehsan Shareghi. Yixuan Su, Charlotte Collins, and Nigel Collier. 2022. Rewire- then-prohe: A contrastive recipe for probing biomed- ical knowledge of pre-trained language models. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 4798-48 10, Dublin, Ireland. As- sociation for Computational Linguistics.
Jason Krone, Yi Zhang, and Mona Diab. 2020. Learn- ing to classify intents and slot labels given a handful of examples. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI. pages 96-108, Online. Association for Computa- tional Linguistics.
Mahdi Namazifar, Alexandros Papangelis, Gokhan Tur, and Dilek Hakkani- Tür. 2021. Language model is all you need: Natural language understanding ques- tion answering. ICASSP 2021-2021 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7803-7807. IEEF.
Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From zero hero: On the limitations of zero-shot language transfer with mul- tilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 4483-4499, On- line. Association for Computational Linguistics.
Jonas Pfeiffer, Sebastian Ruder, Ivan Vulié, and Edoardo Maria Ponti. 2023. Modular deep learning. CoRR, abs/2302.11529,
Edoardo Maria Ponti, Julia Kreutzer, Ivan Vulic. and Siva Reddy. 2021. Modelling latent transla- tions for cross-lingual transfer. ArXiv preprint, abs/2107.11353
Shining Liang, Linjun Shou, Jian Pei, Ming Gong. Wanli Zuo, Xianglin Zuo, and Daxin Jiang. 2022. Multi-level contrastive learning for cross-lingual spoken language understanding. ArXiv preprint, abs/2205.03656.
Libo Qin, Qiguang Chen, Tianbao Xie, Qixin Li, Jian- Guang Lou, Wanxiang Che, and Min- Yen Kan. 2022.

---

GL-CLeF: A global-local contrastive learning frame- work for cross- lingual spoken language understand- ing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Pupers), pages 2677-2688 Dublin, Ireland. Association for Computational Linguistics.
of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3795-3805, Minneapolis, Min- nesota. Association for Computational Linguistics.
Kaitao Song, Xu Tan. Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2020. Mpnet: Masked and permuted pre- training for language understanding. In Advances in Neural Information Processing Systems 33: An- nual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Libo Qin, Minheng Ni, Yue Zhang, and Wanxiang Che. 2020. Cosda-ml: Multi- tinuulligual code- switesiitiiig data augmentation for zero-shot cross- mooss-linuull NLP. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3853-3860. ijcai.org.
Evgeniia Razumovskaia, Goran Glavas, Olga Majew- ska, Edoardo Ponti, Anna Korhonen, and Ivan Vulic. 2022a. Crossing the conversational chasm: A primer on natural language processing for multi- lingual task-oriented dialogue systems. Journal of Artificial Intelligence Research, 14:1351-1402.
Yixuan Su. Fangyu Liu, Zaiqiao Meng, Tian Lan, Lei Shu, Ehsan Shareghi, and Nigel Collier. 2022. TaCL: Improving BERT pre-training with token-aware con- trastive learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2497-2507, Seattle, United States. Association for Computational Linguistics.
Evgeniia Razumovskaia, Ivan Vulié, and Anna Korho- nen. 2022b. Data augmentation and learned layer aggregation for improved multilingual language un- derstanding in dialogue. In Findings of the Asso- ciation for Computational Linguistics: ACL 2022. pages 2017-2033 Dublin, Ireland. Association for Computational Linguistics.
Yusheng Su, Xu Han, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Peng Li, Jie Zhou, and Maosong Sun. 2021. Css-Im: A contrastive framework for semi- supervised fine-tuning of pre- pre-trained language mod- els. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:2930-2944
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural L Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-UJCNLP), pages 9982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Gokhan Tur and Renato De Mori. 2011. Spoken lan- guage understanding: Systems for extracting seman- tic information from speech. John Wiley & Sons.
Shogo Ujiie, Hayate Iso, and Eiji Aramaki. 2021. Biomedical entity linking with contrastive context matching. ArXiv preprint, abs/2 1055075.3.
Rob van der Goot, Ibrahim Sharaf, Aizhan Imankulova. Ahmet Ustün, Marija Stepanović, Alan Ramponi, Siti Oryza Khairunnisa, Mamoru Komachi, and Bar- bara Plank. 2021. From masked language modeling to translation: Non-English auxiliary tasks improve zero-shot spoken language understanding. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2499, Online. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual us- ing knowledge distillation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512-4525, Online. Association for Computational Linguisties.
Nils Rethmeier and Isabelle Augenstein. 2021. A primer on contrastive pretraining in language processing: Methods, lessons learned & perspectives. ACM Com- puting Surveys (CSUR).
Laurens van der Maaten and Geoffrey E. Hinton. 2012. Visualizing non-metric similarities in multiple maps. Machine Learning, (7(1)333555
Peter J Rousseeuw.
1987. Silhouettes: a graphical aid the interpretation and validation of cluster analysis.
Journal of computational and applied mathematics, 20:53-65.
Ivan Vulié, Goran Glavaš, Fangyu Liu, Nigel Col- lier, Edoardo Maria Ponti, and Anna Korhonen. 2022. Exposing cross-lingual lexical knowledge from multilingual sentence encoders. ArXiv preprint, abs/2205.0022
Sebastian Ruder, Ivan Vulié, and Anders Søgaard. 2022. Square one bias in NLP: Towards multi- dimensional exploration of the research manifold. In Findings of the Association for Computational Linguistics: ACL. 2022, pages 2340-2354, Dublin, Ireland. Association for Computational Linguistics.
Ivan Vulić. Pei-Hao Su, Samuel Coope, Daniela Gerz, Pawel Budzianowski, lñigo Casanueva, Nikola Mrkšić, and Hsung-Hsien Wen. 2021. ConvFiT: Con- versational fine-tuning of pretrained language models. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages
Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019. Cross-lingual transfer learning for multilingual task oriented dialog. In Proceedings

---

1151-1168, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
A Language codes
Language codes which are used in the paper are provided in Table 6.
Danqing Wang, Jiaze Chen, Hao 7hou, Xipeng Qiu, and Lei Li. 2021. Contrastive aligned joint learn- ing for multilingual summarization. In Findings of the Association for Compulational Linguistics: ACL- IJCNLP 2021, pages 2739-2750 Online. Association for Computational Linguistics.
Language code
Language
Arabic
da
Danish
de
German
Weijia Xu, Batool Haider, and Saab Mansour. 2020. End-to-end slot alignment and recognition for cross- lingual NLU. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 5052-5063, Online. Association for Computational I Linguistics.
German
de-st
Youth-Tyrolean
French
Hindi
Indonesian
Italian
Japunese
kk
Kazakh
Dutch
Portuguese
Serbian
tr
Turkissh
Table 6: Language codes used in the paper
B Training Hyperparameters
The values for training hyperparameters for each stage of TWOSL and token classification are pro- vided in Table 7. For information about Adam and AdamW optimizers we refer the reader to Kingma and Ba (2015) and Loshchiloy and Hutter (2019), respectively.
Model
Epochs
Batch
Optim
LR
WD
Token classification
50
32
Adam
le 5
0.01
TWOSL: Stage
10
32
AdamW
2c 5.
0.01
TWOSL:
Stage 2, Step
30
32
Adam
1e-5 5
TWOSL; Stage 2, Step
100
32
Adam
le- 5 0.0
Table 7: Training hyperparameters for token classifica- tion and TWOSL, Acronyms: LR -eerning rate, WD weight decay rate.
C Full Scores on MultiATIS++
The exact numerical scores on MultiATIS++ across different languages and setups, which used as the source for Figure 3 in the main paper, are provided Table 8.
D Standard Deviation for xSID results using mpnet as a sentence encoder
The standard deviation of 5 runs for xSID are pre- sented in Table The standard deviation which is considerably lower than the margin between the performance of systems without and with con- trastive learning, proving the significance of im- provements that CL provides.

---

TR
AVG
50 examples
XLM-R
22.9
39.6
28.9
21.1
22.3
27.0
mpnet w/o CL
37.2
35.6
31.5
38.8
25.5
33.7
mpnet
59.4
57.2
50.6
58.8
41.1
100 examples
XI.
45.3
53.6
42.4
22.3
38.1
40.3
mpnet w/o CL
46.5
44.2
38.4
45.3
35.0
41.9
mpnet CL
69.4
69.8
62.4
67.3
56.8
65.1
200 examples
XLM-R
46.9
45.4
58.7
49.7
51.9
50.6
mppett w/o CL
53.2
52.6
44.0
51.4
44.3
49.1
mpnett w/ CL
69.6
74.4
61.3
70.5
58.3
66.8
500 examples
XLM-R
83.1
75.6
74.2
74.3
69.1
75.3
mpnett w/o CL
65.8
63.0
54.8
62.3
54.8
60.1
mpnet w/ CL
82.2
79.8
73.8
79.3
68.8
76.8
800 examples
XLM-R
88.7
71.6
N/A
79.7
72.1
78.0
mpnet w/o CL
68.0
67.6
N/A
63.6
56.1
63.8
mpnet w/ CL
84.9
85.4
N/A
82.2
71.9
81.1
Table Results on MultiATIS++. are averaged across 5 random seeds.
The results
6.70
Legsuge
0.01
0.00
0.0
Naxber cozegke
Figure 6: Impact of the number of negative examples per each positive example for the 50-example and 100- example setups. For clarity, the results are shown for a subset of languages in MultiATIS++, and the similar trends are observed for other languages.
E Results on xSID with and without the Binary Filtering Step
A comparison of results with and without applying the binary filtering step (i.e., Step in Stage 2 of TWOSL. is provided Table 10; see $5.1 for the discussion supported by this set of results.
F Number of negative examples for MultiATIS++

---

AR
DE
DA
DE-ST
ID
IT
KK
NL
SR
TR
AVG
50 training examples
mpnet w/o CL
1.88
3.14
1.79
2,91
2.55
1.52
2.51
1.82
3.87
2,42
2.44
mpnet w/ CL
3.04
1.70
4.18
2.53
11.72
2.97
2.86
1.93
1.95
3.14
3.60
100 training examples
mpnntt CL
2.74
2.01
3.65
2.29
1.11
2.29
3.48
0.81
2.12
mpnett w/ CL
3.18
2.96
2.64
3.25
1.71
1.89
1.62
0.95
5.62
2.69
2.65
training examples
mpnet w/o CL
1.72
1.28
1.94
1,84
2,41
0.96
1.41
2.40
4.17
1.74
1.99
mpnet w/ CL
3.99
4.87
1.72
2.01
2.44
6.58
1.75
3.52
3.18
0.63
3.06
Table 9: Standard deviation on xSID's test set using subsample of 50, 100, and 200 examples from its validation portion for training. The standard deviation calculated for random seed settings.
AR
DE
DA
DE-ST
ID
IT
KK
NL
SR
TR
AVG
examples
mpnet w/o CI. w/ step
23.9
25.1
26.8
21.8
28.5
26.3
22.6
29.7
23.9
24.8
25.3
mpnet w/o CL w/o step
24.8
25.5
28.3
21.6
28.4
26.7
23.1
26.9
22.8
24.0
25.2
impuet w/ CL w/ step 1
"36.8"
40.4
39.2
39.1
43.4
42.2
33.3
33..8
39.7
42.8
39.6
mpnet w/ CL w/o step
37.0
41.6
41.7
40.4
44.8
40.8
35.2
41.4
39.7
40.8
40.3
100 examples
mpnet w/o CL w/ step 1
34.3
37.1
36.2
31.2
37.6
38.7
30.0
37.1
33.7
32.8
34,9
mpnet w/o CI. w/o step
34.5
35.6
34,6
30.5
37.5
35.6
28.1
36.5
33.5
33.7
34,0
impnet w/ CL w/ step 1
43.55
551.1
44.2
44.1
52,6
47.2
40.4
48.3
46.5
51.1
46.9
mppeet w/ CL w/o step
42.7
50.3
44.9
46.2
48.6
48.0
41.0
47.5
45.3
47.6
46.2
200 examples
mpnet w/o CI. w/ step
41.3
44.9
44,9
42.2
46.5
46.5
37.8
46.3
41.7
41.2
43.3
mpnet w/o CL w/o step
42.8
45.8
44,4
43.3
47.6
46.2
36.6
46.9
43.4
41.3
43.8
impuet w/ CL w/ siep
48.2
55.8
50.3
52.1
59.0
55.2
46.1
53.8
52.9
52.5
52.6
mpnet w/ CL w/o step
47.3
52.3
51.6
51.9
60.8
54.0
45.3
54.5
53.4
52.6
52.4
Table 10: Results on xSID across different languages and setups with and without applying the binary filtering step: Step 1 Stage 2 of TWOSL.


Reviews:
[{"review_id": "8C05MVYxhW", "reviewer": "Reviewer_qbcu", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for the insightful comments and suggestions and will make sure to address them in the camera-ready version of the paper!\n\nRegarding the Reason-to-Reject 1, we will make sure to spell out the exact formulas for the contrastive loss in section 3.1 in the camera-ready version of the paper. At the same time, we would like to note that the process of contrastive learning is described in detail in section 3.1 and the loss more specifically is described in lines LL 288-302. \n\nRegarding the Reason-to-Reject 2, the focus of the paper is on few-shot multilingual slot labelling. We agree with the reviewer that TWOSL could be applied in few-shot cross-domain scenarios as well and see it one as one of the directions for future work (as mentioned in LL.619-620). Given the additional page in the camera-ready version, we will gladly expand the description of potential application of TWOSL in multi-domain setups. \n\nRegarding the Reason-to-Reject 3, We aimed to run a comprehensive comparison of TWOSL with the standard and representative baselines which were relevant and could be reasonably applied in the transfer-free setup (and with different amounts of training data).  We would be happy to include comparison to additional  baselines, given the additional space in the camera-ready version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method utilizes contrastive learning to bring the gap of the slot in different language-domain-task scenarios, and yields significant improvements in low-resource language scenarios, where only 50,100, or 200 training examples are in the training set.\nFurther, the proposed contrastive learning method could effectively distinguish different slot types.\nBesides, the authors also provide detailed analyses of the proposed method in terms of the robustness and generalization, in the low-resource language scenarios.", "reasons_to_reject": "1. Lack of the corresponding formulas of the contrastive learning objection, which makes it indirect to understand the content.\n2. Lack of evaluation on low-resource domain scenarios.\n3. Lack of solid comparison with other related methods.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors aim to address the inherent data scarcity issue on the new language-domain-task scenario of the slot labeling task.\nThe authors propose a two-stage slot labeling method to make the model (trained on English data) transfer-free in new scenarios.\nBy the proposed method, the model could perform well on few-shot setups.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "S7mpLYPcWG", "reviewer": "Reviewer_7N7r", "summary": "", "questions": "A) Given that the method doesn't rely on transfer, is there a need to use massively Multilingual PLM's for this method? For languages where a monolingual (or more specific language family based PLM such as AfriBerta or Bertic) exists, would they be better choices since they avoid negative transfer from pretraining? \n\nB) Could the method be applied in scenarios where transfer data does exist? It doesn't seem uniquely \"transfer-free\", so might have broader use as a data efficiency method for transfer scenarios as well right? It would also be interesting to see if this method continues to be advantageous as labeled data continues to scale beyond 200 examples.\n\nC) Is the method applicable to annotation structures which have nested slots? It seems like perhaps no since stage 2 assigns only one slot per token?\n\nD) You mention that as a limitation the method relies on white-space separated words. What about the method prevents you from using the outputs of a tokenizer directly?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for a really thorough review and thoughtful comments and suggestions! Please find the responses to the questions below. If given the opportunity, we will also address them in the camera-ready version of the paper. \n\nWith regards to Reason to Reject 1 (about the low-resource languages), at the time of the experimentation for the paper the dialogue NLU datasets with low-resource languages (including MASSIVE) were not available. At the same time, as we were interested in the performance of TWOS in low-resource languages, we conducted a preliminary experiment for Yoruba (yor) and Luo (luo) languages on a related task of Named Entity Recognition (NER). The choice of the task was motivated by two reasons: a) similarly to slot labelling, NER is a sequence labeling task; b) there is a dataset built specifically for NER for low-resource African languages. The experiments were conducted with 100 training examples, using XLM-Roberta as a sentence encoder. TWOSL has brought considerable improvements for the languages, namely, 14.46 to 45.38 F-1 for yor and 16.56 to 36.46 F-1 for luo. These results prove the effectiveness of the method for low-resource languages as well. We will make sure to include these results into the camera-ready version if given the opportunity, and we also plan to run TWOSL on a representative subset of low-resource languages sampled from the MASSIVE dataset.\n\nWith regards to Reason to Reject 2 (applying the techniques such as IA3), we thank the reviewer for a great suggestion! We did not include the PEFT results as the work focuses more on sample efficiency and not parameter efficiency for which methods like IA3 were proposed. At the same time, the training of the fully tuned model was regularised with more standard techniques such as early stopping and by using Adam optimiser with reducing learning rate. TWOSL can also work well with PEFT techniques such as IA3 and adapters, and this is definitely something to add to enrich the discussion in the paper.\n\nWith regards to the questions:\n\n* A: Thanks for the question! We fully agree that more specialised, language-/language-family specific sentence encoders could also be applied in the setup instead of multilingual encoders. In our experimental setup we opted for the universality of the underlying models, i.e., making sure that we apply the same setup to all languages while not every language in our experiments had a suitable sentence encoder openly available. We will make sure to add this point to the potential future work directions in the final version of the paper - the main benefits of TWOSL remain in cases when the multilingual encoder is replaced with a monolingual language-specialised one.\n\n* B: In the experiments on MultiATIS++ (FIg. 3), we experiment with up to 800 in-language training examples available. These experiments show that the performance improvements scale with the number of training examples, although the gap between full finetuning and TWOSL gradually decreases as the number of training examples increases. We also ran a small experiment where we rely on large transfer data from English (with the target languages being German and French) - the results are reported in Table 4 of the main paper. It would be interesting to extend this set of experiments to more (lower-resource) target languages as well in future research.\n\n* C: In this work we focused on single-level slot annotations, while TWOSL can be extended to nested slot annotations by training another slot classifier on top of first level slot annotators. We see this as a potential extension of the method and will make sure to point to it as a potential future direction for the work - thanks for the insightful suggestion!\n\n* D:  In terms of methodology, it can be applied on top of tokenised as well as word-separated inputs. However, tokenized inputs would need alignment between predicted labels and the annotations and we opted to avoid this additional level of noise and discrepancy between TWOSL and baselines. \n\nThanks a lot for the ideas on improvements for the notations! We will make sure to change the notations in the paper accordingly!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The work poses and tackles an interesting challenge of a learning a slot filling model for a novel annotation scheme. This is well justified since the granularity of slot filling tasks often makes them less transferable than more universal span filling tasks like NER, especially across linguistic and cultural contexts. Overall, it seems like improving capabilities in this regard could enable into tools which enable new applications of NLP.\n\n- The work is quite thorough in showing the effects of their approach across multiple languages, model families, and data scales. The effect is consistently significant across settings. Each component of the approach is shown to be valuable through targeted ablations which motivates the entire pipeline. This thoroughness leaves little doubt on the empirical effectiveness of the method, especially across multiple languages.\n\n- The work justifies the method well and makes it relatively easy to follow what is being done and why. This leads to a paper that reads easily and doesn't introduce unnecessary complexity in it's methods description or in it's analysis.", "reasons_to_reject": "- It would have been nice to see more truly low-resource languages in the evaluation. The authors cite a lack of evaluation resources as the reason for this in limitations, but the MASSIVE dataset (released in June 2022 on Arxiv, though only at ACL 2023 in official publication) would have been a prime candidate to up the linguistic diversity of the experiments. Whether or not that work counts as contemporaneous depends on one's opinion on Arxiv vs. peer reviewed venues though, so not a major critique especially since it's acknowledged in limitations.\n\n- While the transfer free framing of the method is novel, it seems heavily related to low resource scenarios in general. Ideally, the work would have also compared to data efficient learning techniques such as IA3 rather than only using full-finetuning as a baseline. Even if not full low-resource learning techniques, simple baselines of regularization to avoid overfitting on small data quantities would be great context for the technique. Without this, the work is slightly less sound since it's not clear how it compares to more established methods for low-data settings.", "questions_to_authors": "A) Given that the method doesn't rely on transfer, is there a need to use massively Multilingual PLM's for this method? For languages where a monolingual (or more specific language family based PLM such as AfriBerta or Bertic) exists, would they be better choices since they avoid negative transfer from pretraining? \n\nB) Could the method be applied in scenarios where transfer data does exist? It doesn't seem uniquely \"transfer-free\", so might have broader use as a data efficiency method for transfer scenarios as well right? It would also be interesting to see if this method continues to be advantageous as labeled data continues to scale beyond 200 examples.\n\nC) Is the method applicable to annotation structures which have nested slots? It seems like perhaps no since stage 2 assigns only one slot per token?\n\nD) You mention that as a limitation the method relies on white-space separated words. What about the method prevents you from using the outputs of a tokenizer directly?", "paper_topic_and_main_contributions": "This work looks at developing an effective method for bootstrapping a slot filling model using limited annotated data across multiple languages. They first refine a pretrained sentence encoder for slot filling using a span-contrastive loss with hard positive and hard negative sampling. Then, they train a two step classifier. The first step classifies whether a span is a valid slot at all, while the second step predicts which slot type the span falls under. They show this procedure is empirically effective for bootstrapping slot filling models using MultiATIS and xSID for XLM-R, mpnet, and LaBSE.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Works on Multilingual Slot Labeling / Semantic Parsing which might be of interest:\n\nhttps://arxiv.org/pdf/2204.08582v2.pdf (Task-Oriented Dialogue Dataset that would allow for extensions to more typologically diverse languages)\n\nhttps://aclanthology.org/2022.acl-long.285/ and https://aclanthology.org/2023.acl-long.199/ (Adversarial Alignment methods for zero-shot slot labeling/semantic parsing)\n\nhttps://aclanthology.org/2021.findings-emnlp.279.pdf (State of the art label projection method for translation and slot filling)\n\nNot missing references, but things which would be stronger baselines:\n\nhttps://arxiv.org/abs/2205.05638 (Existing work for low-data tasks vs. the full finetuning baseline used)\n\nhttps://aclanthology.org/2021.acl-long.172.pdf (Adapters are more robust to overfitting, whereas full finetuning seems likely to overfit in these low-data scenarios)", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 280: Introduce variable K before using it for \"2K\". Initially, this read confusingly as 2,000 since K hadn't been defined before use.\n\nTable 2: Can you label all significant results with an asterisk or a sword to avoid needing the appendix to confirm significance. Full STDev could be left to the appendix, but marking whether a particular P value is reached in the main table would be useful.\n\nThe references to \"Step 1\" vs. \"Stage 1\" get slightly confusing and I found myself having to go back to remind myself which was a step and which was a stage. It might be easier to read if you use a notation that distinguishes granularity such as renaming Step 1 and 2 to \"Stage 2.A and 2.B\". \n\nIn between transfer-free and full transfer, it seems like there are cases where semantically related classes from one labeling scheme to another. The contrastive learning technique here seems like it would be useful pretraining in those cases. It seems worth noting that, especially since the full transfer-free setting is relatively extreme."}}, {"review_id": "2M8G8wQfbS", "reviewer": "Reviewer_N33Q", "summary": "", "questions": "In general, zero-shot models trained on a single language are penalised by annotation guidelines that may be inconsistent across languages, for example including or excluding determiners or prepositions in a slot value. Having a small number of examples for each language helps in fixing language specific annotation inconsistencies. A model trained on a single language will default to the same behavior for other languages, most of the time.\n\nIn the Table 2 experiment, it is not clear if and how English data is used to train the model.\n\nFigure 4: is the “with” column the sum of the binary and slot classifier times, or just the inference time of the latter on the XSID examples?\n\nWhat about seq2seq methods for parsing such as mT5 or ByT5?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are extremely grateful for your in-depth review and your thoughts and suggestions for the paper.\n\nWith regards to the seq2seq baselines, thanks for the suggestion of an additional baseline! We will add a full set of results to the final version of the paper. However, our preliminary results with mT5 demonstrated that seq2seq formulation with mT5 as a base model is only marginally better than the standard supervised model. \n\nWith regards to the questions, we agree that TWOSL improves cross-lingual performance both by avoiding source language inductive bias and by avoiding the discrepancies between languages in annotation guidelines. \n\nIn the experiment in Table 2 the training data did not include any English data. The models were trained only using the few-shot target language data. We will make sure to add further clarifications in the corresponding (Main Results) subsection of the paper. Table 4 provides some results in transfer-based scenarios where we assume availability of large annotated data in English and we perform few-shot transfer.\n\nIn Fig. 4, the bars “With” included the total time it took to classify the spans, i.e., the sum of binary classifier and slot classifier. The reduction in time processing arises from the difference in the processing of inputs: in the setup “With” binary classifier all sentence-span pairs are processed by the fast binary classifier and full slot classifier processes only a subset of sentence-span pairs which are classified as related by the binary classifier. At the same time, in “Without” setup all the sentence span pairs go through the slot classifier.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a sample-efficient method for parsing that requires little annotated data to perform reasonably well in a multilingual setting.\n\nResearch on few-shot and scarce data scenarios, especially considering low-resource languages, is extremely important.\n", "reasons_to_reject": "There is a risk that results from multilingual seq2seq baselines could affect the paper story.\n", "questions_to_authors": "In general, zero-shot models trained on a single language are penalised by annotation guidelines that may be inconsistent across languages, for example including or excluding determiners or prepositions in a slot value. Having a small number of examples for each language helps in fixing language specific annotation inconsistencies. A model trained on a single language will default to the same behavior for other languages, most of the time.\n\nIn the Table 2 experiment, it is not clear if and how English data is used to train the model.\n\nFigure 4: is the “with” column the sum of the binary and slot classifier times, or just the inference time of the latter on the XSID examples?\n\nWhat about seq2seq methods for parsing such as mT5 or ByT5?\n", "paper_topic_and_main_contributions": "The paper proposes a parsing approach that works better than standard token classification in scarce data multilingual scenarios. Instead of using the few examples available for each language to train a token classification model, i.e. outputting a slot label for each input token, they decompose the parsing task into two classification tasks. The authors first train a binary classifier to identify which among the spans that can be extracted from a sentence may be a slot value. This classifier is used to filter the n^2 spans generated from a sentence of length n and reduce the inference time of the second classification step. For the latter, the authors train a sentence encoder using a contrastive objective that pushes the representations of slot values having the same (or different) label closer (or farther). The sentence encoder updated with this contrastive loss is further fine-tuned on slot classification data and used to label the spans from the sentence.\n\nThis method works better than token classification especially when there is little annotated data for each language. The authors convincingly show that the contrastive loss step is crucial to obtain better performance. The quality of the base sentence model affects the final performance motivating for increasingly better base models.\n\nThe paper is missing a seq2seq baseline powered by some pretrained multilingual LM (mT5 or mBart, for example), which have become standard for parsing tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "For the related work section, there are also slot projection methods that do not require word alignment and rely on a small amount of data and multilingual PLMs [1].\n\n[1] Translate & Fill: Improving Zero-Shot Multilingual Semantic Parsing with Synthetic Data\n"}}]

Author rebuttal:

Meta-review: The paper proposes a two-stage contrastive learning based slot labeling approach to solve the language-domain-task adaptation challenge   for task-oriented dialogue system development. Reviewers agree that the work is well-motivated and achieves significant improvements in low-resource language scenarios. Several questions regarding the evaluation on truly low-resource languages, and comparison with seq2seq model were sufficiently addressed during rebuttal by authors.
Decision: Accept-Main