Adaptive End-to-End Metric Learning for Zero-Shot Cross-Domain Slot Filling
Authors: Yuanjun Shi
Linzhi Wu
Minglai Shao
Venue: EMNLP 2023 Main (2023)
Keywords: end-to-end
zero-shot learning
metric learning
slot filling
TL;DR: Considering simplicity, efficiency and generalizability, we present a cascade-style joint learning framework coupled with slot-level contrastive representation learning.

Abstract: Recently slot filling has witnessed great development thanks to deep learning and the availability of large-scale annotated data. However, it poses a critical challenge to handle a novel domain whose samples are never seen during training. The recognition performance might be greatly degraded due to severe domain shifts. Most prior works deal with this problem in a two-pass pipeline manner based on metric learning. In practice, these dominant pipeline models may be limited in computational efficiency and generalization capacity because of non-parallel inference and context-free discrete label embeddings. To this end, we re-examine the typical metric-based methods, and propose a new adaptive end-to-end metric learning scheme for the challenging zero-shot slot filling. Considering simplicity, efficiency and generalizability, we present a cascade-style joint learning framework coupled with context-aware soft label representations and slot-level contrastive representation learning to mitigate the data and label shift problems effectively. Extensive experiments on public benchmarks demonstrate the superiority of the proposed approach over a series of competitive baselines.

Adaptive End-to-End Metric Learning for Zero-Shot Cross-Domain Slot
Filling
Yuanjun Shi', Linzhi Minglai Shao
'School of New Media and Communication,
Tianjin University, Tianjin, China
"Schoo of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China [[swi tchsyj, shaoml j@tju. edu. 2 $lindgew@std. westc. edu. cn
Abstract
Source Domaiu /Taininol
Recently slot filling has witnessed great devel- opment thanks to deep learning and the avail- ability of large-scale annotated data. How- ever, it poses critical challenge handle novel domain whose samples are never seen during training. The recognition performance might be greatly degraded due severe do- main shifts. Most prior works deal with this problem two-pass pipeline manner based metric learning. In practice, these dominant pipeline models may be limited in computa- tional efficiency and generalization capacity because of non-parallel inference and context- free discrete label embeddings. To this end, we re-examine the typical metric-based methods, and propose new adaptive end-to-end metric learning scheme for the challenging zero-shot slot filling. Considering simplicity, efficiency and generalizability, we present a cascade-style joint learning framework coupled with context- aware soft label representations and slot-level contrastive representation learning to mitigate the data and label shift problems effectively. Extensive experiments public benchmarks demonstrate the superiority of the proposed ap- proach over a series of competitive baselines.!
Musicttum Include the album
Artist
Playlar In urban poct
by arthur rhames
Target Domain (Teating)
Musictom
Album
Artist lam lahrer
Service
Play
the
album
jack
lakes
the
foor
by
on
nelffix
Figure 1: Examples from SNIPS dataset. Apart from data distribution shifts, the target domain contains novel slot types that are unseen in the source domain label space (e.g., Album and Service). Moreover, the slot enti- ties tend to embody domain -spccii nature in contrast to the counterpart contexts.
Despite the remarkable accomplishments, there are at least two potential challenges in realistic ap- plication scenarios. First the data scarcity prob- lem in specific target domains (e.g., Healthcare and E-commerce). The manually-annotadle train- ing data in these domains is probably unavailable, and even the unlabeled training data might be hard to acquire (Jia al., 2019; Liu et al., 2020a). As result, the performance of slot filling models may drop significantly due to extreme data distribution shifts. The second is the existence of label shifts (as shown in the example in Figure 1). The target domain may contain novel slot types unseen in the cource-domann label space (Liu et al., 2018; Shah et al., 2019; Liu et al., 2020b; Wang al., 2021), namely there is mismatch between different do- main label sets. This makes it difficult to apply the source models to completely unseen target domains that are unobservable during the training process. Zero-shot domain generalization has been shown to be feasible solution to bridge the gap of domain shifts with no access to data from the target domain. Recent dominating advances focus the two-step pipeline fashion learn the zero-shot model using the metric learning paradigms (Shah et al., 2019; Liu et al., 2020b; He et al., 2020; Wang et al., 2021: Siddique et al., 2021). Nevertheless, besides inef- ficient inference resulted from non non-parallelizatoon.
1 Introduction
Slot filling, essential component widely ex- ploited in task-oriented conversational systems, has attracted increasing attention recently (Zhang and Wang, 2016; Goo al., 2018; Gangadharaiah and Narayanaswamy, 2019). It aims identify a spe- cific type (e.g., artist and playlist) for each slot entity from a given user utterance. Owing to the rapid development of deep neural networks and with help from large-scale annotated data, research slot filling has made great progress with consid- erable performance improvement (Qin et al., 2019; Wu al., 2020; Qin et al., 2020, 2021).
*Corresponding author.
'The souree code is available at httpp:://gihhll com/ SwitchsyyjAdaE200 -SSF.

---

the generalization capability of these models may be limited due lack of knowledge sharing be- tween sub-modules, and context-independet dis- crete static label embeddings. Although the al- ternative question-answering (QA) based methods (Du et al., 2021; Yu et al., 2021; Liu et al., 2022) are able to achieve impressive results, they need to manually design and construct the questions/- queries, essentially introducing detailed descriptive information about the slot types.
orate our solution to this problem.
2.1 Problem Statement
Suppose we have the source domain Ds " No with Ns labeled samples from dis- tribution Ps. and the (testing) target domain D7 {(v]) ) j=1 with C slot types from target distribu- tion PT. We define Ωs as the label set of source domain Ds. and Qt as the label set of target do- main Dy. San = Ωg n sty denotes the common slot label set shared by Ds and Dr. In the zero- shot scenario, the label sets between different do- mains may be mismatching, thus Qash S Ωg and pS + PT. The goal is to learn robust and gen- eralizable zero-shot slot filling model that can be well adapted novel domains with unknown test- ing distributions.
In this work, we revisit the metric-based zero- shot cross-domain slot filling under challenging domain (both data and label) shifts. We propose adaptive end-to-end metric learning scheme to improve the efficiency and effectiveness of the zero- shot model in favor of practical applications. For one thing, we provide cascade-style joint learning architecture well coupled with the slot boundary module and type matching module, allowing for knowledge sharing among the sub-modules and higher computational efficiency. Moreover, the soft label embeddings are adaptively learnt by cap- turing the correlation between slot labels and utter- ance. For another, since slot terms with same types tend to have the semantically similar contexts, we propose a slot-level contrastive learning scheme to enhance the slot discriminative representations within different domain context. Finally, to verify the effectiveness of the proposed method, we carry out extensive experiments on different benchmark datasets. The empirical studies show the superi- ority of our method, which achieves effective per- formance gains compared to several competitive baseline methods.
2.2 Overall Framework
In order to deal with variable slot types within an unknown domain, we discard the standard se- quence labeling paradigm by cross-labeling (e.g., B E-playlist, I-playlist). Instead, we adopt cascade-style architecture coupled with the slot boundary module and typing module under a joint learning framework. The boundary module is used to detect whether the tokens in an utterance are slot terms or not by the CRF-based labeling method with BIO schema, while the typing module is used to match the most likely type for the corresponding slot term using the metric-based method. Since pre- training model is beneficial to learn general repre- sentations, we adopt the pre-trained BERT (Devlin et al., 2019) as our backbone encoder Figure 2 shows the overall framework, which is composed of several key components as follows:
Overall, the main contributions can be sum- marized as follows: (1) Compared with existing metric-based methods, we propose more efficient and effective end-to-end scheme for zero-shot slot filling, and show our soft label embeddings perform static
much better than previous commonly-use label representations. (2) We investigate the slot- level contrastive learning to effectively improve generalization capacity for zero-shot slot filling. (3) By extensive experiments, we demonstrate the benefits of our model in comparison to the existing metric-based methods, and provide insightful quantitative and qualitative analysis.
Context-aware Label Embedding Let c (cI."".,
Gasls (ci E Ωs) denotes slot la- bel sequence consisting of all the elements of Ωs. Given input utterance sequence x [x,,... ,I,] of tokens with the correspond- ing ground-truth boundary label sequence ydd [w/d.... ..... (d) (ufet c (B, I, 0]) and slot label se- quence yd = -. ....,, (1," E Ωs), the slot
label sequence acts as prefix of the input utter-
2 Methodology
"Notice that we assume the BERT model is used as our encoder, hut our method can also be integrated with other model urchitectures (e.g., RoBERTa (Liu et al,, 2019b)).
this section, we first declare the problem to be addressed about shoo slot filling, and then elab-

---

Metric-based Slot Typing
Slor Boundary Detection
Masked Typing Loss
Gold calvtr
Sior Mask
Manclimm
Masttid
Token-wise Similarity Matching
CRF
Nauntiee
Semalarty Meric
Boundary
Emhedding
add
this
track
to
global Playllist
funk Playlist
FC
Musiclltm
PlaylissOwner
Lifi Disccipke HEaTE
Sili Reproccaaioo
BiLSTM
ML.P
FC
Futer
BERT Encoder
(b) Slot-level Contrastive Learning
[CLS] Artist Playlist ... Musicltem add this track to my global funk
Slot Labels
Utterance
(a) Cascado-style Joint Leaminng Architecture
Figure 2: Illustration of the overall framework. Figure (a) shows the cascade-style joint learning architecture coupled with two core components: Metric-based Slot Typing and Slot Boundary Detection. Figure (b) shows the slot-level contrastive learning module used only during training. The slot entity tokens with the same type are positive pairs (i.e. the blue lines) while those with different types negative ones (i.e. the red lines).
ance, which is then encoded by 3
pendency. The negative log-likelihood objective function can formulated as follows:
Tlabel; Nutter] ([;
(1)
e Lincar Lincar(huster).
where Flahet and Tutter denote the fused contextual representations of the label and utterance sequence, respectively.
score(xxy) =
> (Ty-iss + +elly.)).
(3)
For each slot type. the slot label matrix is ob- tained by averaging over the representations of the slot label tokens. Unlike the conventional discrete and static label embeddings (Liu et al., 2020b; Sid- dique et al., 2021; Ma al., 2022) that capture the semantics of cach textual label separately, we attempt to build the label-utterance correlation, and the adaptive interaction between the slot labels and utterance tokens encourages the model to learn the context-aware soft label embeddings dynamically, which will be exploited as the supervision informa- tion for the metric learning.
Lody llg p(ybdlx))
= = log exp(score(x, yhd))
Eyerr, exp(score(xx y"))
where E RRxx3 denotes the three-way emission vectors containing boundary information, is the 3x3 learnable label transition matrix, and V is the set of all possible boundary label sequences of utterance x. While inference, we employ the Viterbi algorithm (Viterbi, 1967) to find the best boundary label sequence.
Metric-based Slot Typing Although slot bound- ary module can select the slot terms from an ut- terance, it fails to learn discriminative slot entities. Thus, we design typing module to achieve in parallel by semantic similarity matching between slot labels and utterance tokens.
Slot Boundary Detection To determine the slot terms, we obtain the contextualized latent repre- sentations of the utterance through a single-layer BiLSTM,
batter BiLSTM((ruter).
(2)
Coneretely, we take advantage of the above boundary information locate the slot entity kens of the utterance. We specially exploit the soft-weighting boundary embedding vectors for en- abling differentiable joint training, which are com- bined with the contextual utterance representations
Then, CRF layer is applied to the slot boundary decoding, aiming model the boundary label de-
CConsidering the slot label sequence is not a natural lan- guage sentence linguistically, we remove the [SEP] token used to concatenate sentence pairs in BERT.

---

obtain the boundary--nhanceed representations:
More specifically, we define supervised con- trastive objective by decreasing the similarities be- tween different types of slot entities while increas- ing the similarities between the same ones. We just pay attention to the slot entities by masking out the parts with O boundary labels. Then, we gather in-baich positive pairs P with the same slot type and negative pairs p with different ones:
Thound = softmax(e) Es,
(4)
Lincar(Concatin (Concat(Fuster: (bound));
where E, R33d is look-up table store train- able boundary embeddings, and de indicates the embedding dimension. Meanwhile, the label em- beddings are calculated by a bottlencck module consisting of an up-projection layer and a down- projection layer with a GELU (Hendrycks and Gim- pel, 2016) nonlinearity:
ReLU (einnaLinaarFattte
3 i j}, p = [((s. ( lb: = 8j
(7)
p {(s' yB? * uft. #1)
v = Lincarap (Cnncarap(CELLU (CEL. (Ntabel)) (5)
where s denotes the projected point embeddings, and all example pairs are extracted from a mini- batch. Furthermore, we adapt the NT-Xent loss (Chen et al., 2020) to achieve the slot-level discrim- ination, and the contrastive learning loss function can be formulated as:
Furthermore, we leverage token-wise similarity matching between L2-normalized utterance repré- sentations and label embeddings. Since the slot entities are our major concern for predicting types, we ignore the non-entity tokens by mask provided by the boundary gold labels, resulting in the slot typing loss function defined as follows:
log [P+] E(st (oo(d(ssss)//>
Letr
expppi
exp(d(s', s))/T)
exp((us,)) sg( sg((+))))
(8)
tyyp
1 zo log プ== exp((u,, s)(),)))
[2s
where denotes p+ dd-,(---)) denotes the dis- tance metric function (e.g., cosine similarity dis- tance), and T is a temperature parameter. We will investigate different kinds of metric functions the following experiment section.
exp( ( (sg (u:) Vi-))
+o) log
12sl exp (ssg(()))))) vz))
where (.-) measures the cosine similarity of two embeddings, sg(-) stands for the hoopggadient oper- ation that does not affect the forward computation, i' indicates the index corresponding to the gold slot label y and 1 [o* (0, 1} is an indicator function, evaluating to if yfa is non-O tag. Eq. 6 makes sure the label embeddings act as the super- vision information (the /st term) and meanwhile are progressively updated (the 2nd term).
2.3 Training and Inference
During training, our overall framework is opti- mized end-to-end with min-batch. The final train- ing objective is to minimize the sum of the all loss functions:
C L + Ltyp + Cetr:
(9)
where each part has been defined in the previous subsections. During inference, have the slot type set of the target domain samples, and the test- ing slot labels constitute the label sequence, which then concatenated with the utterance as the model input. The CRF decoder predicts the slot bound- aries of the utterance, and the predicted slot type corresponds to the type with the highest-matching score. We take the non-O-labeled tokens as slot terms while the O-labeled tokens as the context.
Slot-level Contrastive Learning Recent line of works have investigated the instance-level con- trastive learning by template regularization (Shah et al., 2019; Liu et al., 2020b; He et al., 2020; Wang et al., 2021). As slots with the same types tend to have the semantically similar contexts, inspired by Das al. (2022), we propose to use the leot-level contrastive learning to facilitate the discriminative slot representations that may contribute to adapta- tion robustness."
3 Experiments
'Different from Das et al. (2022), we do not use the Gaus- sian embeddings produced by learnt Gaussian distribution parameters. There are two main reasons: one is to ensure the stable convergence of training, and the other is that the token representations may not follow normal distribution.
3.1 Datasets and Settings
To evaluate the proposed method, we conduct the experiments on the SNIPS dataset for zero-

---

Domain
ATP
BR
GW
PM
RB
SCW
SSB:
Model, -Tt..
40%)
(39
(67))
Conchgene (Lio ot 2020b)
50.28
31,87
$2,30
31.75
23.33
70.76
29.33
41.37
PCL PCLCarer (Wang et al., 2021)
30.38
20.89
32.99
25.55
20.76
62.40
13.82
29.54
LEONABeRT (Siddique
2021)
51.23
46.68
68.72
43.20
25.23
47.01
27.99
44.01
QASFIEENC (Du et al., 2021)
39.29
43.13
59.02
33.62
33.34
$9.90
22.83
44.45
SLMRC1 (L.n 2022)
50.16
55.17
G2P (Iá et al., 2023)
59.83
61.23
62,58
62.73
45.88
71.30
48.26
58.82
Ours (wio Slor CL.)
61.13
41.67
71.47
34,77
30.75
68.81
34,64
49.03
Table 1: Fl -scores of zero-shot slot filling across different domains. Slot-CL denotes the slot-level contrastive learning. We show the number labels in the source and target domains. The unseen rate refers to the proportion of non-overlapped source-target domain labels in the target label set. denotes the QA-based methods that introduce manually-designed query for each slot label. 3 denotes the generative method along with prompts for each slot label.
shot settings (Coucke et al., 2018), which con- tains 39 slot types across seven different do- mains: AddToPlaylist (ATP), BookRestaurant (BR), GetWeather (GW), PlayMusic (PM), RateBook (RB), SearchCreativeWorkk (SCW) and SearchhcrreeningEvent (SSE). Following previous studies (Liu et al., 2020b; Siddique et al., 2021), we choose one of these domains the target domain used for training, and the remaining six domains combined to form the source do- main. Then, we split 500 samples in the target domain as the development set and the remainder are used for the test set. Moreover, we consider the case where the label space of the source and target domains are exactly the same, namely the zero-resource setting (Liu et al., 2020a) based on named entity recognition (NER) task. We train model the CoNLL-2003 (Sang and Meulder, 2003) dataset and evaluate on the CBS SciTech News dataset (Jia et al., 2019).
chitecture (Li al., 2020).
3.3 Implementation Details
We use the pre-trained uncased BERT BASE model as the backbone encoder. The dimension of the boundary embedding set to 10. We use 0.1 dropout ratio for slot filling and 0.5 for NER. For the contrastive learning module, we use the cosine metric function and select the optimal tempera- ture from 0.1 to 1. During training, the AdamW (Loshchilov and Hutter, 2019) optimizer with mini-batch size 32 applied to update all trainable parameters, and the initial learning rate is set 2e-5 for BERT and 1c-3 for other modules. All the models are trained on NIVIDIA GeForce RTX 3090Tï GPUs for up to 30 epochs. The averaged Fl-score over five runs is used to evaluate the per- formance. The best-performing model the de- velopment set is used for testing.
3.4 Zero-Shot Slot Filling
3.2 Baselines
As shown in Table our method achieves more promising performance than previously proposed metric-based methods on various target domains, with an average about 5% improvements com- pared with the strong baseline LEONA. We at- tribute it to the fact that our proposed joint learn- ing model make full use of the sub-modules. and the context-aware soft label embeddings provide better prototype representations. Moreover, we also observe that the slot-level contrastive learn- ing plays an important role in improving adapta- tion performance. Our model with Slot-CL obtains consistent performance gains over almost all the target domains except for the SSE domain. We suspect that it may result from slot entity confu- sion. For example, for slot entities "cinema" and
We compare our method with the following com- petitive baselines using the pre-trained BERT as encoder: (1) Coach. Liu et al. (2020b) propose two-step pipeline matching framework assisted by template regularization; (2) PCLC. Wang al. (2021) propose a prototypical contrastive learning method with label confusion; (3) LEONA. Sid- dique et al. (2021) propose to integrate linguistic knowledge (e.g., external NER and POS-tagging cues) into the basic framework.
Although not our focused baselines, we also compare against the advanced generative baselines (Li et al., 2023) with T5-Large and QA-based meth- ods (Du et al., 2021; Liu et al., 2022) that require manual efforts to convert slot type deseriptions into sentential queries/questions, and process by means the machine reading comprehension (MRC) ar-
hnttps://huggingface co/ btw.bbbrr-baseeuccseededdd

---

"theatre" from SSE, they are usually annotated with abject_location_type but "cinemas" in "caribbean cinemas" and "theatres" "star the- atres" are annotated with location_name which is prone to be misled by the contrastive objective. Additionally, without introducing extra manual prior knowledge, our method achieves very com- petitive performance compared with the QA-based baselines.
hand, the QA-based methods require introducing different queries for all candidate slot labels regard- ing each utterance, increasing the decoding latency of a single example.
Model
Time Cost (s)
Speedup
CoachBERT
70.98
17.14x
PCLCBERT
76.61
18.50
LEONA
57.49
13.89
3.5 Zero-Resource NER
SLMRC
29.21
7.06x
In particular, examine method in the zero- resource NER setting. As presented in Table our method also adaptable to this scenario, and exceed or match the performance of previous com- petitive baselines. Meanwhile, the slot-level con- trastive learning can yield effective performance improvements.
Ours
4.14
1.00x
Table 3: Comparison of inference efficiency. Speedup denotes the ratio time taken by slot prediction part of different models to one epoch the BookRestau- rant with batch size 32.
Label-Utteranee Interaction Here we examine how our model benefits from the label-utterance in- teraction. As presented Table 4, the performance of our model drops significantly when eliminating the interaction from different aspects justifying our design. Compared to the other degraded in- teraction strategies, the utterance-label interaction helps learn the context-aware label embeddings, namely the utterance provides the context cues for the slot labels. Furthermore, we also notice that interaction between slot labels also makes sense. When only let each slot label attend to itself and the utterance, we observe the performance drop prob- ably due to the loss of discriminative information among different slot labels.
Model
F1
Liu et al. (2020a)
69.53
Jia et al. (2019)
73.59
Devlin al. (2019)
74.23
Jia and Zhang (2020)
75.19
Wu et (2022)
75.06
Ours (w/o Slot-CL) Ours
74.41
75.29
Table 2: NER results on the target domain (i.c., SciTech News).
3.6 Ablation Study and Analysis
order to better understand our method, we further present some quantitative and qualitative analyses that provides some insights into why our method works and where future work could potentially im- prove it.
Interaction Strategy
F1
Ours (w/o Slot-CL)
49.03
w/o Label Utterance
45.42
w/o Utterance - Label
45.24
w/o Label ++ Utterance
47.25
Inference Speed One advantage of our frame- work is the efficient inference process benefiting from the well-parallelized design. We evaluate the speed by running the model one epoch on the BookRestaurant test data with batch size set to 32. Results in Table 3 show that our method achieves x 13.89 and x 7.06 speedup compared with the ad- vanced metric-based method (i.c., LEONA) and QA-based method (i.e., SLMRC). This could be attributed our batch-wise decoding in parallel. On the one hand, previous metric-based methods use the two-pass pipeline decoding process and instance-win wise slot type prediction. On the other
w/o Label + Label
45.24
Table 4: Comparisons of different label-utterance inter- action strategies for slot filling.
Effect of Context-aware Label Embedding We study the effect of different types of label embed- dings. Figure 3 shows the comparison results. We can see that the proposed context-aware soft label embedding outperforms other purely discrete or decoupled embeddings, including discrete BERT, decoupled BERT or Ve (Pennington et al.,

---

BERERT 3a
GloVe
8B BERT. GGooVe
Few-Shot Setting To verify the effectiveness of our method the few-shot setting where the target domain has a small amount of training examples, we conduct experiments in the 20-shot and 50-shot scenarios. In line with previous works, we take the first K examples in the development set for train- ing named the K-Shot scenario and the remaining keeps for evaluation.
3OBERT aee BBBRTT..+GINV Backbone+Glove
40
(16)
46
W/ Fineruning
WHO Finetuning
Table 5 illustrates that our method achieves su- perior performance compared with other represen- tative metric-based methods. However, we also notice that our method without the slot-level con- trastive learning obtains limited absolute improve- ments data size increase, indicating the slot-level contrastive learning performs better in this case.
Figure 3: Comparisons of different label representa- tions for slot filling. BERTa and de denote the discrete label embedding and the label-utterance decou- pled embedding. respectively. The gray dashed line represents the performance of our model without the slot-level contrastive learning.
2014) embeddings. Interestingly, when fine-tuning, we find that BERT works slightly better than deRT as might be harmful to tune soft label embeddings without utterance contexts. Further- more, we observe significant improvement of our model when incorporating the GloVe static vectors, suggesting that richer label semantics can make positive difference. Meanwhile, the discrete or decoupled label embeddings without fine-tuning may yield better results.
Model
20-Shot (1%)
50-Shot (2.5%)
Coach
64.27
75.51
LEONA PCLC
71.10
76.42
54.32
72.18
Ours
75.16 70.49
82.39
w/o Slot-CL
81.42
Table 5: Fl-scores over all target domains in the 20-shot and 50-shot settings.
Metric Loss for Contrastive Learning Here we explore several typical distance metric func- tions (including Cosine, MSE, Smooth L1, and KL- divergence) for the slot-level contrastive objective, and we also consider the influence of temperature т. Figure reveals that the temperature value directly affects the final performance. Also, shows better results overall at around - = 0.5 for each metric function we take. We select the cosine similarity function as our desired distance metric function, due to its relatively good performance.
Unseen Slot Generalization Since label shift is critical challenge in zero-shot learning, to ver- ify the generalization capacity, we specifically test our method on the unseen target data. Following Liu al. (2022), we split the dataset into the seen and unseen group, where we only evaluate un- seen slot entities during training in the unseen group, while evaluate on the whole utterance the unseen group. From Table 6, our method performs better than other metric-based baselines, showing the superiority of our method for unseen domain generalization.
Cosine -... MSE
KI.-Dvv.. SmoothL,
50
Model
Seen
Unseen
Unseen
(86)
Coach
51.73
34.23
11.66
E1
48
PCLC
58.76
35.08
10.92
LEONA
63.54
40.06
12.32
Ours
65.80
46.85
21.23
0.03
0.05
0.07
0.1
0.5
0.7
1.0
2.0
w/o Slot-CL
64.24
41.04
14.57
Figure 4: Comparisons of different metric functions with varying temperature 7 for slot filling.
Table 6: Fl-scores for seen and unseen slots over all the target domains.

---

Cross-Dataset Setting Considering slot labels and utterances may vary significantly across dif- ferent datasets, we further evaluate the proposed method under the cross-dataset scenario, a more challenging setting. Here we introduce another popular slot filling dataset ATIS (Liu et al., 2019a). It is used for the target (source) domain data while the SNIPS for the source (target) domain data', shown in Table 7. The results confirm that our method still works well in this challenging setting.
test data based on their similarities with prototypes (Snell et al., 2017). For slot filling, the seman- tic embeddings of textual slot descriptions usually serve as the prototype representations (Bapna et al., 2017; Lee and Jha, 2019; Zhu et al., 2020). Shah et al. (2019) utilize both the slot description and a few examples of slot values to learn semantic repre- sentations of slots. Furthermore, various two-pass pipeline schemes are proposed by separating the slot filling task into two steps along with template regularization (Liu et al., 2020b), adversarial train- ing (He et al., 2020), contrastive learning (Wang et al., 2021). linguistic prior knowledge (Siddique et al.. 2021). However, these mostly utilize the context-free discrete label embeddings, and the two-pass fashion has potential limitations due to a lack of knowledge sharing between sub-modules as well as inefficient inference. These motivate to exploit the context-aware label representations under an end-to-end joint learning framework.
Sre-TTg SNIPS ATIS-SNIPS
Coach 14.35
9.76
LEONA 20.80
14.36
Ours
27.01
16.61
Table 7: Fl-scores the cross-dataset setting.
Visualization Figure 5 shows the visualization of normalized slot entity representations before similarity matching using t-SNE dimensionality reduction algorithm (van der Maaten and Hinton, 2008). Obviously, our method can better obtain well well-gatheree gathered clusters when introducing the slot- level contrastive learning, facilitating the discrimi- native entity representations.
Another line of research is the QA- QA-based meth- ods that borrow from prestion-answering systems, relying on manually well-designed queries. Du et al. (2021) use a set of slot-to-question generation strategies and pre-train on numerous synthetic QA pairs. Yu et (2021) and Liu al. (2022) apply the MRC framework (Li et al., 2020) to overcome the domain shift problem. Heo et al. (2022) modify the MRC framework into sequence-labeling style by using each slot label as query. Li et al. (2023) introduce a generative framework using each slot label prompt. In our work, mainly focus the metric-based method without intentionally in- troducing external knowledge with manual efforts.
# tnD IES pdlaee
Contrastive Learning The key idea to learn discriminative feature representations by contrast- ing positive pairs against negative pairs. Namely, those with similar semantic meanings are pushed towards each other in the embedding space while those with different semantic meanings are pulled apart each other. Yan et al. (2021) and Gao et al. (2021) explore instance-level self-supervised con- trastive learning where sample pairs are constructed by data augmentation. Khosla et al. (2020) further explore the supervised setting by contrasting the set of all instances from the same class against those from the other classes. Das et al. (2022) present a token-level supervised contrastive learning solution to deal with the few-shot NER task by means of Gaussian embeddings.
(a) w/ Slot-CL
(b) w/o Slot-CI.
Figure 5: L-SNE visualization of the normalized rep- resentations of different slot entities drawn from the BookRestaurant that contains much unseen slot labels.
4 Related Work
Zero-shot Slot Filling In recent years, zero-shot slot filling has received increasing attention. A dominating line of research is the metric-learning method, where the core idea is to learn a proto- type representation for each category and classify
WWe ignore the evaluation on the SGD (Rastogi et al., 2020), which is fairly large-scale dataset with extremely unhalanced label distributions.
Previous studies for slot filling mainly focus on

---

instance-level contrastive learning, which may be sub-optimal for a finegrraine sequence labeling task. Inspired by supervised contrastive learning, we leverage a slot-level contrastive learning scheme for zero-shot slot filling to learn the discriminative representations for domain adaptation. For all ex- isting slot entities within a mini-batch, we regard those with the same type as the positive example pairs and those with different type negative ones.
References
Ankur Bapna, Gökhan Tür, Dilek Z. Hakkani- and Larry Heck. 2017. Towards zero-shot frame semantic parsing for domain scaling. In INTERSPEECH.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. simple framework for contrastive learning of visual representations. In Pro- ceedings of ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119, pages 1597-1607.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calt- agirone, Thibaut Lavril, Maël Primet, and Joseph Dureau, 2018. Snips voice platform: an embedded spoken language understanding system for private- by-design voice interfaces. CoRR, abs/ 1805. 10190.
5 Conclusion
In this paper, we tackle the problem of general- ized zero-shot slot filling by the proposed end-to- end metric learning based scheme. We propose a cascade-style multi-task learning framework to effi- ciently detect the slot entity from a target domain ut- terance. The context-aware soft label embeddings are shown to be superior to the widely-used discrete ones. Regarding domain adaptation robustness, propose a slot level contrastive learning scheme to facilitate the diseriminative representations of slot entities. Extensive experiments across various do- main datasets demonstrate the effectiveness of the proposed approach when handling unseen target domains. Our investigation also confirms that se- mantically richer label representations enable help further boost the recognition performance, which motivates us further explore externl knowledge enhanced soft label embeddings for advancing the metric-based method.
Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca J. Passonneau, and Rui Zhang. 2022. Container: Few- shot named entity recognition via contrastive learn- ing. In Proceedings of ACL, pages 6338-6353.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of NAACL-HLT, pages
Xinya Du, Luheng Hc, Qi Li, Dian Yu, Panupong Pasu- pat, and Yuan Zhang. 2021. Qa-driven zero-shot slot filling with weak supervision pretraining. In Proceed- ings of ACI, Virtual Evens, August 1-6, 2021, 1, pages 654-664 Association for Computational Linguistics.
Rashmi
Gangadharaiah and Balakrishnan Narayanaswamy. 2019. Joint multiple intent detection and slot labeling for goal-oriented dialog. In Proceedings of NAACL-HLT, pages 564-569.
Limitations
Although our work makes a further progress in the challenging zero-shot slot filling, it is subject to several potential limitations. Firstly, since slot label sequence is used as the prefix of the utter- ance, this directly results in a long input sequence. Secondly, our method may be negatively affected by severe label ambiguity. There are some slot entities with rather similar semantics, leading to wrong slot type predictions. For example, "book a manadonese restaurant", the slot entity type of "manadonese" is actually cuisine, but is easily identified as country. One major reason is that some utterances are relatively short and lack suf- ficient contextual cues. Thirdly, the recognition performance of metric-based methods may remain difficult to exceed that of advanced QA-based or generative methods due to the fact that the latter manually introduces detailed slot label description by well-designed queries or prompts.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. In Proceedings of EMNLP, pages 6894- 6910. Association for Computational Linguisties.
Chih- Wen Goo, Guang-Lai Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung-Chieh Chen, Keng-We Wei Hsu, and Yun- Nung Chen. 2018. Slot-gated modeling for joint slot filling and intent prediction. In NAACL-HLT.
Keqing He, Jinchao Zhang, Yuanmeng Yan, Weiran Xu, Cheng Niu, and Jie Zhou. 2020. Contrastive zero-shot learning for coss-domain slot filling with adversarial attack. In COLING.
Dan Hendrycks and Kevin Gimpel. 2016. Gaus- sian error linear units (gelus). arXiv preprint 1rXiv:1666.08455
Seong-Hwan Heo, WonKee Lee, and Jong-Hyeok Lee. 2022. mobert: Momentum contrastive learning with BERT for zero-shot slot filling. In Interspeech 2022, pages 1243-1247.

---

Chen Jia, Liang Xiao, and Yue Zhang, 2019. Cross- domain NER using cross-domain language modeling. In Proceedings of ACI., pages 2464-2474.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9. 2019.
Chen Jia and Yue Zhang. 2020. Multi-cell composi- tional LSTM for NER domain adaptation. In Pro- ceedings of ACL. pages 5906-5917.
Jie Ma, Miguel Ballesteros, Srikanth Doss, Rishita Anubhai, Sunil Mallya, Yaser Al-Onaizan, and Dan Roth. 2022. Label semantics for few shot named entity recognition. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 1956-1971.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. 2020. Super- vised contrastive learning. In Advances in NIPS
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532-1543. ACI.,
Sungjin Lee and Rahul Jha. 2019. Zero-shot adaptive transfer for conversational language understanding. In In AAAL, pages 6642 6649, AAAI Press.
Libo Qin, Wanxiang Che, Yangming Li, Haoyang Wen, and Ting Liu. 2019. A proppaattoaationn framework with token-level intent detection for spoken language understanding. In EMNLP/IJCNLP.
Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2020. A unified MRC framework for named entity recognition. In Proceed- ings of ACL.
Libo Qin, Fuxuan Wei, Tianbao Xie, Xiao Xu, Wanx- iang Che, and Ting Liu. 2021. Gl-gin: Fast and accurate -on-autoregressvve model for joint multiple intent detection and slot filling. In ACI/IJCNIP.
Xuefeng Li. Liwen Wang. Guanting Dong, Keqing He, Jinzheng Zhao, Hao Lei, Jiachi Liu, and Weiran Xu. 2023. Generative zero-shot prompt learning for cross- domain slot filling with inverse prompting. In Find- ings of the ACL 2023, Toronto, Canada, July 9-14, 2023, pages 025-834. Association for Computational Linguistics.
Libo Qin, Xiao Xu, Wanxiang Che, and Ting Liu. 2020. Towards iine-grained transfer: An adaptive graph- interactive framework for joint multiple intent detec- tion and slot filling. In Findings of EMNLP, pages 1807-1816.
Jian Liu, Mengshi Yu, Yufeng Chen, and Jinan Xu. 2022. Cross-domain slot filling machine reading com- prehension: A new perspective. IEEF/ACM Trans- actions on Audio, Speech, and Language Processing, 30:673-685.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghay Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In AAAI 2020, pages 8689-8696. AAAI Press.
Shichen Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. 2018. Generalized zero-shot learn ing with deep calibration network. In NeurlPS.
Erik Tjong Kim Sang and Fien De Meulder. 2003. In- troduction the conll-2003 shared task: Language- independent named entity recognition. In CoNLL.
Xingkun Liu, Arash Eshghi. Pawel Swietojanski, and Verena Rieser. 2019a. Benchmarking natural lan- guage understanding services for building conversa- tional agents. In IWSDS 2019, volume 714 of Lec- ture Notes in Electrical Engineering, pages 165-183. Springer.
Darsh J. Shah, Raghav Gupta, Amir A. Fayazi, and Dilek Hakkami- Tür. 2019. Robust zero-shot cross- domain slot filling with example values. In Proceed- ings of ACI. pages 5484-5490. Association for Com- putational Linguistics.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Michael Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining ap- proach, arXiv:
A.B. Siddique, Fuad Jamour, and Vagelis Hristidis. 2021. Linguistialll-enrrcheed and context-aware zero-shot slot filling. Proceedings of the Web Confer- ence 2021.
Computation and Language.
Zihan Liu, Genta Indra Winata, and Pascale Fung. 2020a. Zero-resource cross-domain named entity recognition. In Proceedings of RepLANLP ACL, pages 1-6.
Jake Snell, Kevin Swersky, and Richard S. Zemel. 2017. Prototypical networks for shot learning. In Ad- vances in Neural Information Processing Systems 30: Annual Conference on Neural Information Process- ing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4077-4087.
Zihan Liu, Genta Indra Winata, Peng Xu, and Pascale Fung. 2020b. Coach: A coarse-to-fine approach for cross-domain slot filling. In Proceedings of ACL, pages 19-25, Association for Computational Linguis- tics.
Laurens van der Maaten and Geoffrey E. Hinton. 2008 Visualizing data using I-sne. Journal of Machine Learning Research, 9:2579-2605.

---

Andrew Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding al- gorithm. IEEE transactions Information Theory, 13(2):60-266.
(bsz, seq_len)
1bl_ score matmul (normalize (u),
normalize (v). detach
1_sglbl cross_entropy score
1bl_typ)
Liwen Wang, Xuefeng Li, Jiachi Liu, Keqing He, Yuan- meng Yan, and Weiran Xu. 2021. Bridge to target domain by prototypical contrastive learning and la- bel confusion: Re-explore zero-shot learning for slot filling. EMNLP.
1_sglb1 lbl_bd.ne(
utt_score = matmul (normalize (u).
detach (), normalize
1_sgutt cross_entropy,
utt_score, 1bl_typ)
1_sgutt *= 1b1_bd ne (
Di Wu, Liang Ding, Fan Lu, and Xie. 2020. Slotre- fine: A fast non-autoregressive model for joint intent detection and slot filling. In EMNLP.
(bsz, , 1)
1_typ 1_sglbl. sumb (-1)
1 1_sgutt. sum (-1)
filter out paddings and non
Linzhi Wu, Pengjun Xie, Jie Zhou, Meishan Zhang. Chumping Ma. Guangwei Xu, and Min Zhang. 2022. Robust self- augmeeegmetttioonn for named entity recog- nition with meta reweighting. In Proceedings of NAACL 2022, Seattle, WA,, United States, July 10-15, 2022, pages 4049-4060.
slot tokens
f_utt = (roj(r_utter)
filt_ids 1bl_typ != idx_0
filt_emb = f_utt[filt_ids]
filt_typ 1bl_tyo[filt_ids]
repeat on the second dimension inter_emb = filt_emb unsqueeze
Yuanmeng Yan, Rumei Li, Sirui Wang. Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. Consert: A con- trastive framework for self supervised sentence rep- resentation transfer. Proceedings of ACL. pages 5065-5075. Association for Computational Linguis- tics.
(1) repeatt1,, num_slot, view nun_slot=num_slot -1)
int_typ filt_typ unsqueeze (1),
repeat num_slot) view (num_slot* num_slot)
repeat the first dimension rept_emb filt_emb (nsqueeze
1). nun_slot=num_slos -1)
repeat
Mengshi Yu, Jian Liu, Yufeng Chen, Jinan Xu, and Yujie Zhang. 2021. Cross-domain slot filling as machine reading comprehension. In ICCAI
rept_typ filt_typ unsqueeze (0)
repeat(nun_slot, 1). (num_slot*
(um_slot)
Xiaodong Zhang and Houfeng Wang. 2016. A joint model of intent determination and slot filling for spo- ken language understanding. In IJCAI.
sim_score cosine_similarity
inter_emb. rept_emb)
view as (num slot, num_slot) denom_mask (inter_emb rept_emb)
Su Zhu, Zijian Zhao, Rao Ma. and Kai Yu. 2020. Prior knowledge driven label embedding slot filling in natural language understanding. IEEE ACM Trans. Audio Speech Lang. Process., 28:1440-BP 111140--4451.
numer_mask denom (int_typ rept_typ)
loss softmax (attmaxxssimsrreee / temperature
A
Appendix
# calculate SLot-CL with Eq. (bsz, 1)
A.1
Code Implementation
l_ctr = -(loss*numer (anssnummm___sskk log (loss=denom) dessod__s_____))..... mask) num_mask. sum log
log ()
Here we present the core pseudo code of the pro- posed method.
l_ctr l_ctr mean
class EndzendSLUTagger (nn. Module):
return 1_bdy
1_typ + 1_ctr
def (bb num_type,
Listing 1: Pseudo code for our proposed method.
1bl_bd, lbl_typ):
# (bsz. seq_len. hsz)
bert_repr bebb(tbert_inp)
r_utter bert_repr I: type +1:]
r_label bert_repr D: n:numttype
+1]
# label adapter in Eq.5
v (r_label) r_label
h_utter = 1stm(r_utter)
e proj
r_bound = matmul matm (e), bound_emb. weight)
# (bsz, 1)
1_bdy = crf(e, 1bl_bd)
u = proj (cnnatt (token_repr-
r_bound))


Reviews:
[{"review_id": "jNAgbVUePJ", "reviewer": "Reviewer_aDbx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful and detailed reviews. \n\n*R1: “comprehensive comparison with other existing soft label embedding methods”*\n\nThe soft label embeddings in this work are closely derived from the framework design (e.g., need interact with the utterance context representations). Since they are not isolated components, changing to other existing soft label embedding scheme may not fit our framework well and need to adjust the overall structure, which may lead to unfair comparisons among different soft label embedding methods. There may be more powerful and suitable soft label embeddings to further improve our current  model, we are very willing to explore that in depth. Thanks for your insightful suggestions. \n\n\n*R2: “consider including state-of-the-art methods that have emerged”*\n\nThanks for your suggestion. We will complement the latest advancements (e.g., prompt-based generative baseline methods) to provide a more comprehensive evaluation in the final version.\n\n\n*R3: “lacks a thorough explanation for choosing a 1:1:1 weighting for these terms”*\n\nAs a cascade-style framework, the impact of each module is closely correlated. The accuracy of slot boundary prediction directly affects the effect of type prediction, and meanwhile the accuracy of type prediction determines the final recognition performance. Thus, these two parts (i.e., L_{bdy} and L_{typ}) of training objective are considered equally important. L_{ctr}, though as an auxiliary loss, plays a key role in improving domain adaptation capability. Our preliminary ablation experiments (not presented in the original paper) also showed the loss weights of 1:1:1 in Eq (9) could yield relatively better performance (as listed in the following table):\n| **$L_{bdy}$: $L_{typ}$: $L_{ctr}$** | **F1** |\n|-----------------------|:------------------:|\n| 1: 1: 1                   | 49.67              |\n| 1: 0.5: 0.5               | 46.42              |\n| 1: 0.5: 0.1               | 45.34              |\n| 1: 0.1: 0.5               | 41.30              |\n| 1: 1: 0.5                 | 47.72              |\n\nWe will complement more detailed explanation about the loss function design in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper proposes a novel end-to-end metric learning scheme specifically tailored for zero-shot slot filling, which is more efficient and effective compared to existing metric-based methods. This new approach offers a fresh perspective and contributes to advancing the field of slot filling.\n2) The introduction of context-aware soft label representations presents a significant improvement over commonly-used static label representations. This innovation showcases the paper's ability to enhance the performance of the model through novel representation techniques.\n3) The investigation and incorporation of slot-level contrastive learning demonstrate the paper's comprehensive exploration of different learning strategies to effectively improve the generalization capacity for zero-shot slot filling.", "reasons_to_reject": "1) While the paper introduces context-aware soft label representations and shows their superiority over commonly-used static label representations, it lacks a comprehensive comparison with other existing soft label embedding methods. A more thorough analysis of various soft label embedding techniques would enhance the credibility of the proposed approach.\n2) The comparison in Table 1 is made against relatively outdated baseline methods. To provide a more meaningful evaluation, the paper should consider including state-of-the-art methods that have emerged since the development of the paper. This would help in understanding how the proposed approach performs in comparison to the latest advancements in the field.\n3) The paper introduces a loss function in Equation (9) with three terms, but lacks a thorough explanation for choosing a 1:1:1 weighting for these terms. The justification for this specific choice is not adequately provided, raising questions about the rationale behind the loss function design.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on the task of slot filling and its recent developments using deep learning and large-scale annotated data. The authors address the critical challenge of handling novel domains in slot filling, where samples from these domains were not seen during training, leading to degraded recognition performance due to domain shifts. The main contributions of the paper lie in the design of a more efficient and effective end-to-end metric learning scheme for zero-shot slot filling, the introduction of context-aware soft label representations, and the exploration of slot-level contrastive learning to enhance generalization capacity. The experimental evaluation further validates the superiority of the proposed approach and contributes valuable insights into the task of slot filling.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "p9cphHwYQA", "reviewer": "Reviewer_MsTt", "summary": "", "questions": "I hope the author can response to the experiment problems. If the author can provide more results on other datasets or give a clear explaination about using only SNIPS dataset, I would be inclined towards improving the **soundness** score.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed reviews.\n\n*R1: \"provide more results on other datasets\"*\n\nWe conduct experiments with the cross-dataset setting where our model is trained on one dataset and evaluated on another one. Due to deadline time limitation, we just verify our method on SNIPS (multi-domain) and ATIS (single-domain). The results (F1) are shown as follows:\n\n| **Model**            | **SNIPS->ATIS** | **ATIS->SNIPS** |\n|---------------------|-----------------|-----------------|\n| Coach               | 14.35           | 9.76            |\n| LEONA               | 20.80           | 14.36           |\n| Ours | 25.40           | 14.49           |\n\nThe results demonstrate our method can still work in the challenging cross-dataset setting.\n\nWe will complement the complete experiments of cross-dataset settings (including large-scale multi-domain dataset SGD) and provide detailed analysis in our final version.\n\n*R2: “some minor writing suggestions”*\n\nThanks for pointing out these. We will clear these writing issues and update in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The proposed end-to-end scheme for zero-shot slot filling demonstrates a notable advance over conventional two-stage methods, showcasing enhanced simplicity, elegance, and notably improved efficiency. The zero-shot performance and the inference speed are both satisfactory.\n\n(2) In addition to the overall end-to-end pipeline, the author also propose a soft context-aware label embedding and a slot-level contrastive learning, to further improve the zero-shot performance.\n\n(3) The experiments is extensive and verify the effectiveness of the approach.", "reasons_to_reject": "For **experiment**, I notice LEONA (Siddique et al. 2021) also adopt other datasets (e.g. ATIS, SGD) to evaluate the slot-fliing task. So I recommend the author to provide more results on these datasets.\n\nAlso, I believe there are some minor **writting** suggestions I can offer.\n\n(1) The alignment between the module names in Figure 1 and the corresponding titles in the main text appears to be less than optimal. For instance, 'Context-aware Label Embedding' and 'Slot Boundary Detection' do not possess direct visual linkages within the model framework. Such misalignment could potentially impede readers, especially those who are less acquainted with the task, and necessitate additional efforts to locate pertinent information.\n\n(2) Source slot type number can be added in line121 since you specify the target slot type number.", "questions_to_authors": "I hope the author can response to the experiment problems. If the author can provide more results on other datasets or give a clear explaination about using only SNIPS dataset, I would be inclined towards improving the **soundness** score.", "paper_topic_and_main_contributions": "The main contributions of this paper can be summarized as follows: (1) Compared with existing metric-based methods, this work propose a more efficient and effective end-to-end scheme for zero-shot slot filling, and show the proposed soft label embeddings perform much better than previous commonly-used static label representations. (2) This work investigate the slot-level contrastive learning to effectively improve generalization capacity for zero-shot slot filling.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None.", "Typos_Grammar_Style_and_Presentation_Improvements": "Please refer to the writing issues in **Reasons To Reject**."}, "extra_text": {}}, {"review_id": "Y0VaYUEODu", "reviewer": "Reviewer_yN6g", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed reviews.\n\n*R1: “describe the background of the paper and motivation of this paper in detail”, “ analyze their short shortcomings more clearly”*\n \nThe existing competitive metric-based methods mainly have the following shortcomings:\n+ Most of them adopt the two-pass pipeline fashion (i.e., train boundary and typing modules separately), which motivates us to design an end-to-end framework for efficient inference and knowledge sharing between sub-modules.\n+ Most of them tend to utilize discrete and context-free label embeddings (i.e., solely encode each label name and lack of interaction with the utterance context), which motivates us to utilize the context-aware soft label embeddings with rich semantics as the supervision information.\n+ Regarding domain adaptation capacity, instance-level contrastive learning has been widely explored before, which is sub-optimal to slot filling as a fine-grained labeling task. Considering slot entities tend to embody domain-specific nature in contrast to the counterpart context, we investigate the slot token level contrastive learning scheme to learn the discriminative representations of slot entities for domain adaptation.  \n\nMainly inspired by the above several key points, we ultimately propose our improved method. We will clarify our motivation more clearly in our introduction and related work sections in the final revision.\n\n*R2: “The figure 2 is ambiguous...”*\n\nThanks for pointing out this. We will refine the Figure 2 to well group the key components and give more detailed descriptions of the figure. All these will be updated in final version.\n\n*R3: “add the Pseudo code of proposed method”*\n\nThanks for your suggestion, we will complement the pseudo code to help better understand the implementation ideas in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) This paper is well-written and the proposed idea is very interesting!   \n(2) Experiments on benchmark datasets also demonstrate the superiority of the proposed method. ", "reasons_to_reject": "(1)\tFirst of all, the authors should describe the background of the paper and motivation of this paper in detail. In essence, there are many deep learning-based methods [1-2] which leverages deep learning to solve the zero-shot cross-domain slot filling or recognition problem, authors should analyze their short shortcomings more clearly, and how do these shortcomings promote the creation of the new method.   \n(2) The figure 2 is ambiguous, and the authors should explain more details about the individual one and its impact on output production.  \n(3) It would be better to add the Pseudo code of proposed method if possible.\n\n[1] Shah D J, Gupta R, Fayazi A A, et al. Robust zero-shot cross-domain slot filling with example values[J]. arXiv preprint arXiv:1906.06870, 2019.  \n[2] Shi B, Wang L, Yu Z, et al. Zero-shot learning for skeleton-based classroom action recognition[C]//2021 International Symposium on Computer Science and Intelligent Controls (ISCSIC). IEEE, 2021: 82-86.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper re-examines the typical metric-based methods, and proposes a new adaptive end-to-end metric learning scheme for the challenging zero-shot slot filling. And the experimental results also demonstrate the effectiveness of our method. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles zero-shot cross-domain slot filling, a challenging but meaningful setting. It proposes a novel adaptive end-to-end metric learning method that includes context-aware soft label representations and slot-level contrastive learning. Experiment results on the SNIPS dataset as well as under cross-dataset settings (from rebuttal) show competitive performance of the proposed method.

Strengths:
- Innovative, well-motivated method designs
- Clear and stylish writing
- Competitive empirical results

Weaknesses:
- As the reviewers pointed out, the clarity in several places, including motivation, could still be improved. the authors are strongly encouraged to revise accordingly in the future revision.
Decision: Accept-Main