HierarchicalContrast: A Coarse-to-Fine Contrastive Learning Framework for Cross-Domain Zero-Shot Slot Filling
Authors: Junwen Zhang
Yin Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: slot-filling
task-oriented dialogue
contrastive learning
cross-domain adaption
zero-shot learning

Abstract: In task-oriented dialogue scenarios, cross-domain zero-shot slot filling plays a vital role in leveraging source domain knowledge to learn a model with high generalization ability in unknown target domain where annotated data is unavailable. However, the existing state-of-the-art zero-shot slot filling methods have limited generalization ability in target domain, they only show effective knowledge transfer on seen slots and perform poorly on unseen slots. To alleviate this issue, we present a novel Hierarchical Contrastive Learning Framework (HiCL) for zero-shot slot filling. Specifically, we propose a coarse- to fine-grained contrastive learning based on Gaussian-distributed embedding to learn the generalized deep semantic relations between utterance-tokens, by optimizing inter- and intra-token distribution distance. This encourages HiCL to generalize to the slot types unseen at training phase. Furthermore, we present a new iterative label set semantics inference method to unbiasedly and separately evaluate the performance of unseen slot types which entangled with their counterparts (i.e., seen slot types) in the previous zero-shot slot filling evaluation methods. The extensive empirical experiments on four datasets demonstrate that the proposed method achieves comparable or even better performance than the current state-of-the-art zero-shot slot filling approaches.

HierarchicalContrat::: A Coarse-to-Fine Contrastive Learning Framework for Cross-Domain Zero-Shot Slot Filling
Junwen Zhang and Yin Zhang College of Computer Science and Technology Zhejiang University
(junwenzhang, vinzh)ezju.edi edu. cn
Abstract
B Unuorn Sluts
91.02
00.32
Sern Slots
54.12
task-oriented dialogue scenarios, cross- domain zero-shot slot filling plays vital role in leveraging source domain knowledge to learn a model with high generalization ability un- known target domain where annotated data is unavailable. However, the existing state- of-the-art zero-shot slot filling methods have limited generalization ability in target domain, they only show effective knowledge transfer seen slots and perform poorly unseen slots. To alleviate this issue, we present novel Hierarchical Contrastive Learning Framework (HiCL) for zero-shot slot filling. Specifi- cally, we propose a coarse- to faif---ggiii con- trastive learning based on Gaussian-distre distribbbed embedding to learn the generalized deep se- mantic relations between petraance-tokens. by optimizing inter- and intra-token distribution distance. This encourages HiCL to generalize slot types unseen at training phase. Further- more, we present
50
65.84
62.84
Slot Fi. Soore
40-
20
14.2
2.11
3,0
CT
RET
Clach
PCLC
LRONA
TOp MBKT
matEKT
Ost
Molel
Figure Cross-domain slot filling models' perfor- mance unseen and seen slots in GetWeather target domain on SNIPS dataset.
Figure 2, given a user utterance "send a reminder for a tire check next week" belonging to reminder domain, the slot filling task is to identify slot enti- ties: "a tire check" and "next week" that correspond to slot types (an alias of slot type is slot), todo and date s_mee, respectively.
iterative label sel mantics inference method to unbiasedly and separately evaluate the performance of unseen slot types which entangled with their counter- parts (i.e., seen slot types) in the previous zero- shot slot filling evaluation methods. The ex- tensive empirical experiments' on four datasets demonstrate that the proposed method achieves comparable
Supervised slot filling methods (Kurata et al.. 2016; Wang al., 2018; Li al., 2018; Goo et al., 2018; Qin al., 2019) have achieved promis- ing performance. Nevertheless, these methods are strongly dependent on substantial and high-quality annotation data for each slot type, which prevents them from transferring to new domains with little or no labeled training samples.
better performance than the current state-of-the-art zero-shot slot filling approaches.
1 Introduction
Slot filling models are devoted to extracting the con- tiguous spans of tokens belonging to pre-defined slot types for given spoken utterances and gathering information required by user intent detection, and thereby are an imperative module of task-on task-oriented dialogue (TOD) systems. For instance, as shown in
To solve this problem, more approaches have emerged to deal with this data scarcity issue by leveraging zero-shot slot filling (ZSSF). Typically, these approaches can be divided into two cate- gories: one-stage and two-stage. In one-stage paradigm, Bapna al. (2017); Shah et al. (2019); Lee and Jha (2019) firstly generate utterance repre- sentation in token level to interact with the repre- sentation each slot description in semantic space,
*Corresponding author 'The offcial implementation of HiCI. is available at https///gtuub.comaaag/////

---

Step One
Enity-Labail Level Contrastive Learinng
Send jominder for (Stkee check weee
Breader
Trandfrrudaan
Semanntc Space
Trandorenaion
UM Broaker
Ishat 0
the weather
G2 pou wookend
Indo date, time
(Taitgh shrg))
Iocation
cite Simne
Coarse Find
Step Two
Taken-Labal Lavel Contrantive Learning
sendi reminder Tor of (recheck/eed) treak
Dicader
Transformaton
Transformation (attuldd
UM Fathpp
Whhtt weather
tile weekond
0
Brodo
fflogld shull
Aecuttoo
E-scatiion di Broe
Figure 2: Hierarchical contrastive learning (CL), where grarse-grained and fine-grained slot labels are used supervised signal for CL, respectively, i.e., step one is entity-label level CL and step two is token-label level CL. Entity-label is pseudo-label in our Hierarchical CL. Different colors rectangular bounding box denote different slot types.
and then predict each slot type for utterance to- ken individually. The primary weakness for this paradigm is multiple prediction issue where a token will probably be predicted multiple slot types (Liu et al., 2020; He et al., 2020). To overcome this issue, Liu al. (2020): He al. (2020); Wang al. (2021) separate slot filling task into two-stage paradigm. They first identify whether the tokens in utterance belong to BIO (Ramshaw and Mar- cus, 1995) entity span or not by binary classifier, subsequently predict their specific slot types by pro- jecting the representations of both slot entity and slot description into the semantic space and inter- act each other. Based the previous works, Siddique al. (2021) propose two-stage vari- ant, which introduces linguistic knowledge and pre- trained context embedding, along with the entity span identify stage (Liu et al., 2020), to promote the effect on semantic similarity modeling between slot entity and slot description. Recently, Heo et al. (2022) develop another two-stage variant that ap- plies momentum contrastive learning technique to train BERT (Devlin al., 2019a) and to improve the robustness of ZSSF. However, shown in Fig- ure we found that these methods perform poorly on unseen slots the target domain.
performance. Subsequently, their variants (Sid- dique et al., 2021; Heo et al., 2022: Luo and Liu, 2023) still struggle in the actual unseen slots per- formance evaluation due to following this biased test set split of unseen slots (Siddique et al., 2021; Heo et al., 2022), the intrinsic architecture limit of one-pass inference (Luo and Liu, 2023). In an- other line (Du al., 2021; Yu et al., 2021), ZSSF is formulated as a one-stage question answering task, but is heavily reliant upon handcrafted question templates and ontologies customized by human experts, which is prohibitively expensive for this method to generalize to unseen domains. Besides, the problem with multiple slot types prediction that happened to them (Siddique et al., 2021; Heo al., 2022; Du et al., 2021; Yu al., 2021) seriously degrades their performance. In this paper, we intro- duce new iterative label set semantics inference method to address these two problems.
Moreover, a downside for these two orthogo- nal methods is that they are not good at learning features (e.g., generalized token- class properties) by making full use of source- domain data while keeping target-domain of in- terest unseen. This may lead them to overfit to the limited slot types in source training domains. Ac- tually the current models' performance on unseen slots in target domain is still far from upper bound.
Although two-stage approaches have flour- ished, one-pass prediction mechanism of these ap- proaches (Liu et al., 2020; He et al., 2020; Wang al., 2021) inherently limit their ability infer un- seen slots and seen slots separately. Thus they have to adopt the biased test set split method of unseen slots (sce more details Appendix F), being inca- pable faithfully evaluating the real unseen slots
To tackle the above limitation, intuitively, through contrastive learning (CL), can distribute the distance of token embeddings in mantic space to learn generalized token-class fea- tures across tokens, and better differentiate between token classes, and even between new classes (slot-

---

Fine-uriig
Log Libelthood
Pinn-Conttruttii
CAY
Eiturekicad Cuntrast
Eatiy-Srrll CL
Tobnn-lovdi CL.
Tober-Lovel
Jinnac
Transrrmation
Trassfeuuttion Network If Fi)
Trunsformtion Netweck If fe)
Euboodäags
Cuntestual
Tolio Eacheddag
Zccrea divrgencs
Contontual Ehoxder
Transformer Layer (x 12)
Soodo
E-4at, Stns
Heestton
Embi
Emii
Buck
pay
Input
all Lucpel slat typw
label a
label act
utlacmnnott
Iabel
labed wot 2
unternnco)
puaal
Sodo
lecalion
done Jino
Pecatioe not somentice
Figure 3: The main architecture of HiCL. For simplicity, we only draw detailed illustration for fine- ane-grained token- level CL. Different colors of rectangular box utterances and token-level CL (right side) denote different slot types.
agnostic features), which beneficial for new to- ken class generalization across domains (Das et al., 2022). However, it's hard to train token-level class since its supervised labels are closely distributed, which easily leads misclassifications (Ji al., 2022). While training entity-level class relatively easy since its training labels are dispersedly dis- tributed and it does not require label dependency that exists in token-level sequence labeling training (Ji et al., 2022).
Gaussian embedding to learn the representations of tokens (Vilnis and McCallum, 2015; Mukherjee and Hospedales, 2016; Jiang et al., 2019a; Yüksel et al., 2021) due to their superiority in capturing the uncertainty in representations, This motivates us employ Gaussian embedding in our HiCL represent
-tokens more robustly, where each token becomes density rather than a sin- gle point in latent feature space. Different from existing slot filling contrastive learners (He et al., 2020; Wu et al., 2020; Wang et al., 2021; Heo et al., 2022) that optimize training objective of pair- wise similarities between point embeddings, HiCL aims optimize distributional divergence by lever- aging effectively modeling Gaussian embeddings. While point embeddings only optimize pairwise distance, Gaussian embeddings also comprise ad- ditional constraint which preserves the class dis- tribution through their variance estimates. This distinctive quality helps to explicitly model entity- or token- class distributions, which not only en- courages HiCL to learn generalized feature repre- sentations to categorize and differentiate between different entity (token) classes, but also fosters zero- sample target domain adaptation.
We argue that entity-level class knowledge con- tributes to token-level class learning, and their com- bination is beneficial for ZSSF task. Entity-level class learning supplements token-class learning with entity type knowledge and boundary informa- tion between entity and non-entity, which lowers the difficulty of token class training.
Hence, we propose new hierarchical CL frame- work called HiCL, coarse-to- fine CL approach. As depicted in Figure 2, it first coarsely learns entity-class knowledge of entity type and boundary knowledge via entity-level CL. Then, it combines features of entity type and boundary, and finely learns token-class knowledge via token-level CL. In recent years, some researchers have employed

---

Concretely, as shown in Figure 3, our token-level CL pulls inter- and intra-token distributional dis- tance with similar labels closer in semantic space, while pushing apart dissimilar ones. Gaussian dis- tributed embedding enables token-level CL to bet- ter capture semantics uncertainty and semantics coverage variation of token-class than point em- bedding. This facilitates HiCL to better model generalized slot-agnostic features in cross-domain ZSSF scenario.
label sequence 0 {or, Onl. where is the length of
3 Methodology
The architecture of HiCL illustrated in Figure 3. HiCL adopts an iterative label set semantics inference enhanced hierarchical CL approach and conditional random field (CRF) scheme. Firstly, HICL successively employs entity-level CL and token-level CL. optimize distributional diver- gence between different entity- (token-) class em- beddings. Then, HiCL leverages generalized entity (token) representations refined the previous CL steps, along with the slot-specific features learned in the CRF training with BIO label, to better model the alignment between utterance-tokens and slot- entities.
Our major contributions are three-fold:
We introduce a novel hierarchical CL (coarse-to-fiind fine CL) approach based on Gaussian embedding to learn and extract slot- agnostic features across utterance-tokems, ef- fectively enhancing the model's generaliza- tion ability identify new slot-entities.
3.1 Hierarchical Contrastive Learning
We find unseen slots and seen slots overlap- ping problem in the previous methods for un- seen slots performance evaluation, and rec- tify this bias by splitting test set from slot type granularity instead of sample granularity, thus propose
Encoder Given an utterance of n tokens U {xi) 12 target slot type with k tokens S { and all slot types except the target slot type with g tokens A = {as } we adopt pre-trained BERT (Devlin et al., 2019a) as the encoder and feed
iterative label set seman- ties inference method train and test unseen slots separately and unbiasedly. Moreover, this method also designed to relieve the multiple slot types prediction issue.
as input to the encoder to obtain di-dimensional hidden representation h, R4 of cach input instance:
H = BERT (BRTTCCHSSSSBPPPASPPPPMM (1)
Experiments on two evaluation paradigms, four datasets and three backbones show that, compared with the current state-of-the-at (SOTA) models, proposed HICI. frame- work achieves competitive unseen slots per- formance, and overall performance for cross- domain ZSSF task.
SnA== 0. We adopt two encoders for our HiCL, i.c.. BiL- STM (Hochreiter and Schmidhuber, 1997) and BERT(Devlin et al., 2019a), to align with base- lines.
where H h+ | H
Gaussian Transformation Network We hy- pothesize that the semantic representations of en- coded utterance-tokens follow Gaussian distribu- tions. Similar to (Jiang et al., 2019a; Das et al., 2022), we adopt exponential linear unit (ELU) and non-linear activation functions fa and fx to gen- erate Gaussian mean and variance embedding, re- spectively:
2 Problem Definition
Zero-shot Setting For ZSSF, a model is trained in source domains with a slot type set {AA% and tested in new target domain with a slot type set (A'' ( U { where i.j and k are in- dex of different slot type sets, Ale are the slot types that both exist in source domains and target domain (seen slots), and Afu are the slot types that only cxist in target domain (unseen slots). Since {Ar) {Att 0. is a big challenge for the model to generalize unseen slots in target do- main.
14: fu(h)) Σ, ELU((s((h))
(2)
where 1 Id is an array with all value set 1, E..U and 1 are designed to ensure that every
Task Formulation
Given an utterance
element of variance embedding is negatvgative. #4 E RRd Σ, E d denotes mean and diagonal
{ [z1...., }n the task of ZSSF aims to output a

---

covariance matrix of Gaussian embeddings, respec- tively. frx and f are both implemented with ReLU and followed by one layer of linear transformation. Positive and Negative Samples Construction In the training batch, given the anchor token Tu. token 2, shares the same slot label with token Tu. i.e., y/- = y/u. then T. is the positive example of Zu. Otherwise, if yo A Yus is the negative example of Tu.
where [A7Ao]] {0, 10 is an indicator function eval- uating 1 iff lo + u, is scalar temperature parameter and nu is the total number of positive instances in Tu-
Coarse-grained Entity-level Contrast goarse-grained CL, entity-level slot labels are used as CL supervised signals in training set Ta. Coarse- grained CL optimizes distributional divergence be- tween tokens Gaussian embeddings and models the entity class distribution. According E3-(3)- Eq.(6), we can obtain coarse- goarse-graieed entity-level contrastive loss Leaurse. and the in-batch coarse- grained CL loss formulated:
Contrastive Loss Given a pair of positive sam- ples Iu and Iys their Gaussian embedding dis- tributions follow Ta N (Nu. Σn) and Ze ~ (Hr: E,), both with m dimensional. Following the formulas (Iwamoto and Yukawa, 2020; Qian et al., 2021), the Kullback-Leibler divergence from N( Ex) to NooNNoo Ex) is calculated as below:
N.
1 No
Loarree
Cewarse
DKI,WW/ING]] DKI. DKK,(N((HL, (Hu En)lIN En) N (Muur Eu)N(Hy Ex
Fine-grained Token-level Contrast In fine- grained CL, token-level slot labels are used CL supervised signals in training set Ta- As illus- trated in Figure fine- lineggrained CL optimizes KL- divergence between tokens Gaussian embeddings and models the token class distribution. Similarly, the in-batch fine mne-grained CL loss is formulated:
(m (m)) E um)
|Er
+ log
m7
Σn|
No
+ T. E En)
(3)
fine
No
1
Σ
fine
(8)
1=1
3.2 Training Objective
where T, is the trace operator. Since the asymmetry features of the Kullback-Leible divergence, follow the calculation method (Das et al., 2022),
The training objective L is the weighted sum of regularized loss functions. Slot Filling Loss
and calculate both directions and average them:
1
m2
s(u, v) 2 (DKI[WalIAG) DRL[AG//WW)) (4)
C.:=
ΣΣ :=
log
v/
Suppose the training set in source domains is Ta, at each training step, a randomly shuffled batch T E
where 8J is the gold slot label of j-th token and ny is the number of all slot labels. Overall Loss
To has batch size of N₁, each sample (x/z My) T. For each anchor sample Ta, match all positive instances Ta E for F. and repeat for all anchor samples:
L L L= + B.euarse + yL + 1||
(10)
where a, 3 and y are tunable hyper-parameters for each loss component, À denotes the coefficient Ig-reuularivation, O represents all trainable param- eters of the model.
Tn = ( Vn) E T yo = yu, u t
(5)
Formulating the Gaussian embedding loss in each batch, similar to Chen al. (2020), calculate the NT-Xent loss:
Experiments
4.1 Dataset
xxp(-s(u,5)/T)
We evaluate our approach on four datasets, namely SNIPS (Coucke et al., 2018), ATIS (Hemphill al., 1990), MIT_corppp (Nie et al., 2021) and SGD (Rastogi et al., 2020).
log
K
Σ
I1 [RsSu expl
k/),T
k=
(6)

---

Training Sciing
Training Setting
Zero-shue
Mribol
Dvansin
AF
CN
PM1
R5
SCW
FSE
IWOFT
Method
Domain =
Movie
Restaurant
AVG
C1
18.05
31.34
06.45
12.56
14.54
39
16:7
30.55
42.77
10.06
10.38
12.12
16.43
64.05
12:24
17.89
TOD BERT
71.72
48.23
59.90
RNN
Coaca
PCTE3
CZSL
50.90
19.34
53.89
34.01
34.06
16:16
50.47
12.34
14:21
12.01
34.59
14.90
22.06
11.53
79.31
55.61
50.01
46.05
17
55.63
30.05
57.39
60.99
63:37
BERT based
mcBERT*
HiCL+BERT (ours)
77.75
76.51
57.37
58.35
66.94
68.05
BiCL.BILSSM (oavs)
53.16
30.97
55.78
35.13
27.16
54.00
36.51
41.63
w/o course CL fours)
74.99
51.23
63.11
61M3 bane
HICL-ELMo lours)
50.35
52.86
48.57
06.38
02:01
64.35
60.00
12.67
27.38
19.96
65.00
30.23
10
62.41
45.14
w/o line CL (ours)
75.15
57.63
66.39
TOD-BRT
07.26
11.11
14.10
25.00
02235
11
15.00
14.18
18:88
19:71
6977
81.19
BCSF
68:70
Table 3: Slot Fl scores on MIT _orpus dataset for dif- ferent target domains that are unseen in training.
BECL+BERT (sus))
77.91
4335
53.95
12.44
1500
46:21
6484
54.9
WAB Ine (mun))
55.10
53,68
77.44
44.94
34.14
70.03
40.54
53.76
Table 1: Slot Fl scores on SNIPS dataset for different target domains that are unseen in training- denotes the results reported in (Wang et al., 2021). denotes that we run the publicly released code (Siddique et al., 2021) to obtain the experimental results and denotes that we reimplemented the model. AP, BR, GW, PM, RB, SCW and FSE denote AddToPlaylist, BookRestau- rant, GerWeather, PlayMusic, RateBook, SearchCre- Work and FindScreeningEvent, respectively. AVG denotes average.
Training Setting
Zero
Method & Dumain -
Bases
Evenss
Homes
Remaal
PV
TOD-BERT
35,04
56.43
79.02
53.40
56.30
AuBERT
51.38
80.14
59.43
54.52
BERT based
HICT-RRERT (oum)
27.63
50:30
81.73
90.34
ITD--TOD- (oing)
28.07
58.29
$4.51
55.55
36.60
weo course CL (ouns)
28.44
53.62
83.45
53.17
53.80
CI (mun)
20.36
45.09
77.81
56.39
52.39
Table Slot scores on SGD dataset for different target domains that are unseen in training. BS, ET, HE, RC denote Buses, Events, Homes, Rental Cars, respectively.
Trainin Setting
Mothod
Domain
AR
OS
AVC
MIT_ corpus (Nie et al., 2021).
LEONA" HICL ML (ouss)
$1.36
96.56
93.11
84.05
50.93
$4.50
76.03
50.64
98.34
93.24
86.19
52.47
$4.63
73.58
TOTAEESCY
2427
96.4
17.0
35.50
M177
75.50
75.42
mcBERT*
38,10
14.42
K1.35
4.4 Baselines
BERT Iuwee
HICL-BERT
92.20
85.96
76.17
K2.41
(ours)
05.25
94.30
5750
69.66
71.08
81.19
We compare the performance of our HiCL with the previous best models, the details of baseline models are provided in Appendix E.
(oand
62.85
98.00
8791
81.04
64.00
84,62
Table 2: Slot Fl scores on ATIS dataset for different target domains that are unseen in training. AR, AF, AL, FT, GS and OS denote Abbreviation, Airfare, Airline, Flight, Ground Service, Others, respectively.
4.5 Training Approach
Training Sample Construction The output of HiCL is a BIO prediction for each slot type. The training samples are of the pattern A,, V() where S, represents a target slot type, A, repre- sents all slot types except S, 24, represents an utter- ance, D/ represents BIO label for S,, all slot types A = S, Ar, for simple depiction, hicrarchical CL labels are omitted here. For a sample from given dataset with the pattern (1(;)% that contains enti- ties for k slot types, k positive training samples for U; can be generated by setting each of k slot types as S, in turn and generating the corresponding A, and y/. Then m negative training samples for U4; can be generated by choosing slot types that be- longs to A and does not appear in Ug For example, in Figure 3, the utterance "what the weather in st paul this weekend" has the original label 0 O O B locatioon llccctio B-date_ting I-data_tinge time". The positive samples are formatted as ["location", 0 0 O B I O"] and ['date time".
4.2 Unseen and Seen Slots Overlapping Problem in Test Set
The problem description, and the proposed rectified method for unseen and seen slots test set split are presented in Appendix F.
4.3 Evaluation Paradigm
Training on Multiple Source Domains and Test- ing on Single Target Domain A model is trained on all domains except a single target domain. For instance, the model is trained on all domains of SNIPS dataset except a target domain GetWeather which is used for zero-shot slot filling capability test. This multiple training domains towards sin- gle target domain paradigm is evaluated on datasets SNIPS (Coucke et al., 201 8), ATIS (Hemphill et al., 1990) and SGD (Rastogi et al., 2020).
Training on Single Source Domain and Testing on Single Target Domain A model is trained on single source domain and test on a single target domain. This single training domain towards single target domain paradigm is evaluated on dataset of
00000000 I"]. While the negative samples are formatted ["todo". O 0 O O"] and ["attendee", O
'000
Iterative Label Set Semantics Inference
Itera-

---

tively feeding the training samples constructed in $ 4.5 into HiCL, the model would output BIO label for each target slot type. We named this training or predict paradigm iterative label set semantics inference (ILSST). Algorithm and in Appendix elaborate on more details of ILSSI.
grained CL with entity-level boundary information and entity type knowledge, while fine-grained CL complements caarse-grained CL with token-level boundary information (BIO) and token class type knowledge. Their combination is superior to either of them and helps HiCL to obtain better perfor- mance.
5 Experimental Results
Unseen Slots vs. Seen Slots As shown in Table 6, 7, 8, 9, the proposed HiCL siggifiigyl improves the performance on unseen slots against the SOTA models, while maintaining comparable seen slots performance, which verifies the effectiveness of our HiCL framework in cross domain ZSSF task. From the remarkable improvements on unseen slot, we clearly know that, rather than only fitting seen slots in source domains, our model have learned gen- eralized slot-agnostic features of entity- classess in- cluding unseen classes by leveraging the proposed hierarchical contrast method. This enables HiCL to effectively transfer domain-invarian of slot types in source domains to unknown target domain.
5.1 Main Results
We examine the effectiveness of HiCL by compar- ing it with the competing basclines. The results of the average performance across different target de- mains on dataset of SNIPS, ATIS, _orpus and SGD are reported in Table 1, 2, 3, 4, respectively, which show that the proposed method consistently outperforms the previous BERT-based and ELMo- based SOTA methods, and performs comparably to the previous RNN-based SOTA methods. The de- tailed results of seen-slots and unseen-slots perfor- mance across different target domains on dataset of SNIPS, ATIS, MITT_corpp and SGD are reported in Table 6, 7, 8, 9, respectively. On seen-slots side, the proposed method performs comparably to prior SOTA methods, and on unseen-slots side, the proposed method consistently outperforms other SOTA methods.
invariant knowledge
model
unseen
HiCL (BERT Backbone)
-Gaussian Embedding + KL--div. 68.94
29.71
Point Embedding Euclidean
64.52
28.
Point Embedding Cosine
68.30
26.86
5.2 Quantitative Analysis
Table 5:
The ablation study of HiCL adopting different types of embedding on SNIPS dataset.
Ablation Study To study the contribution of dif- ferent component of hierarchical CL, we conduct ablation experiments and display the results in Ta- ble 1 to Table 9.
The results indicate that, on the whole, both coarse-grained entity-level CL and fine-grained token-level CL contribute substantially to the per- formance of the proposed HiCI. on different dataset. Specifically, taking the performance of HiCL on SNIPS dataset for example, as shown in Table 1, the removal of token-level CL ("w/o fine CL") sharply degrades average performance of HiCL by 2.17%, while the removal of entity-level CL ("w/o coarse CL.") significantly drops average per- formance of HiCL by 1. 1.26%. Besides, as shown in Table 6, removing entity-level CL ("w/o Lcourse") "), the unseen slots effect is drastically reduced by 4.61%, and removing token-level CL ("w/o Liine") "), the unseen slots effect of the proposed model con- siderably decreased by 4.01%.
Toeal Lomen
Fore Enkirt Colfee
Figure 4: HiCL's performance on unseen-slots and seen slots of each domain in SNIPS dataset when equipped with Gaussian embedding and point embedding. The lines denote average scores over all domains.
Coarse-grained CL vs. Fine-grained CL On the basis of ablation study results (8 5.2), our anal- yses are that, coarse -aaained CL complements fine-
Gaussian embedding vs. Point embedding We provide ablation study for different forms of em- bedding that are migrated to our HiCL, i.e., Gaus-

---

sian embedding and point embedding, and inves- tigate their performance impact on unseen slots and seen slots. As shown in Table and Figure 4, HiCL achieves better performance on both seen slots and unseen slots by employing Gussian em- bedding over its counterpart, i.c., point embedding. This suggests that Gaussian embedding may be more suitable than point embedding for identify- ing slot entities of novel slot types in cross-domain generalization task.
BERT and their knowledge remains unseen for TOD-BERT. (2) There a discrepancy of data distribution between the corpora that TOD-BERT pre-trained on and the datasets of SNIPS, ATIS and
_orpus. The datasets that TOD-BERT pre- trained on are multi-turn task-oriented dialogues of modeling between user utterances and agent re- sponses, whereas the datasets of SNIPS, ATIS and MIT MIT_corpus are single-turn utterances of users in task-oriented dialogues. Perhaps this intrinsic data difference affects the performance of TOD-BERT on these single-turn dialogue datasets. (3) TOD- BERT may also suffer from catastrophic forget- ting (Kirkpatrick et al., 2016) during pre-training, TOD-BERT is further pre-trained by initializing from BERT, catastrophic forgetting may prevent TOD- BERT from fully leveraging the general pur- pose knowledge of pre-trained BERT in zero-shot learning scenarios. From the experimental results, we observe that the performance of TOD-BERT is even much lower than BERT-based models (e.g., mcBERT), which may be a possible empirical evi- dence for the above analysis.
Multi-domains Training vs. Single-domain Training From the results in Table 6, 7, 8, 9, we clearly see that, for unseen slots cross -oos-- transfer, single-domain training is much more diffi- cult than multi-domain training. The averaged F1 score of unseen slots performance of HiCL across all target domains three multi-domani training datasets (SNIPS, ATIS and SGD) is 38.97, whereas the averaged Fl score of unseen slots performance of HiCL on single-domain training dataset of MIT_corpuu is only 12. This tremendous gap may reveal that the diversity of source domains in training is a very critical factor that determines the model's 's capability of cross-domaij migration of un- seen slots. To have closer analysis, plural source domains in training stage mean that there are abun- dant slot types that help the model learn generalized comaanninnvarian features of slot types, to avoid overfitting to the limited slot type classes in source domains. The results in Table 8 verify this analysis, we observe that, without the constraint of profes- sionally designed generalization technique, learn- ing with the limited slot types, TOD-BERT barely recognizes any unseen slot, and mcBERT performs poorly unseen slots. Although HiCL achieves the best results against basclines due to specially designed generalization method of hierarchical CL, the proposed model also suffers from limited diver- sity of slot types in single-domain training mode, and its performance is significantly lower than that of multi multi-domain training.
In contrast, TOD-BERT performs extremely well on SGD dataset and it beats all BERT-based models. This is because that TOD-BERT is pre- trained on SGD dataset (Wu et al., 2020) and it can thoroughly leverage the prior pre-trained knowl- edge SGD to tackle various downstream tasks including zero-shot ones on this dataset. How- TOD-BERT ever, when our HiCL migrates (HiCL+TOD-BERK) -ERTT, as shown in Table 4 and 9, the performance of the model again achieves an up- lift. Concretely, the overall performance increases by 0.4% and unseen slots performance increases by 9.18%, which is prodigious boost, only at the expense of a drop of 2.43% on seen slots perfor- mance. This demonstrates that, in terms of unscen slots performance, even on the seen pre-training datasets, our method of HiCL can still compensate the shorteoming of pre-trained TOD models (e.g..
Performance Variation Analysis of TOD-BERT
Surprisingly, shown in Table 1, 2, 3, TOD- BERT (current pre-trained TOD SOTA model) per- forms inferiorly on datasets of SNIPS, ATIS and MIT_corpus and fails to meet our expectations. We present analysis of causes as below: (1) TOD- BERT is unable to directly take advantage of the prior knowledge of pre-training on the datasets of SNIPS, ATIS and MIT_corpus. These datasets are not included in the pre-training corpora of TOD-
5.3 Qualitative Analysis
Visualization Analysis Figure 5 in Appendix provides t-SNE seatter plots to visualize the perfor- mance of the baseline models and HiCL on test set of GetWeather target domain of SNIPS dataset.
In Figure 5(a) and Figure 5(c), we observe that TOD-BERT and mcBERT have poor clustering styles, their representations of predicted slot en-

---

tities for unseen slots (digital number 2, 3, and in the figure) sparsely spread out in embedding space and intermingle with other slot entities' rep- resentations of unseen slots and outside (0) in large areas, and and its slot entities' representations of seen slots (digital number 5, 6, 7, 8 and 9 in the figure) are little sparsely scattered. This phenomenon possibly suggests two problems. On the one hand, without effective constraints of generalization technique, restricted to prior knowledge of fitting the limited slot types in source training domains, TOD-BERT and mcBERT learn little -nvaaiiant and slot- agnostic features that could help them to recognize new slot-entities, they mistakenly classify many slot entities of unseen slots into outside (0). On the other hand, although TOD-BERS and mcBERT generate clear-cut clusters of slot-entity classes of seen slots, they possess sub-optimal discriminative capability between new slot-entity classes of un- seen slots, they falsely predict multiple new slot types for the same entity tokens.
type for certain entity, due to learning little prior knowledge of generalized token- or entity-classes, resulting in inferior capacity to differentiate be- tween token- or catego-cs,,,,,,ss which discloses the frangibility of mcBERT on unseen slots perfor- mance. Whereas, HiCL performs more robustly for entity tokens prediction versus different unseen slots, and significantly outperforms mcBERT on unseen slots in different target domains. Thanks hierarchical CL and ILSSI, our HiCL learns gen- eralized knowledge to differentiate between token- or entity-classes, even between their new classes, which is a generalized slot-agnostic ability. (2) HiCL is more capable of recognizing new enti- ties over mcBERT by leveraging learned general- ized knowledge of token- and entity-class. For instance, in target domain of GetWeather, both HiCL and mcBERT can recoginze token-level en- tity "warmer" and "hotter" that belong to unseen class of condition condition_temperauure but mcBERT fails to recognize "freezing" and "temperate" that also belong to the same class, owing to the limited gen- eralization knowledge of token-level class. With the help of hierarchical CL that aims at extracting the most domain- invariann features of token- and entity-classes,
is even more worse
In Figure 5(b) and 5(d), we can see clearly that, HiCL produces better clustering division between new slot-entity classes of unseen slots, and between new slot-entity class and outside (O), due to gen- eralized differentiation ability between entity-class (token-class) by extracting class agnostic features through hicrarchical CL. Moreover, equipped with Gaussian embedding and KL-divergence, HiCL ex- hibits even more robust performance on unseon slots than equipped with point embedding and Eu- clidean distance, the clusters of new slot-entity classes of unseen slots and outside (0) distribute more compactly and separately.
HiCL succeed in recogniz- ing these novel entities. (3) HiCL performs slightly better than mcBERT on the seen slots. The two models demonstrate equivalent knowledge trans- fer capability of seen slots from different training domains to target domains.
Conclusion
In this paper, we improve cross-domain ZSSF model from a new perspective: to strengthen the model's generalized differentiation ability be- tween entity-class (token-class) by extracting the most domain-invariant and class-agnostic features. Specifically, we introduce novel pretraining- free HiCL framework, that primarily comprises hier- archical CL and iterative label set semantics in- ference, which effectively improves the model's 's ability of discovering novel entities and discriminat- ing between new slot-entity classes, which offers benefits to cross-domain transfcrability of unseen slots. Experimental results demonstrate that our HiCL is a backbone-independenl framework, and compared with several SOTA methods, performs comparably or better on unseen slots and overall performance in new target domains of ZSSF.
Case Study Table 11 in Appendix demonstrates the prediction examples of ILSSI on SNIPS dataset with HiCL and mcBERT for both unseen slots and seen slots, in target domain of BookRestaurant and GetWeather, respectively. The main observations are summarized as follows:
(1) mcBERT is prone to repeatedly predict the same entity tokens for different unseen slot types, which leads to its misclassifications and perfor- mance degradation. For instance, in target domain of BookRestaurant, given the utterance "i d like table for ten in 2 minutes french horn son- ning eye", mcBERT repeatedly predicts the same entity tokens "french horn sonning eye" for three different types of unseen slots. This phenomenon can be interpreted as a nearly random guess slot

---

Limitations and Future Work
Speech Communication Association, Stockholm, Swe- den, August 20-24, 2017, pages 2476-2442 ISCA.
Large language models (LLMs) exhibit powerful ability in zero-shot and few shot scenarios. How- ever, LLMs such ChatGPT seem not to be good at sequence labeling tasks (Li et al., 2023; Wang et al., 2023), for example, slot filling, named-entity recognition, etc. Our work endeavors to remedy this shortage with light-weighted language models. However, if the annotated datasets are large enough, our method will degenerate and even possibly hurt the generalization performance of the models (c.g.. transformer based language models). Since the models would generalize pretty well through thor- oughly learning the rich data features, distribution and dimensions, without the constraint of certain techniques that would become a downside under these circumstances, which reveals the principle that the upper bound of the model performance de- pends on the data itself. We directly adopt slot label itself in contrastive training of our HiCL, which does not model the interactions between la- bel knowledge and semantics. In our future work, we will develop more advanced modules via label- semantics interactions, that leverages slot descrip- tions and pre-trained transformer-baa large lan- guage models to further boost unseen slot filling performance for TOD.
Ting Chen, Simon Korblithh Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A simple framework for contrastive learning of visual representations. In Pro- ceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Vir- rual Event, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR.
Samuel Coope. Tyler Farghly, Daniela Gerz, Ivan Vulić, and Matthew Henderson. 2020. Span- :onn-Conv:RT: Few- shot span extraction for dialog with pretrained conver- sational representations, In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 107-121, Online. Association for Computational Linguisties.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calt- agirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018. Snips voice platform: an embedded spoken language understanding system for private- by-design voice interfaces. CORR, 0ts//805.10099
Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca Passonneau, and Rui Zhang. 2022. CONTaiNER: Few-shot named entity recognition via contrastive learning. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 6338-6334 66333, Dublin, Ireland. Association for Computational Linguistics.
Acknowledgement
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2019a. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-41866 Minneapolis, Minnesota. Association for Computational Linguistics.
We thank Xiangfeng Li for grammar revision, Jinyan Wang for data statistics, Fanqi Shen for his- togram plotting, and anonymous reviewers for their helpful suggestions. This work is supported by China Knowledge Centre for Engincering Sciences and Technology (CKCEST-2002-1-7))
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers). pages 4171-4186. Association for Computational Linguistics.
Ethics Statement
Our contribution in this work is fully method- ological, namely a Gaussian embedding enhanced coarse-to-fine contrastive learning (HiCL) to boost the performance of cross-domain zero-shot slot fill- ing, especially when annotated data is not available in new target domain. Hence, this contribution has no direct negative social impacts.
Xinya Du, Luheng He, Qi Li, Dian Yu, Panupong Pa- supat, and Yuan Zhang. 2021. QA-driven zero-shot slot filling with weak supervision pretraining, In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the l 11th International Joint Conference on Natural Language Processing ( (Volume 2: Short Papers), pages 654-664, Online. Association for Computational Linguistics.
References
Ankur Bapna, Gökhan Tür, Dilek Hakkani-Tür, and Larry P. Heck. 2017. Towards zero-shot frame se- mantic parsing for domain scaling. In Interspeech 2017, 18th Annual Conference of the International

---

Chih-Wen Goo, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Ssung-Chieh Chen, Keng- Wei Hsu, and Yun-Nung Chen. 2018. Slot-gated modeling for joint slot filling and intent prediction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 2 (Short Papers), pages 753-757, New Orleans, Louisiana. Association for Computational Linguistics.
IJCAI-19, pages 2642-2648. International Joint Con- ferences on Artificial Intelligence Organization.
Junyang Jiang, Deqing Yang, Yanghua Xiao, and Chenlu Shen. 2019b. Convolutional gaussian embed- dings for personalized recommendation with uncer- tainty. In Proceedings of the Twenty-Fighth Interna- tional Joint Conference on Artificial Intelligence, IJ- CAI 2019, Macao, China, August 10-16. 2019, pages 2642-2648. ijcai.org.
Prakhar Gupta, Cathy Jiao, Yi-Ting Yeh. Shikib Mehri, Maxine Eskenazi, and Jeffrey Bigham. 2022. In- structDial: Improving zero and few-shot general- ization in dialogue through instruction tuning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 505- 525, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabi- nowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka
Demis Hassabis. Clau- dia Clopath, Dharshan Kumaran, and Raia Hadsell. 2016. Overcoming catastrophic forgetting in neural networks. CoRR, abs/ 2.0.0996..
Keqing He, Jinchao Zhang, Yuanmeng Yan, Weiran Xu, Cheng Niu, and Jie Zhou. 2020. Contrastive zero-shot learning for cross- doss--dmain slot filling with adversarial attack. In Proceedings of the 28th Inter- national Conference on Compulational Linguisties, pages 1461-1467, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics.
Gakuto Kurata, Bing Xiang, Bowen Zhou, and Mo Yu. 2016. Leveraging sentence-level information with encoder LSTM for semantic slot filling. In Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2077-2083, Austin, Texas. Association for Computational Lin- guistics.
Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language sys- tems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990.
Sungjin Lee and Rahul Jha. 2019. Zero-shot adaptive transfer for conversational language understanding. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innova- tive Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educa- tional Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February I, 2019, pages 6642-6649. AAAI Press.
Seong-Hwan Heo. WonKee Lee, and Jong-Hycok Lee, 2022. mebert: Momentum contrastive learning with BERT for zero-shot slot filling. In Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022, pages 1243-1247. ISCA.
Changliang Li, Liang Li, and Ji Qi. 2018. self- attentive model with gate mechanism for spoken lan- guage understanding. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 3824-3833 2224,,33, Brussels, Bel- gium. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Tere Term Memory. Neural Computation. 18):1733-1780.
Ran Iwamoto and Masahiro Yukawa. 2020. RIJP SemEval-2020 task Gaussian-bart based embeddings for semantic change detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 08-104, Barcelona (online). International Committee for Computational Linguistics.
Xianzhi Li. Xiaodan Zhu, Zhiqiang Ma, Xiaomo Liu, and Sameena Shah. 2023. Are chatgpt and gpt-4 general-purpose solvers for financial text analytics? an examination on several typical tasks.
Han Liu, Feng Zhang, Xiaotong Zhang, Siyang Zhao. and Xianchao Zhang. 2021. An explicit-joint and learning framework for few- shot intent classification and slot filling. In Find- ings of the Association for Computational Linguis- rics: EMNLP 2021, pages 1945-1955, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Bin Ji, Shasha Li, Shaoduo Gan, Jie Yu, Jun Ma, Huijun Liu, and Jing Yang. 2022. Few-shot named entity recognition with entity-level prototypical network enhanced by dispersedly distributed prototypes. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1842-1854, Gyeongju. Republic of Korea. International Com- mittee on Computational Linguistics.
supervised-contrasin
Zihan Liu, Genta Indra Winata, Peng Xu, and Pascale Fung. 2020. Coach: coarse-to fine approach for cross-domain slot filling. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 19-25, Online. Association for Computational Linguistics.
Junyang Jiang. Deqing Yang, Yanghua Xiao, and Chenlu Shen. 2019a. Convolutional gaussian em- beddings for personalized recommendation with un- certainty. In Proceedings of the Twenty-Eige Eighah Inter- national Joins Conference on Artificial Intelligence

---

Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representatinss ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
A.B. Siddique. Fuad Jamour, and Vagelis Hris- tidis. 2021. Linguistically-eniie awarezero-shot slot filling. In Proceedings of the Web Conference 2021, WWW '21, page 3279-3290, New York, NY, USA. Association for Computing Machinery.
and context-
Qiaoyang Luo and Lingqiao Liu, 2023. Zero-shot slot filling with slot-prefix prompting and attention rela- tionship descriptor. Proceedings of the AAAI Confer- ence on Artificial Intelligence,
Luke Vilnis and Andrew McCallum. 2015. Word rep- resentations via gaussian embedding. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings.
Tanmoy Mukherjee and Timothy Hospedales. 2016. Gaussian -ssuul-inggiitic embedding for zero-shot recognition. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Process- ing, pages 912-918, Austin, Texas. Association for Computational Linguistics.
Liwen Wang, Xuefeng Li. Jiachi Liu, Keqing He,, Yuan- meng Yan, and Weiran Xu. 2021. Bridge to target domain by prototypical contrastive learning and la- bel confusion: Re-explore zero-shot learning for slot filling. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9474-9480, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Binling Nie. Ruixue Ding, Pengjun Xie, Fei Huang. Chen Qian, and Luo Si. 2021. Knowledge-aware named entity recognition with alleviating heterogene- ity. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium Educational Advances in Artificial Intelligence, EAAI 2021, Vir- tual Event, February 2-9, 2021, pages 13595-13603. AAAI Press.
Qilong Wang, Peihua Li, and Lei Zhang. 2017. G2denet: Global gaussian distribution embedding network and its application to visual recognition. In 2017 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2017, Honolulu, HI. USA, July 21 21-26, 2017, pages 65 IBEE Computer Society.
Chen Qian. Fuli Feng. Lijie Wen, and Tat-Seng Chua. 2021. Conceptualized and contextualized gaussian embedding. In Thirty-Fifth AAAI Conference on Ar- tificial Intelligence, AAAI 2021, Thirty- Third Con- ference on Innovative Applications of Artificial In- telligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAT 2021, Virtual Event, February 2-9, 2021, pages 13683-13691, AAAI Press.
Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023. Gpt-ner: Named entity recognition via large language models.
Yu Wang, Yilin Shen, and Hongxia Jin. 2018. A bi- model based RNN semantic frame parsing model for intent detection and slot filling. In Proceedings of the 2018 Conference of the North American Chap- of the Association for Computational Linguistics: Human Laguage Technologies, Volume (Short Pa- pers), pages 309-314, 31, New Orleans, Louisiana. As- sociation for Computational Linguistics.
Libo Qin, Wanxiang Che, Yangming Li, Haoyang Wen, and Ting Liu. 2019. A stock-propagation framework with token-level intent detection for spoken language understanding. In Proceedings of the 2019 Confer- ence on Empirical Methods Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIPP), pages 2078-2087, Hong Kong, China, Association for Computational Linguistics.
Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020. TOD-BERT: Pre-trained natural language understanding for task-oriented di- alogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 917-929, Online. Association for Computational Linguistics.
Lance Ramshaw and Mitch Marcus. 1995. Text chunk- ing using rranformationn-bael learning. In Third Workshop on Very Large Corpora.
Mengshi Yu, Jian Liu, Yufeng Chen, Jinan Xu, and Yujie Zhang. 2021. Cross-domain slot filling as ma- chine reading comprehension. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 3992-3998. Interna- tional Joint Conferences on Artificial Intelligence Organization. Main Track.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Schema- guided dialogue state tracking task at DSTC8. CoRR, 0bs/2002.01355
Darsh Shah, Raghav Gupta, Amir Fayazi, and Dilek Hakkani- Tur. 2019. Robust zero-shot cross-domain slot filling with example values. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 5484-5490, Florence, Italy. Association for Computational Linguistics.
Arda Yüksel, Berke Ugurlu, and Aykut Koç. 2021. Se- mantic change detection with gaussian word embed- dings. IEEE ACM Trans. Audio Speech Lang. Pro- cess., 29:3349-3361.

---

Appendix
Algorithm 2: Iterative Label Set Semantics Inference
// prediction in target domain Input: All Test Samples D Em D.,
Algorithm 1: Iterative Label Set Semantics
Inference
where D, = (St, A., 34))
training in source donain Input: Batch Samples Ei=1 Di,
Output: Total Prediction Set O, Ouaseen and Oseen. and FI Scores for O Ounseen and Oseen in Target Domain
where D, A4, )1, 4.5
// Testing Data Dia includes all negative
and positive samples constructed as $ 4.5 in Target Domain
11 DI and y denote coarse and fine-grained CL label.
Data:
Testing Data D:x, All Slot Types in Source Training Domains A All Slot Types in
Output: Batch Loss
11 Eq
Target Domain A²
Data: Training Data Dir for training batch D Drr do for D. D do
for testing sample D. DD. do A then
if E A'aand
Dia Dinseen + D; // add D; to Diassen
O:
CRF(W, (P, +.)
4 else if E and S. E A then
IA (L(P Ar (.)) // Eq.2
s
Dicc" + D; // add Dj to D
E. (LU( Ac,4))
Eq.2
testing batch D Dr for Dj E do
7
calculate Conaree. Liines C.
s
= CRF(W, @0 (., +,))
calculate L back-propugat and update parameters to optimize L.
0 O. // add O; to
for unseen slots testing batch D D Dunseen do for E D. do CRF(W, @ (I,,M((S..AA,444 bb))
A Related Work
Ownocen -0, // add O, Ounscen
A.1 Gaussian Embedding
for seen slots testing batch D. Deeen do for D. E D." do
Vilnis and McCallum (2015) initially explore to learn word embedding in Gaussian distribution space, they find that density-based Gaussian em- bedding helps capturing uncertainty of word repre- sentation and presenting more natural expression for asymmetries than point embedding. Wang et al. (2017) incorporate Gaussian distribution embed- ding into deep CNN architectures through an end- to-end pattern to discriminate first- and second- order image characteristics, which leverages the rich geometry and smooth representations of Gaus- sian embedding. Jiang et al. (2019b) employ Gaus- sian embedding in convolutional operations to cap- ture the uncertainty of users preferences in recom- mendation system. Qian et al. (2021) advocate contextualized Gaussian embedding that inte- grates inner-word knowledge and outer-word con- texts into word representations and capture their more accurate semantics. Das et al. (2022) lever- age Gaussian embedding in contrastive learning for few-shot named entity recognition task, which is work closer to ours.
O CRF(W; @ (PLM((S., As, A., +b))
Oseen + O, // add O, to Oseen
calculate fll score for O calculate f'l score for Ounseen
20
calculate fll score for Oseen
in NER task, while it is the first of its kind for us to introduce token level CL, and we innovatively present a hierarchical CL. architecture and empir- ically verify that the combination of entity- and token- level CL will significantly outperform either of them. Secondly, we explore more challenge re- search orientation, i.e., zero-shot task of single-turn and multi-turn arientreetedd dialogues instead of fhw-shot NER task that comprises single indepen- dent sentences (Das et al., 2022). Finally, we take deep dive into the pre-trained general language models for task-oriented dialogue (TOD), evaluate and compare it with professionally designed expert model of ZSSF (HiCL). Most researchers will be curious about whether the vanilla capability of pre-trained general TOD models would replace that of small expert models of ZSSF this see- nario, and whether specially designed generaliza-
However, our work is fundamentally different from the research (Das et al., 2022) in many as- pects. First of all, Das et al. (2022) present the method of entity level CL (entity-tole tokens level CL)

---

532
392
-33
-70
-44
533
530
1003
2599
.00
-00
430
-20
230
420
(a) TOD-BERT
(b) HiCL Point Embedding Euclidean
330
hece
octakd
716
250
299
-32
-255
-666 -100
-900
-203
McBERT
(d) HiCL Gaussian Embedding KL
Figure 5: t-SNE visualization in testset of Get target domain on SNIPS dataset for different methods. 1-4 denote unseen slots and 5-9 denote seen slots in all the subfigures.
tion techniques would still work or bring benefits for pre-trained general models in this specific field. This even brings search of large language models (LLMs) like Chat- GPT our work bring some explorations into this kind of thoughts. We introduce the pre-trate pre-training ex- pert model (TOD-BERT) that pre-trained on large corpora of task-oriented dialogue as a baseline to explore that whether this model is good enough for unseen slots generalization, and whether our method can continue improve the unseen slots performance top of TOD-BERT, which is also missing from the research (Das al., 2022).
and response contrastive loss (RCL), achieving im- provements on slot filling performance for dialogue state tracking. Liu et al. (2021) propose joint contrastive learning for few-shot intent classifica- tion and slot filling task-oriented dialogue sys- tem. Wang et al. (2021) propose a prototypical con- trastive learning to bridge semantics gap between token features and slot types in ZSSF. Heo et al. (2022) train BERT encoder with momentum con- trastive learning to develop a robust 7SSF model. In this work, we introduce a Gaussian embed- ding based hierarchical CL framework. At first, coursely learns entity-class knowledge of en- tity type and boundary via entity-level CL. Then, it combines the learned entity-level features, to finely learn token-class knowledge of BIO type and boundary via token-level CL.. Our method is essen- tially different from the above approaches.
enlightenment to the re-
A.2 Contrastive Learning in Slot Filling
He et al. (2020) advocate an adversarial attack strengthened contrastive learning with an objec- tive of optimizing the mapping loss from slot tity slot description in representation space for cross-domain slot filling. Wu et al. (2020) pre- train natural language understanding BERT (De- vlin et al., 2019b) for task-oriented dialogue with the joint masked language modeling (MLM) loss
B Additional Analysis
B.1 Performance Gain vs. Dataset Volume
In Tabel 1, 2, 3, 4, we observe that, with the in- crease dataset volume (see Table 10), the per- formance gain HiCL against baselines gradually
hhanss//opena..com/bbggggggg

---

Training Setting
Zero shot
Method J
Domain
AP
BR
GW
PM
KB
SCW
FSE
AVG
CT
47.15
5.18
51.43
1.87
39.54
2,11
46.48
0.13
25.10
0.13
39.59
11.32
10.84
37.23
3.38
RZT
57.50
43.84
47.45
0.13
2.19
RNV based
Coach"
64.65
7.94
55.88
2.89
63.97
2.20
31.69
8.78
40.42
18.81
44.35
27.27
15.21
46.22
9.31
PCLC!
73.32
2.57
62.81
16.56
65.84
14,20
45.17
17.53
34.70
25.70
53.51
29.66
22.71
51.68
17.38
HCCL+BiLSSMM (ours)
65,83
17.67
54.31
14.78
66.47
14.54
46.87
16.92
40.54
19.68
54,07
24.57
18.96
50.38
17.10
Mo based
LEONA
59.82
8.62
67.73
17.49
77.91
7.53
60.12
13.94
42.31
11.02
45,86
32.16
23.43
55.13
13.67
Mo (ours)
65.34
17.43
71.75
15.17
73.55
18.18
58,69
20.18
49.96
35.32
23.97
56.43
18.84
TOD-BERT
70.27
15.97
72.57
16.75
84.12
3.03
51.63
10.79
39.57
16.42
62885
46.60
38.36
61.00
16.89
mcBERT!
77.60
17.73
81,12
34.06
91.02
10.26
52.32
10:42
68.25
10.00
70.72
35.21 46.27
69.61
19.61
BERT bused
RCSF
25.44
HiCL+BERT (ours)
77.91
18.89
81.73
38.09
90.32
31.17
58.81
21.09
55.96
26.85
73.22
44.65
42.15
68.94
29.71
w/o coarse CL. (ours)
75.17
17.90
83.79
29.79
88.17
26.51
67.34
12.59
67.04
22.
19
72.06
46.31
41.63
71,41
25.10
CL (ours)
78.29
14.18
73.41
32.92
91.95
61.41
24.31
56.86
21.15
70.63
35.81
67.90
25.70
Table 6: Detailed Fl scores on SNIPS for seen and unseen slots across all target domains. denotes the results reported in (Wang al., 2021). denotes that we run the publicly released code (Siddique al., 2021) to obtain the experimental results and denotes that we re-implemented the model, and we reevaluate their performance on seen and unseen slots following the split method unseen- and secn-slots sub-test set in Appendix F and the test method of iterative label set semantics inference in Algorithm 2.
Training Setting
Zero-shot
AR
AF
AL
FT
GS
OS
AVG Fi
Method
V
Domain =
seen
unseen
seen
unseen
seen
unseen
seen
unssen
seen
unseen
seen
unseen
seen
unseen
ELMo based
LEONA
59.54
6.17
96.56
93.11
85.65
3.71
51.07
31.27
84,70
78.44
13.72
HECL+ELMo (ours)
55.34
14.81
98.
93.24
86.47
5.55
52.61
40.00
83.67
78.28
20.12
TOD-BERT
47.70
9.73
96.54
89.02
87.09
6.45
61.30
22.22
75.80
76.24
12.80
mcBERTS
66.32
9.05
98.56
92.82
89.26
9.20
74.58
64.44
81.35
83.82
27.56
BERT based
HiCL+BERT (ours)
72.51
17.65
98.09
92.20
88.87
12.12
77.95
85
82,41
85.34
38.32
wlo coarse CL (ours)
68.27
11.76
98.75
94.20
90.34
9.38
69.75
85.71
71.08
82.07
35.62
w/o fine CL (ours)
65.58
15.76
98.40
93.51
89.00
10.66
81.02
83.33
84.00
85.25
36.58
Table 7: Detailed Fl scores on ATIS for seen and unseen slots across all target domains.
Training Setting
Zerovshoot
invariant slot features for cross-domain ZSSE
Method 1
Domain
Movie
Kestaurant
AVG Fl
0E3044
BUD-BERT
71.72
56.87
0.50
64.30
0.90
C Dataset and Split Details
mcBERT
76.51
67.81
5.71
72.16
5.71
BERT based
HiCL+BERT (ours)
77.75
66.93
10.12
72.34
10.12
w/o CL (ours)
24.99
58.16
3.71
66.58
3.71
C.1 Dataset
w/o CL (ours)
75.15
67.51
71.33
We evaluate approach on four TOD tasks datasets, i.e., SNIPS (Coucke et al., 2018), ATIS (Hemphill et al., 1990), MIT _UT_corpus (Nie et al., 2021) and SGD (Rastogi et al., 2020).
Table 8: Detailed scores on MIT _erpuus for seen and unseen slots across all target domains.
Transn Seibig
SNIPS (Coucke et al., 2018) is a personal voice assistant dataset that contains 7 domains.
Matheel
75.6
16
13
28 14
08:00
13:0
17
01.1
16.8
IICLATID BELTIID---EEOOO
25.15
22.32
25.22
5511
9:37
ATIS (Hemphill et al., 1990) is a dataset that con- tains transcribed audio recordings of people mak- ing flight reservations with 18 domains. Domains that contain less than 100 utterances are merged into a single domain Others in our experiments.
11:1
67.8
15ST
8130
38.9
96.55
55.67
Table 9: Detailed F1 scores on SGD for seen and unseen slots across all target domains.
MIT _rT__poo_'' is a spoken query dataset that con- sists of MIT restaurant domain and MIT movie domain. MIT movie domain contains eng corpus and trivia 10k corpus, namely simple query ver- sion and complex query version, respectively, We merge the two version into one corpus and call it MIT movie domain.
diminishes. Besides, as indicated in Table 6, 7,8, 9, on the large dataset, HiCL. needs to sacrifice more seen slots performance to improve unseen slots performance. This phenomenon indicates that the zero-shot generalization ability of the baseline models gradually becomes stronger with the growth of dataset volume and the diversity of slot types, which helps baseline models to learn more domain-
'The original MIT corpora can be downloaded from https://groups csail. mit. mit.eedu//l wdw/ses/d/soooeeeeoeeeee http///ls/dwwooddss

---

SGD (Rastogi et al., 2020) contains 16 domains. However, we find that training on 15 domains ex- cept single target domain, almost all slot types become seen slots. To increase unseen slots and knowledge transfer difficulty, we adopt the same dataset span as (Gupta et al., 2022; Coope et al., 2020) and choose four domains for SGD dataset, namely buses, events, homes, rental cars.
adopts slot descriptions to promote the perfor- mance on unscen slots in the target domain.
Robust Zero-shot Tagger (RZT) (Shah al., 2019) incorporates additional slot example entities combined with slot descriptions to im- prove zero-shot adaption.
Coarse-to-fine Approach (Coach) (Liu et al., 2020) is pioneer of two-stage framework for ZSSF, which divides the ZSSF task into two stages: coarse-grained slot entity segmen- tation in the form of BIO and fine- grained alignment between slot entities and slot types by utilizing slot descriptions. We use their stronger variant Coach+TR and call Coach for brevity.
C.2 Unseen Slots and Seen Slots in Different Domains
Table 12 presents detailed unseen slots and seen slots in different domains for four datasets, i.e.. SNIPS, ATIS, MIT_corpus and SGD.
Implementation Details
We use uncased BERT to implement the encoder in our model, which has 12 attention heads and 12 transformer blocks. For TOD-BERT, we the stronger variant that pre-trained using both the MLM and RCL objectives. We uses 100 dimensional Gaussian embeddings, A op- timizer (Loshchilov and Hutter, 2019) with 8 = -0.9, 0.999) and warm-up strategy (warm-up steps is 1% of total training steps). Early stop of patience is set to 30 for stability. 7=0.07 for (6) dropout rate is 0.3. We set a=1 and 8=1 in Eq.(1 We select the best hyper- parameters by searching combination of batch size, learning rate and y in (10): learning rate is in {1 10 5,5 10 6,1 x 10 5, 5.5 10 "). batch size is {8, 16, 32, 64), and {0.001, 0.01, 0.05, 0.1, 0.5). For instance, an opti- mal learning rate is X 10-5 for SNIPS dataset, and 10 for the best-performing model on dev set and evaluate it on test set. We run 5 times for all our experiments and then average them to generate the results. We train and test our model on NVIDIA GeForce RTX 3090 GPUs and NVIDIA Tesla A100 GPU, and it takes averagely less than one hour to reach convergence.
Contrastive Zero-Shot Learning with Ad- versarial Attack (CZSL-Advv) (He et al., 2020) is an improver of Coach, which em- ploys contrastive learning and adversarial at- tack training optimize the performance of the framework.
Prototypical Contrastive Learning and La- bel Confusion (PCLC) (Wang et al., 2021) is two-stage based approach, which em- ploys prototypical contrastive learning and label confusion strategy to enhance the robust- ness of unseen slots filling under zero-shot setting.
in
and Context- Aware (LEONAA (Siddique et al., 2021) is advocate of three-stage ZSSF model, which utilizes context-aware and linguistic token representation to improve the effect on semantic similarity modeling between utterance tokens and slot descriptions based on attention mechanism.
Linguistically-Eriiee
dataset. We select
Reading Comprehension for Slot Filling (RCSF) (Yu et al., 2021) is the current ques- tion answering (QA) based SOTA model, which formulate ZSSF task machine read- ing comprehension (MRC) problem.
E Baseline Details
We compare HiCL with the following competing models.
Momentum Contrastive Learning with BERT(mBBERT) (Heo et al., 2022) is cur- rent state-of-the-a model (to our knowledge). which improves ZSSF performance by adopt- ing BERT backbone and training with mo- mentum contrastive learning.
Concept Tagger (CT) (Bapna et al., 2017) is one-stage leading model for ZSSF, which
"ntups://uuggangffce.co//// htt.://gg.nn.a.ee...oooottteeeee http://hggiiggaac..ooooo NO-BERRT-NNT-V

---

TOD-BERT (Wu et al., 2020) Pre-trained Natural Language Understanding for Task- Oriented Dialogue. This model current Ktate-of-the-ar pre-training model for TOD, we use their stronger variant that pre-trained with the joint technique of masked language modeling (MLM) loss and response con- trastive loss (RCL), and name it TOD-BERT for brevity.
domains and target domain, while unseen slots re- fer to those slot types that only appear in target domain. As shown in Figure 6, Liu al. (2020) divide SNIPS (Coucke et al., 2018) test set into unseen and seen subset according to whether an utterance contains at least one unseen slot, which leads to unseen-seen slots overlapping problems in unseen slots performance evaluation. Since the per- formance results of the unseen samples are actually entangled with seen slots (unseen sample test set contains both unseen slots and seen slots), which scriously causes a bias in testing a model's actual performance on unseen slots.
Seen Sample
Unseen Somple
1533
1500
For example, for an utterance "will it be colder in connorville" in target test set, its corresponding slot label is "O O O B-condition_ terpiiattutmmmatuuuee O B- city", this utterance sample comprises both unseen slot type ceandition_Impperrnn city. Nevertheless, this sample will be classified into unseen test according to the method of Coach (Liu al., 2020).
1250
1214
1062
1000
977
625
and seen slot type
480
250
0
PM R6
We rectify this bias by splitting unseen and seen test with slot granularity strategy instead of sam- ple granularity method used in Coach (Liu et al., 2020). This slot granularity split method illus- trated in $ 4.5, Algorithm and (iterative label set semantics inference), which is able to train and test unseen and seen slots, separately and unbiasedly. For instance, following this new split method, shown in Figure slot type condition pettriioon_mmmaratuure and city, will be re-classified into unseen subset and seen subset for the utterance "will it be colder in connorville" respectively ($ 4.5). Through the comparison between Figure and Figure 7, is observed that unseen and seen sub-test set across domains distribute more evenly by employing our approach over Coach (Liu et al.. 2020) method. Furthermore,
Figure 6: Unseen and seen slots set split on SNIPS dataset with the method of Coach (Liu et al., 2020). The figure on the top of bar chart denotes the number of seen unseen samples for different target domain in test set. "0" represents no sample (unseen seen) exists.
14000
Seen Siots
Unseen Siots
10000
9438
8000
8000
6400
6400
6236
6000
4626
4000
084
3112
3108
2000
find the defect of original Coach (Liu al., 2020) split method. For example, in target domain RateBook, as shown in the split result indicates that the number seen samples (or seen slots) is zero and all samples are unseen ones. However, splitting with our new method, the number of seen slots is 3, 112 and the number of un- seen slots is 7,780. We can see approximately one third of slots in domain RateBook are seen slots, but they were wrongly classified into unseen test by this biased split method of Coach (Liu al., 2020), and their performance was regarded as that of unseen samples. To our knowledge, almost all baselines follow the split method of Coach (Liu
GW
SCW
FSE
Figure 7: Unseen and seen slots set split SNIPS dataset with our method. The figure on the top of bar chart denotes the number of seen or unseen slots for different target domain in test set. "0" represents no unscen-slot exists.
F Rectified Test Set Split for Unseen Slots and Seen Slots
the cross-domain slot filling scenario, seen slots refer to those slot types that appear in both source

---

al., 2020), so we have reevaluate their real performance for unseen slots with our new split method, i.e., iterative label set semantics inference illustrated 4.5, Algorithm and 2.
Query Numbers
Dataset
Irain
val
Test
soca slots
unacen
SNIPS ATIS
T10/00
27.519
82,488
38,917
46,681
342,574
85,741
256,833
216,147
40,817
carpus
SGD
2,780,748
695,285
2,085.463
1,367,438
718.025
Table 10: Data statistics of training. validation and test for all domains of SNIPS, ATIS, MiT_corpuu and SGD, respectively, after data augmentation. All baselines and HiCL are fine-tuned tened on the same augmented dataset.

---

Gold
mcRERT
bost Inilom highl pubs for Til 0022
None
Nowe
facility: Noze
facility: Noze
facility: Note
cuisine
cuisine Note
type: pub
type:
type pub
served_dish None
served, Nose
served_daa Nose
part sine sumber:
party sumber:: None
party _uze, sumber
pai:
pos: Note
poi: Nouc
purt description
party descciption None
party deseeedddeiippoonn.. None
520 'hii
city: None
city None
city None
Gunerange: 02.22 pm
tinerange: 02:22 pm
Bincrange: 02:22 pm
country: None
country:
country:
soit: lighly rsted
sont lighty
sort: highly spatial_relatio None
apatiel_rclaiioe
retial_rclatio: None
stite:
stdl:: Nome
state: Veer
Wer somple
1d like a lhie for lon in mimiees ai fnnnch bom sonming eye
sdile Alt entiiiez
resturant same: french hom sonning
french sonning
french burn eye
Geclity: Nore
feiliity Nome
facility feench born
cuisine None
cuisine None
french
None
None
None
served. dish Nore
servol dish Nome
dish: Nume
None
party jumberrerrrrrree
None
Mone
french
deseciption None
deseciption None
SSW 160 Vi. 1hif iniiile
None
None
minntes
minntess
timerange: minutes
Mone
Non2
Nowe
None
None
spatiel_ relatio:: None
sparia_ relation: None
spatisl pelation None
Noue
list somple
+
Tow will If fire in securd
umanne sinte ve zhtt enniieg
candition Jemperatere: odd
condittion Jen pompeealre mold
condition Jempensare cold
curren Jocation there
current location None
current locasion: Note
uoedttion description: Nume
crncition description Nome
condition description: None prograptic poi: None
2tographic pos None
grograptic_ pou. Nons
Scele sill Ve 2hit chille
timerange: in second
timerange: in second
timerange in second
stute: Noae
state: Noue
state: Noocc
city: None
city : None
None
countr:: Noue
countr:: None
county:: Nonz
spatial relatio:: : None
spatial relation: None
spatial relation None
sosple
get ezechin
amovns slut zlat minicc
conditio Jemperature: Wasmer
Jemperatiee Wirmer
condition Jompentuse Warmer
cureent Jostion Nore
cutient Jocatinn Nose
cunent Jocatinn condition deseription:
conditon, deseription: Nane
condttion description: Nome
gengraphic poi weveeeee ceeeeeeeee
gengraphic poi
pengraphic_ Nont woe.e
Coo 1606 W. 1061 PnRRiey
timerang::
limznang:: None
timreanes : None state
state Noir
state None
City: : None
city None
city Nore
cnunry crechia
country coechia
country crechia
sputial jelnton: Nore
spatial relation: Nape
spatial relation - None
Jest simple
3
tell if It be Inecoing nest munch rhode Islnd
zlons plof entiliyy
Iotspraateee Eicozing
Jemueeratue Beezing
temprnrue None
Jocation: Nore
Jocation Nose
Jonett_occtoon.o Nose
gengraphic_paa Note
condizon,
condition descriptio Nose
pengraphic_ Nove
grograhhic_po. None
Wol H6lh 3bld chhiiie'
chode island
shole island
shode island
city: None
Nowr
spatial relaico:: None
spaial, mdlaiion: None
spatisl None
Wet somple
3
when will be tempersie lanclori
inseee slon 96. Abf entiley
condition temprrature I tempennt
condition lemperature tompersee
condition tempenzure None
cumnnt bostion:: : None
nummen Inzzaion: Nune
comeni Incsason: Nore
condition description: Note
conditoo, description : None
condition, description: None
grographic pos None
geugraphic pos: None
grographic pot None
Saote 16l V.. Wil chiile
timerange: None
tierangg: None
timerange: : None
stile: Note
stite: Noui
stite: Note
city: laasford
city lussfar
city: lansford
county: None
countr: None
country: Nonz
spatial relutoo: None
spaiial_relation relation: None
rpuiisl_tiationn restion None
somple
chilie:
Unsavo sion pht enblicz
conditio iemprrature: dhillier
monditon temporatire: chillier
condition lemprnzure: : Nume
cureet loction:: None
aurent Jocation Nune
curent Jocstion:
condtion description: Nore
conditio description: Nore
condition, description:
gragaphic poi Noer
poi None
prographic pot Nens eoe
-e. 220 2605 2357 PARDO
timerange: None
timerange: None
timgrange : None
state: Neve
state: Note
state: Nore
city musliana
Hiyy manliaca
city None
county: None
countr:: None
couttr:: None
sputial relation: Nore
spaiial relation: Nove
sputial relozion Nome
Table 11: Iterative label set semantics inference (ILSSI) prediction examples from SNIPS dataset with Restau- rant, Weaterr as target domain, respectively.

---

SNIPS
AdTToPlaylis
came Jume, owwner
aiii playliii insee
RocAResttamnt
party desctipuen, ops, served
city spaiid_ relation, limeRange, countyy
puny cuietre, facilit, resourat nome
GefWeather
condition Jescritton, cureer, Jocation, condioon Jempenaiee,
country sputiii relation sate, timeRange, city
pougraphic pot
Plac
playlist
doloo rsting, unil, nring, valui, host naiing
Ratellook
djjoc_par_of_ serreeuyee
otgect_tame,, name, object_yype
SeackKreeteeWokk
obest (yps,
FinlSeroninggroes
Mijoitiitssioon.yy movie Eype, location nume, movi, nome
ohyeL//ppr, limeRange, 'sperial relition
ATIS
Domain
Unseen
meal aitinee rode, class, type, arport cods, mod. tromace city Jame, sércrall cude. fimm hods mode, iolurcitt nome nestiction ools actine name
Atérevistion
booking claes, meal cude, days oode
light
smiser date
cost relatie,, fromboc name. retum, day pumber. loboccit city nume, depart dhis day number tolocstate name, fight, mod, arive datt day jain, depart dat date, relative, connect. depart Jime jelutive, stoplce city, Janee toloe airppctatann
actine name, antive timetime, jelative artive Jane day tumber.
coomnny, urive time time, depart lime period depert dane day
Arrfrn
duie relatvee
dute munth munih
direraf meal,
aipuorttttttttttcc.. triice dateeaaaemlllllee.
name, light stup, fromocssrpon name,
depan month name, toluc catt code, deport limezime, dase type. airline code, toloc airpon cude. round inp.
depart Joday jelative, depart tine sins-prrood pariod dayy
return date day name, fromloe state code, depert date your. diiht_days
mod, fight stop, toloc code, tromlic some, city pame. airport name, fight fight nunshrr
deport
arrive time persod of day, connect. airline code, relative depart Jime, peinod da_y fromioe pipoort_odde foomloe nime, dapan, jime shet Jime, anive, monhh name, depart Jime and tine, artve_tine fime, toloc Jame, class type. Fromloz state, cudk, toloc sippuri nome, deputt day rumber. depurt monb, day
Aidine
depart one relative. suplectiity round, tnp, depurt date today relative, depart date relatrve lobs slave nome, sirenil coll
depert duir
mod, fare, amount, deport dase month same, day name, arpert name, foboccity city
time time musiiv, and rebiive, flight cime. period of day fare basis pode.
fligtt, number, flight
depart_datt fromlo ony toloc airpect code, returs date jelutve, arive finslor dirport name, class type, meni, descrigtinn depan date.die miliiie, depert time period mud, shate code, flihtt days, rotarnllate day, mumher, depert dite juyneer
dapart name
relasive
ooplocsippot code, soplocsiate code. flight, return lime period mod, day, toloe county nume, arriw tims.nd time, soplrctipper nime, arivv lime peridd med, arrive dntcttdy relatve, rellativ retern, arrive sime.strt time
comqusiment, nsurs,
Flight
usonomy, arriw time penood of day, light sop, meal, sirenal code. depprt time time, tolocsinne jame deport deee relay,cl relative depart time and time, sirport code, airine retunn mouth_nant
retern_datt dats today rehaive.
date day number, depart time pelatieeleiieeee... artive month, rame, sirline_odde conrest, dlppart Jime start fime, sirpon. nume, depart lime. pariod of day, fremloe stute cods, froulo stat sume. serive duw day sume, sreuul cale strple some fromlo city aipee
day
day. depant
day
sinportccode
Ground Service
day_mamber, jelay_cl timee time_relative, nontii_name
depart date.daic relative flight time, state code, city_mame, depun dait month, name, loluc.ity city num,, sirprt tome. penoo of day, innsport type, fmmiic airport name
nime, light tolo strre code, frobor siree same, dipart Jumbeer mund_rri mumber acportt sirpot
Jays, siggu city zame, toloc city name, depart dais today relative trmqurt type, camomy, ainrafl oode, coler dilee rame.
fine
arrive dite month, name, cost, relative city nume, restriction cods. aitpottrpoo_ooo cods, Hight mod, stite cods, fromloo airpoctcodd. mod, meal, depart dase selutive, meal descrationn
Others
dat: month name, artiv jime time, jelative
servee dhie number, sirline onle depart limetime, depart Aaye depart
selutie,
server da nime,
type or. frumlvc.cily
serive time time, flight sep. fuce basis code, state rame, airliw name, deperi time period day
MIT perrpis
Domain
Stots
director. averggg plon, rating, life. traler, actor, revee, your gente, character, song, quote, opinios, award. onigin, soundrack. dizracter jume, relationshp
Movie
year. direcoor, soorr, song, raing, quise maller, opinion, teview, ociaá gear, character, soundrickk relatioship. character Jame, mengss, pverage. award
Restuurant
location, desl, hous, crisine, restuurast hame, price amesity

---

SGD
Tanget Domain
Unseee Slots
Scon Slots
oigit jurport, desunation city orgin_cRy, nonnsuson, Roooooi,,, destitation supper, fire. orgin, from, city, from, location. oigil wwwgorttooaanee name. siglin_suseenuui jume, sousity.
vense, pereest, rating, pdieess, place address destiaaton, movic lill, leaving sone, alam, tim, date, reurr dase, zekup, city. tine, veeree_rttnn rating, dam antnctionitan phone number, therk, in dat, city. eaes easinamme, name, stridlane date mutelumm,, pelup, date, place_ June. category, price_per_tdee alliess, depanure date prie. night
Rinus
auhound arrivll Jime, depettrr, line, distination summontueeetmm Jume, outhound depurture_ jime, to sttiio, pattiaiiionuziionnne leaving jine, inhound depettrre line
railig,
destintion, balance, wat line, leaving ittraction, nume, hotel same, noum
vamue ailloss, pickup
penes, cast, humdity, temperatue,, event date, staring. city event diector, avaiable sart Jime, oddress of Jocation,, mile, evout Jocatinn avallable end time, wind, subcategory, cussine, antit, acconnt belance, ilban, peecipiatios, soug,ntme, directed_ by aggregate_ putng
fare ondddaie, dast, departuee Jane. taee, city, plone plone_nutber, junber. ceeggryy pickap time prize. per day steet sibress venue. stmnt diit, event name, approsimatc ride durston, movie nume, addres. per ticket, pickop lucsion
Evens
destinsion, pickup lime, pickap. location, phone number, address balance, rent, peice. pickap dise, slmm time, apprcimati. ruls. duratoo, abrm nume. vist date, wai time, propery. same
dnppll ahet teaal parn
Homes
LOCA
pickop. clty, prize pickpp avengee raingg
city prise. pir..ight.
ditckkouLddeee location, car_taste
pickap_ locaiio, ghoue mumber, attocciio_tatie addiess,
property nume, movie name. pickap ome, street sddress, hoxel name, check_in, datd pace_per, day, leavaa_ditte. visiidate, miing.
Table 12: Unseen and seen slots in different domains of SNIPS, ATIS, MIT_corpus, and SGD dataset, respectively. The evaluation paradigm adopting each single domain as target test domain and the remainder of domains training domains in the datasct. denotes slot type exists.


Reviews:
[{"review_id": "r9GpPwiyld", "reviewer": "Reviewer_vCNz", "summary": "", "questions": "Question A:  When you calculate coarse-grained loss, how do the slot types encoded? For example, ‘this week’ is date_time, date_time, or ‘this week’ is trained as a whole?\n\nQuestion B: in the figure 3, it shows that a sequence of slot types and two utterances are sent into encoders? Are those slot types related to every token/entity in the two utterances? In the Gaussian Transformation Network, the slot types are also trained? Also in the output, the slot labels are entity-level or token-level? Please illustrate more on the figure. \n\nQuestion C: why only test on one unseen target domain? Did you try other target domains ? Usually, the model should be tested on several target domains to prove generalizability?\n\nQuestion D: what is intra-token? tokens in the same entity?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review.  \nQuestion A: A good question. We are the first case in your example:  ‘this week’ is labeled as date_time, date_time.  \nAssume that coarse-grained slot type set is {'O': 0, 'date_time': 1, 'location': 2 } and fine-grained slot type set is {'O': 0, 'B-date_time': 1, 'I-date_time': 2, 'B-location': 3,  'I-location': 4 }. In coarse-grained loss,  both \" this\" and \"week\" share the same slot type *date_time* (we can use the digital number of dict value to represent it, i.e., so the coarse slot labels of \"this week\" is 1 1 ),  we directly use dict values of coarse slot type set (e.g., 0,1, or 2 ) as the training labels of coarse contrastive loss without encoding them, because these values can be directly used as supervised signals.  In this way, the proposed model will coarsely learn that which tokens are entities and which are not, and the difference between entities via supervised contrastive learning of entity-level label.\n\nQuestion B: Apologize for the confusion in Figure 3.  We divide your question B into 4 sub-questions, we respond them with a, b, c and d, respectively.  \n&emsp;&emsp;a.   Each input instance (or sample) of transformer encoder consists of two parts, one slot type and one utterance. Since there are multiple slot types in one domain,  the volume of instances could be augmented via the combination of single utterance and multiple slot types. For instance, the slot type set of one domain is {'date_time', 'to_do', 'location'} , given a utterance \"what the weather in st paul this weekend\", three instances can be constructed: \"data time [sep] what the weather in st paul this weekend\",  \"to do [sep] what the weather in st paul this weekend\", and \"location [sep] what the weather in st paul this weekend\",   their BIO labels are \"O, O, O, O, O, O , O, O, O, O, B, I\", \"O, O, O, O, O, O, O, O, O, O, O, O\", \"O, O, O, O, O, O, O, O, B, I, O, O\",  respectively. Entity-level slot labels of three instances are \"O, O, O, O, O, O, O, O, location, location, date_time, date_time\" (three labels are the same) and their token-level slot labels  are \"O, O, O, O, O, O, O, O, B-location, I-location, B-date_time, I-date_time\" (three labels are the same) , in training the instances are randomly shuffled and allocated into different batch.\nFor simplicity, we use two instances/utterances to represent a training batch ( the training batch size is two) to illustrate how entity-level or token-level contrastive learning works.\n\n&emsp;&emsp;b.  As described in a., each instance only contains one slot type. In CRF loss,  this slot type is related with tokens or entities of the utterance of the same instance ( this slot type has no relation with other instances) via BIO label, i.e., [\"O\", \"B\", \"I\"].  However,  in hierarchical (entity and token level) contrastive loss, we labeled this slot type as \"O\", therefore, it has no relation with other tokens or entities of both this instance and other instances except for \"O\"(outside) class. \n\n&emsp;&emsp;c. Slot types are labeled as outside or 'O' during entity-level and token-level contrastive learning, so the representations of slot types  transformed by Gaussian transformation network do not involve in token-class or entity-class training except for 'O' class.  However, the representations of slot type are trained during CRF loss, which help HiCL modeling the semantic relationship between slot_type and utterance via BIO three-tag label, i.e., [\"O\", \"B\", \"I\"].\n\n&emsp;&emsp;d. The slot labels of entity-level and token-level only serve for contrastive learning that based on Gaussian embedding, and the entity-level and token-level contrastive learning only perform during training ( not in inference), and they only calculate contrastive loss and predict nothing. While, in the prediction output of CRF,  the entity label is neither entity-level nor token-level, it is just a  BIO three-tag set ,  i.e., y_set = [\"O\", \"B\", \"I\"]  or  [0, 1, 2].  Here is an example,  given <slot type (e.g., date_time)+utterance (e.g., send a reminder ...)> as input, the HiCL outputs the location of this slot type in the utterance with the three-tag set. Specifically, given \"date time [sep] send a reminder for a tire check next week\" as model input, HiCL outputs (CRF in Figure 3) the label sequence of  \"0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2\".\n\nQuestion C: In each dataset, we traverse all domains and make each of them as target domain,  i.e.,  each time make one domain as target domain and the rest domains are training domains, therefore target domain distributes across  all the domains. If you mean each time multiple domains as target domains, this setting never appeared in existing works of zero-shot slot filling of task-oriented dialogue, it is interesting but much more challenging. Especially for unseen slots, you know even only one target domain, the unseen-slots performance of  all the baselines is poor. Nevertheless, multiple domains as target domains are indeed an important indicator of cross-domain generalizability, which should be taken into consideration and continuously optimized in future works.\n\nQuestion D: If two tokens belong to one utterance, they are called intra-token towards each other.  Whereas, if two tokens belongs to two difference utterances, they are called inter-token towards each other. We carry the entity-level contrastive learning and entity-level contrastive learning in batch level during training, each batch may contains from several to dozens of utterances that depends on batch size setting.\n\nAs to reasons to reject, we give the following supplements and explanations:  \nWe verified and summarized two slot-agnostic abilities for our proposed HiCL, discovering new entities and differentiating  between new token- or entity-class, respectively.    \nWe present a multi-dimensional and progressive analysis, to verify slot-agnostic ability step by step,  particularly, in Appendix M (Iterative Prediction Case Study),  we empirically and  qualitatively validate HiCL can differentiate  between new token- or entity-class（line 1349-line1358, which is a important ability of slot-agnostic ; again,  our study (line 1359-line1376) validates HiCL own more stronger capability of discovering new entities over baseline, which is also a critical ability of  slot-agnostic.    \nIn summary, slot-agnostic ability is a cross-domain generalization ability that can NOT be obtained by learning from slot-specific features ( all existing works just focus on slot specific learning and they almost do not have this ability). While, our proposed hierarchical contrastive leaning targets at learning powerful cross-domain slot generalization capacity and is specifically designed to learn self-agnostic ability for unseen slots.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is quite comprehensive in introduction, problem definition, experiments and ablation studies. The author illustrates the problem and different paradigms of zero-shot slot fillings very clearly. It is good for people who are not experts in this field to learn the concept quickly. \n\n2. The idea of adding token-level CL is innovative. It is not very intuitive but the author explains it quite well and the results show the great performance of the framework.\n\n3. The ablation studies are detailed. The author discusses about the components from different aspects. The ablation studies make this work more convincing.", "reasons_to_reject": "The method part needs more clarification (more details in the questions part).\n\nIn the contribution part, the author said the model is to extract self-agnostic features across tokens. However, in the analysis part, they said they hypothesize the model have learned self-agnostic features. It is kind of self-contradictory since the author cannot know if the model really learn self-agnostic features. The concept of self-agnostic sounds intriguing but also abstract. It would be better if author can conduct more analysis on if the relations between tokens are learned. ", "questions_to_authors": "Question A:  When you calculate coarse-grained loss, how do the slot types encoded? For example, ‘this week’ is date_time, date_time, or ‘this week’ is trained as a whole?\n\nQuestion B: in the figure 3, it shows that a sequence of slot types and two utterances are sent into encoders? Are those slot types related to every token/entity in the two utterances? In the Gaussian Transformation Network, the slot types are also trained? Also in the output, the slot labels are entity-level or token-level? Please illustrate more on the figure. \n\nQuestion C: why only test on one unseen target domain? Did you try other target domains ? Usually, the model should be tested on several target domains to prove generalizability?\n\nQuestion D: what is intra-token? tokens in the same entity?\n", "paper_topic_and_main_contributions": "This paper discovers the problem that existing methods on zero-shot slot filling task have poor performance on unseen target domains due to their poor generalizability. They propose a novel hierarchical contrastive learning (CL) approach based on Gaussian embedding to learn and extract slot-agnostic features across utterance-tokens. The main contributions are firstly, they add token-level CL besides entity-level CL and secondly, they use Gaussian embedding so that each token becomes a density instead of a single point. Both innovations help to learn the latent relations between tokens (e.g, if two tokens share the same slot type), in order to improve the generalizability in unseen domains. The results show that the improvement on unseen dataset is non-trivial and the paper conduct comprehensive ablation studies to discuss each component's impact. \n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 025 devote to ->are devoted to\nLine 026 belong -> belonging \nLine 069 pre-treained ->pre-trained\nLine 170 take -> taking\nLine 243 a slot type -> a slot type sequence, based on the context, it should be a sequence of slot types?\nLine 269 in training batch -> in the training batch\n \nToo many long sentences, for example line170-line 175. It would be better to use less many clauses in one sentences. The paper needs more polishing.\n\n"}}, {"review_id": "eiOMEsM8zG", "reviewer": "Reviewer_GdPM", "summary": "", "questions": "1. What is  $n_u$ in Eq.6?\n2. Could you introduce more details in Figure 4 and Figure 5?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. for $n_u$ in Eq.6, we have already presented the explanation and definition of $n_u$ in line 303 and line 304 of our paper.  \n\n2. In Figure 4 and 5,  we provide a visualization to illustrate the effectiveness of our proposed hierarchical contrastive learning(CL), i.e.,  both entity-level CL and token-CL play a critical contribution for the final big performance improvement of unseen slots across various target domains,  while generally they do not hurt the seen slots performance in task-oriented dialogue.     \n\n3. The pictures are saved in SVG format and you can enlarge the PDF file in any size without any loss on fidelity. \n\n$\\textbf{We extremely hard to believe and understand the justice of the scores !}$   \n$\\textbf{please more respect the works of authors,}$  \n$\\textbf{please be more professional and qualified for the reviews, }$  \n$\\textbf{and please be more known about the innovation and contribution of our paper.}$   \n\n$\\textbf{Have you read carefully our paper?  Please list your solid and convincing reasons for your scores.}$\n\nFor novelty, we introduce hierarchical contrastive learning (entity level and token-level) which is the first time study in slot filling of task-oriented dialogue field, moreover, we employ Gaussian embedding to boost the robustness of slot filling models, which is also the first time exploration.   \n\nFor the problem to solve, we aims to improve the unseen slot performance for current zero-shot slot filling models in task-oriented dialogue, whose real performance is not correctly evaluated previously , which is not thoroughly studied and has been proved to have a lot of room for improvement.  \n\nFor experiments, we spend more than half year to do various  experiments and ablation analysis.   \n\nYou should understand whether the problem raised is convincing, why we use entity-level CL, token-level CL and Gaussian Embedding, how we rectify the evaluation for the unseen slot performance, and so on.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed framework achieves SOTA performance.\n2. The paper find unseen slots and seen slots overlapping problem in test set split, rectify the bias by splitting from slot granularity.\n3. The paper conduct experiments on four datasets.", "reasons_to_reject": "1. In my opinion, the paper needs to be rewritten.The article has too much content that is placed in the appendix, which is unreasonable. Moreover, many tables and pictures（Figure 2,3 Table 1,2,3） in the article are too small, making it difficult for readers to see the internal details clearly.\n2. The method is not novel. Contrastive learning has been widely used in slot filling tasks, although the task is not exactly the same. I think the novelty of this simple modification is not suitable for EMNLP.\n3. Many variables (em. $n_u$ in Eq.6) in the formula lack corresponding explanations. I suggest you re-examine the paper carefully and add these explanations.", "questions_to_authors": "1. What is  $n_u$ in Eq.6?\n2. Could you introduce more details in Figure 4 and Figure 5?", "paper_topic_and_main_contributions": "The paper proposes a coarse- to fine-grained contrastive learning based on Gaussian-distributed embedding to learn the generalized semantic relationship between utterance-tokens. Experiments on two datasets show that the proposed framework achieve SOTA performance.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "You should resize the tables and pictures."}, "extra_text": {"Missing_References": "Du X, He L, Li Q, et al. QA-driven zero-shot slot filling with weak supervision pretraining[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). 2021: 654-664."}}, {"review_id": "gAdquZ9ilP", "reviewer": "Reviewer_ohcU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review.  Just a kind reminder, please review more impartially, professionally and conscientiously. \n\n1. Long introduction part aims to help non-expert reviewers to understand the concepts which are difficult to follow, like\nslot filling, slot type, slot entity, etc, more importantly, the background and  the downside or weakness occurred in zero-shot slot filling task which we aims to tackle, that is important because it give the rationales why we propose hierarchical contrastive learning and Gaussian embedding to improve the generalization ability ( unseen slots performance ) of the zero-shot slot filling models. In other words, why we do not use other methods or techniques to improve unseen slots performance (unseen slots generalization ability)?\n\n2. Please point specific claims/arguments that are not sufficiently supported in our paper,  we read carefully for our paper several times again. and find that our claims in the paper is sufficiently and comprehensively supported both qualitatively and quantitatively, for example,  t-SNE visualization and  the detailed case studies, and the extremely detailed and thoroughly diverse ablation studies.\n\n3. Our HiCL mainly focus on big improvement of unseen slots performance and maintain the overall performance of zero-shot slot filling. \nCompared with all baselines, our HiCL improve averaged unseen slots performance by at least 10.1% (absolute value, see table 5) and 9.18% (absolute value, see table 8) for SNIPS and SGD dataset, respectively, which is very significantly and unprecedentedly. Besides the overall performance ( the averaged performance for both unseen and seen slots ) is comparable or even better than existing SOTA models.\n \n4. The previously reported results of the SOTA models for both unseen slots performance and  total performance is biased and exaggerated.  The reasons are:   \n\n &emsp;&emsp;a. the existing methods regards a significant parts of seen slots performance as unseen slots performance, which overstated the actual effect of unseen slots evaluation/reported results in their papers.  \n\n &emsp;&emsp;b. In inference phase, the previous SOTA models only test one or two negative samples (if a given test utterance does not include certain slot type, this sample is negative sample, each sample contains slot_type+utterance) ,  which will results in a higher reported results than actual results,  while we test all negative samples (test all slot types pre-defined in slot type set that are NOT actually appear in test utterance) and results in a more objective overall performance results.  \n\n&emsp;&emsp;c. In practice,  in our iterative label inference method ( described in table 11),  all slot types should indeed be tested one by one (including all negative samples ) for a given utterance in model  inference,   because you never know which slot type exists in one given utterance and which does not exist in advance, which is different from traditional models (like Coach) that predict all slot types once time when given an utterance.  \n\n&emsp;&emsp; In summary, we unbiasedly and correctly re-evaluate the real performance (unseen slots and overall slots performance) both baselines and our methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The idea is simple and reasonable.", "reasons_to_reject": "* The presentation of this paper is a little rough. The Introduction part is too long, and many important results and analysis are in the appendix. The author should reorganize the structure of the paper.\n* The results on SNIPS and SGD seem comparable to previous methods. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a hierarchical CL approach to discover new slot entities for Cross-Domain Zero-Shot Slot Filling. They also use Gaussian embedding to represent token features.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The author should reorganize the structure of the paper."}, "extra_text": {}}, {"review_id": "ajzJz9OVwm", "reviewer": "Reviewer_8SoH", "summary": "", "questions": "1. Is the loss jointly optimized or cross staged as Figure 3 described?\n2. How do the authors get entity level representations in the Coarse training stage? Is it averaged over tokens within a single entity?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review.\n1. For question 1, in implementation, we first update the gradient according to coarse-grained contrastive loss (entity-level),  and then update the gradient according to the combination loss of fine-grained contrastive loss (token-level) and loss of CRF,  however, we empirically found that there is little performance difference between simultaneously gradients update and sequentially gradients update for coarse-grained contrastive learning loss , fine-grained contrastive learning loss  and CRF loss via additional experiments that did not present in our paper.\n\n&emsp;&emsp;The implemented codes are listed below：\n\n            crf_loss, coarse_loss, fine_loss, _ = model(features)\n            coarse_loss = coarse_loss.mean()\n            coarse_losses.append(coarse_loss.detach().cpu().item())\n            loss = crf_loss + model_args.beta*fine_loss\n            loss = loss.mean()\n            losses.append(loss.detach().cpu().item())\n            \n            # first step\n            optim.zero_grad()\n            coarse_loss.backward(retain_graph=True)\n            optim.step()\n            scheduler.step()\n \n            # second step\n            optim.zero_grad()\n            loss.backward(retain_graph=True)\n            optim.step()\n            scheduler.step()\n            pbar.set_description(f\"COARSE_LOSS: {coarse_losses[-1]:.4f}  LOSS: {losses[-1]:.4f}\")\n2.  For question 2, actually both coarse-grained (entity-level) contrastive training and fine-grained (token-level) contrastive training share the same representations of utterances, the difference is their slot labels, e.g, entity-level slot label set is {'O': 0, 'date_time': 1, 'location': 2 } and token-level slot label set is {'O': 0, 'B-date_time': 1, 'I-date_time': 2, 'B-location': 3, 'I-location': 4 }.\nThis representations of utterances  are obtained by firstly encoding utterances token by token with BERT,  and then transforming the  point embeddings of BERT output into Gaussian distributed embeddings.\n\nAs to reasons to reject, we give the following supplements and explanations:  \n1.  We constrain our work and research scope in task-oriented dialogue field, therefore the baselines and datasets for comparison and evaluation are selected from   task-oriented dialogues ( including spoken language understanding sub-field ), the research scope was stated in  abstract(line 001),  introduction(line 030), contribution(line 201), Future work(line 583).\n2. $y_i$ is predicted slot labels and obtained from BERT model output (point embedding) by encoding utterances and then compute the CRF loss (negative log likelihood) with ground truth through Viterbi decoding.   \nFor slot label prediction and compute CRF loss in Eq.9,  they are basically the same as previous SOTA models, like Coach , LEONA or mcBERT, which are not our major innovation and less described in context.\n3. For the similar technique application of Gaussian embedding  in Das et al., 2022, we cited this paper in introduction part (line 122 and line 123) and related work (appendix A.1, line 867-line 869).  If you mean Eq.(2), we could clarify this Gaussian embedding transformation formulation appears even more earlier in several previous literatures(e.g., computer vision paper), it is not initially originated from Das et al., 2022.\nAnd if we implement the Gaussian transformation by referring to the codes of this paper (Das et al., 2022 ), we will clarify it when we release our code in GitHub.\n\n4. Besides, the major difference our method and the method of Das et al., 2022 are below:  \n\n&emsp;&emsp;a. We design a hierarchical contrastive learning (entity-level and token-level, and empirically verify that they complement each other and better than employing only one of them) , while the method of Das et al., 2022 only present the idea of entity-level contrastive learning（the tokens in this paper actually denotes entities). So our paper represents a first of its kind of token-level contrastive learning and \nis the originator and pioneer of proposing the method of token-level contrastive learning. \n\n&emsp;&emsp;b. The second difference is that we focus on improve the unseen slots performance rather than both (unseen slots and seen slots), which did not refer or involve in the work of Das et al., 2022, it did not classify the entities into unseen ones and seen ones.\n\n &emsp;&emsp;c. The research area is very different, we study the field of task-oriented dialogues  (single-turn and multi-turn dialogues datasets),  which inherently has gigantic distinction with other kind of NER tasks, like OntoNotes or CoNLL-2003.\n\n&emsp;&emsp;d. We study in-depth the pre-trained general language model in task-oriented dialogue field and evaluate and compare it with our professionally designed expert model of zero-shot slot filling, which brings some enlightenment even for large language models （LLMs）like ChatGPT, most researchers will be curious about whether vanilla capability of LLMs would replace all small expert models in this scenarios, and whether specially designed generalization techniques still work or bring benefits for pre-trained general models in specific field, our work bring some explorations for this kind of thoughts.\nWe introduce the pre-training expert model (TOD-BERT) as baseline to explore that whether the pre-training model that pre-trained on large corpora of task-oriented dialogue (TOD) are good enough for unseen slots generalization, and whether our method can continue to improve the unseen slots performance on top of it,  this research is also missing in Das et al., 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experiments in this paper are solid under several settings and good performance is achieved over previous baseline methods. \nMoreover, the authors have made fair comparison based on different pretrained backbones.", "reasons_to_reject": "1. Lack of MRC-based SF baselines under the same setting, such as  [1]\n2. The description of method is unclear and it is not easy to follow, such as, how to get y_i in loss computation (Eq.9) is incomplete, which means the CRF seems untrained.\n3. The design of contrastive loss based on Gaussian embeddings seems to be borrowed from Das et al., 2022. But the authors do not cite, explain or clarify it in the methodology section.\n\n[1] Cross-Domain Slot Filling as Machine Reading Comprehension.", "questions_to_authors": "1. Is the loss jointly optimized or cross staged as Figure 3 described?\n2. How do the authors get entity level representations in the Coarse training stage? Is it averaged over tokens within a single entity?", "paper_topic_and_main_contributions": "This paper proposes a hierarchical contrastive learning based method to alleviate the domain shift problem under the cross domain setting.  Extensive experiments are conducted to verify the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Cross-Domain Slot Filling as Machine Reading Comprehension.", "Typos_Grammar_Style_and_Presentation_Improvements": "1. The table design and style of the experimental part need to be improved.\n2. Subsubsection titles(3.2.1-3.2.6) can be replaced by the paragraph styles to save the space."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a novel hierarchical contrastive learning approach for the zero-shot slot filling task. It introduces token-level contrastive learning and uses Gaussian embedding to improve generalizability in unseen domains. Reviewers appreciates the comprehensive nature of the paper and finds the idea of token-level contrastive learning innovative. The detailed ablation studies make the work convincing. However, they have some questions about the method and suggest more clarification and analysis. Reviewers acknowledges the solid experiments and good performance achieved. However, they point out a lack of MRC-based SF baselines, unclear descriptions, and a missing citation. They also have questions about the loss computation and entity-level representations. 

Based on these reviews, the paper is considered of good soundness and shows some originality. However, there are concerns regarding clarity and missing references. The reviewers find the results and ablation studies compelling. The pros of the paper include the clear introduction, innovative token-level contrastive learning, and comprehensive ablation studies. The cons include the lack of clarification in the method section, missing citation for the contrastive loss, and unclear descriptions. Overall, the paper has potential but needs some revisions and clarifications.
Decision: Accept-Findings