Pre-training Intent-Aware Encoders for Zero- and Few-Shot Intent Classification
Authors: Mujeen Sung
James Gung
Elman Mansimov
Nikolaos Pappas
Raphael Shu
Salvatore Romeo
Yi Zhang
Vittorio Castelli
Venue: EMNLP 2023 Main (2023)
Keywords: intent classification
task-oriented dialogue system
TL;DR: We introduce Pre-trained Intent-aware Encoder (PIE), which is designed to align utterances with their intent names.

Abstract: Intent classification (IC) plays an important role in task-oriented dialogue systems. However, IC models often generalize poorly when training without sufficient annotated examples for each user intent. We propose a novel pre-training method for text encoders that uses contrastive learning with intent psuedo-labels to produce embeddings that are well-suited for IC tasks, reducing the need for manual annotations. By applying this pre-training strategy, we also introduce Pre-trained Intent-aware Encoder (PIE), which is designed to align encodings of utterances with their intent names. Specifically, we first train a tagger to identify key phrases within utterances that are crucial for interpreting intents. We then use these extracted phrases to create examples for pre-training a text encoder in a contrastive manner. As a result, our PIE model achieves up to 5.4% and 4.0% higher accuracy than the previous state-of-the-art pre-trained text encoder for the N-way zero- and one-shot settings on four IC datasets.

Pre-training Intent-Aware Encoders for Zero- and Few-Shot Intent Classification
Mujeen Sung
James Gung²
Elman Mansimov
Nikolaos Pappas²
Raphael Shu?
Salvatore Romeo
Yi Zhang?
Vittorio Castelli
Korea University AWS Al Labs²
mujeensung@korea.cer ac.
[gungj mansi maappaaammmmaa........mmmm oon com
(zhongzhu, romeosr, romeosr,iiihngg,iittorrrrr.
witaggnniittttcco hwtp:gggwiiitttrraammaao.o...oo com
Abstract
Pre-training Corpus
Training Example
Utterance: delete this song from playlist Gold Intent: remove from playlist music
Intent classification (IC) plays an important role in lask-oriented dialogue systems. How- ever, IC models ofien generalize poorly when training without sufficient annotated examples for each user intent. We propose novel pre- training method for text encoders that uses con- trastive learning with intent psuedo-labels to produce embeddings that are well-suited for IC tasks, reducing the need for manual anno- lations. By applying this pre-training strat- egy, also introduce Pre-trained iitent-awar Encoder (PIE), which is designed to align en- codings of utterances with their intent names. Specifically, we first train tagger to identify key phrases within utterances that are crucial for interpreting intents. We then these extracted phrases to create examples for pre- training a text encoder a contrastive manner. As result, PIE model achieves up 5.4% and 4.0% higher accuracy than the previous state-of-the- art text encoder for the N- N-way zero- and one-shot settings on four IC datasets.
1
Annotating Pseudo Intent
IRL tagger
delete song playlist
Action: delete Argumentt: song. playlist
Intent-Aware Contrastive Learning
uiterarce
pseudo intent
De Intien
X,
x erase this track from the playlist
K can you buy me some movie tickets
restaurant reservation
open checking account
Figure Overview of pre-training the intent-aware coder (PIE). Given an utterance æ1 from pre- preinrannin corpus, we generate a pseudo intent name yf pecado using labels from the intent role labeling (IRL) tagger. Our PIE model is then optimized by pulling the gold ut- terance gnnd gold intent yı, and pseudo intent yi prouoo close to the input utterance in the embedding space.
1 Introduction
Identification of user intentions, a problem known as intent classification (IC), plays an important role in task-oriented dialogue (TOD) systems. How- ever, it is challenging for TOD developers to col- lect data and re-train models when designing new intent classes. Recent studies have aimed tackle this challenge by applying zero- and few-shot text classification methods and leveraging the seman- tics of intent label names (Liu et al., 2019a; Krone et al., 2020; Burnyshev et al., 2021; Mueller et al., 2022; Zhang et al., 2022; Lamanov et al., 2022; Liu et al. 2022).
with transformer-baset text encoders. Prototypi- cal Networks use text encoders to construct class representations and retrieve correct classes given queries based on similarity metric. Dopierre et al. (2021a) also stress that few-shot learning techniques and text encoders can have an orthog- onal impact on classification performance. Thus, although some studies have focused on improving learning techniques for few-shot IC tasks (Dopierre et al., 202 lb; Chen et al., 2022), better text encoder selection should also be considered as an impor-
Dopierre et al. (2021a) compare various classifi- cation methods on few-shot IC tasks and find that Prototypical Networks (Snell al., 2017) consis- tently show strong performance when combined
Work performed during an interssiipan at AWS Al Labs.

---

tant research direction. Ma al. (2022) observe that sentence encoders pre-trained on paraphrase or natural language inference datasets serve as strong text encoders for Prototypical Networks. However, existing sentence encoders are not explicitly de- signed to produce representations for utterances that are similar to their intent names. Therefore, their abilities are limited in zero- and few-shot set- tings where predictions may heavily rely the semantics of intent names. Pre-training encoders to align user utterances with intent names can miti- gate this issue; however, it is typically expensive to obtain annotations for a diverse intent set.
2 Background: Prototypical Networks for Intent Classification
Prototypical Networks (Snell et al., 2017) is meta- learning approach that enables classifiers to quickly adapt to unseen classes when only few labeled ex- amples are available. Several studies have demon- strated the effectiveness of Prototypical Networks when building intent classifiers with few exam- ple utterances (Krone et al., 2020; Dopierre et al., 202 la; Chen et al., 2022). They first define a few- shot IC task, also known as an episode in the meta- learning context, with K example utterances from N intent classes (i.e., KxN utterances in single episode). At the training time, the intent classifiers are optimized on series of these episodes. Ex- ample utterances for each intent class are called a support set, and are encoded and averaged to produce a class representation, called a prototype. This can be formulated as follows:
In this paper, we propose a novel pre-training method for zero- and few-shot IC tasks (Fig- ure 1). Specifically, we adopt intent role label- ing (IRL) (Zeng et al., 2021), which is an approach for identifying and assigning roles to words or phrases that are relevant to user intents in sentences. Once we obtain the IRL predictions, we convert them to the pseudo intent names of query utter- ances and use them to pre-train the encoder a contrastive learning fashion. This intent-aware con- trastive learning aims to not only align utterances with their pseudo intent names in the semantic em- bedding space, but also to encourage the encoder pay attention the intent-relevant spans that important for distinguishing intents. To the best of our knowledge, this work is the first to extract key information from utterances and use it as pseudo labels for pre- pre-training intent-aware text encoders.
Cn = K
Σ En
fø(In,)
(1)
where S₃ denotes the support set of the n-th intent class, In,i denotes the i-th labeled example of the support set Sn,, fo(-) denotes trainable encoder, and cn denotes the n-th prototype. At the inference time, the task to map the query utterance repre- sentation the closest prototype in a metric space (c.g., Euclidean) among the N prototypes. When there are N intent classes and each intent class has K example utterances, this setting is called N-way K-shot intent classification.
The contributions of our work are as follows:
Ma et al. (2022) suggest that leveraging intent names as additional support examples is beneficial in few-shot IC tasks because the semantics of intent names can give additional hints to example utter- ances. When intents are used as additional support examples, the new prototype representations can be formulated as follows:
First, we propose an algorithm for generating pseudo intent names from utterances across several dialogue datasets and publicly release the associated datasets.
Second, by applying intent-aware contrastive learning on gold and pseudo intent names, we build Pre-trained intent-aware Encoder (PIE), which is designed to align encodings of utter- ances with their intent names.
Jabel
Σ fo(2n,)) + fo(v.)) (2) En
K+1
where Yn is the intent name of the utterance in the n- th support set, and clabel is the n-th prototype using intent names as support. By using intents support examples, is possible to classify input utterances without example utterances in a zero-shot fashion. Specifically, the prototypes in Equation (2) can be calculated as c = fy())) based solely on intent names, which facilitates the zero-shot IC.
Finally, experiments four IC datasets demonstrate that the proposed model out- performs the state-of-the-an work (Dopierre et al., 2021b; Ma et al., 2022) by up 5.4% and 4.0% on N-way zero- and one-shot set- tings, respectively.

---

3 Pseudo Intent Name Generation
tion, and test with 3,121 / /79/ 379 utterances. respectively, which approximately an 80:10:10 ratio.
To pre-train an encoder fo that works robustly in zero- few-shot IC settings, variety of prede- fined intent names are required. Because anno- tating them expensive, we opt to automatically generate pseudo intent names from utterances in our pre-training data. To annotate pseudo intents, we employ tagging method, intent role label- ing (IRL). IRL can be considered similar to se- mantic role labeling (SRL), which is task of as- signing general semantic roles to words phrases in sentences (Palmer al., 2010). However, IRL. focuses on providing an extractive summary of the intent expressed in user's utterance, annotating important roles with respect to the goal of the user rather than a predicate. Specifically, it tags words or phrases that are key to interpret intent.
Label
Count
Example
Action
2,163
I want to book..r flight
Argument
2,011
I want book flight,..
Request
3,002
Can you show.., me my account balance
Query
3,247
Can you show
account balance..
Can you show me my account
Slot
2,030
balance for my checking accountar
Problem
45
I'm starting to get hungry...
Table I: Statisties and examples of each IRI. lahel from 3,879 utterances.
3.2 Training the IRL Tagger
IRL was first introduced by Zeng et al. (2021) for discovering intents from utterances, but their tagger focuses only Chinese utterances. In this section, we outline the process of building the IRL tagger from scratch. We provide a description of how we annotate IRL training data on English utter- ances (Section 3) the training procedure for the IRL tagger (Section 3.2), and the utilization of TRL predictions for generating pseudo intent names to pre-train model (Section 3.3).
Using manually curated IRL. annotations, we for- mulate IRL sequence tagging problem. Specif- ically, assign each token in an utterance with one of the 13 IRL labels under the Beginning- Miide-Outside (BIO) scheme (e.g., B-Action, I- Action, or O). The IRL model is, then, trained predict the correct IRL labels of the tokens using the cross entropy loss. We use RoBERTa-base (Liu et al., 2019c) as the imitial model for the IRL tagger. Table shows the precision, recall, and F1 scores of cach IRL label on the test set.
3.1 Annotating Intent Roles
We define six intent role labels, Action, Argument, Request, Query, Slot, and Problem, for extracting intent-relevant spans from utterances. Action is word or phrase (typically a verb or verb phrase) that describes the main action relevant to an intent in an utterance. Argument is an argument of an ac- tion, or entity/event that is important to interpreting an intent. Request indicates a request for some- thing such as question information-seeking verb. Query indicates the expected type of answer question request for information, or re- quested entity to be obtained or searched for. Slot is an optional/variable value provided by the speaker that does not impact the interpretation an intent. Finally, Problem describes some problematic states or events, and typically makes an implicit request.
Label
P (%)
R (%)
Fl (%)
Action
89.3
86.9
88.1
Argument
85.8
85.4
85.6
Request
90.9
93.5
92.1
Query
92.2
95.8
94.0
Slot
82.8
84.6
83.7
Problem
35.6
41.7
38.1
Table 2: Precision, recall, and F1 scores of each IRL label on the test set.
3.3 Generating Pseudo Intents
After obtaining the IRL tagger, we leverage it to predict IRI. labels for tokens in utterances from pre-training corpus described in Scction 5.2. To generate pseudo intent names, simply concate- nate all spans that have been predicted IRL labels in each utterance. Table 3 shows some examples of IRL predictions from utterances and corresponding pseudo intent names.
Based on these definitions, manually anno- IRL labels SGD (Rastogi et al., 2020). Table shows the statistics and examples of each IRI. label from 3,879 utterances. To train and evaluate the IRL tagger, we split annotations into training, valida-
subset of utterances from

---

Utterances
Pseudo Intents
Gold Intents
I'm calling because i'd like to open.cr account... I'd like to op savings,, account,c: please So need to sign upxrr for savings... account,ac
open account open savings account
open account
sign up savings account
Can you buy vrs I am looking to book,cr movie tickets.
movie tickets....
buy movie tickets book movie tickets purchase movie tickets
buy movie tickets
I am looking to purchase... movie tickets...
Delete,. this songase from playlist,a Erase, this track..o from the playlist
delete song playlist erase track playlist remove song
remove from playlist music
Could you remove.s this song.s permanently
Table 3: Some examples of IRL predictions (boldfaced) from utterances, extracted pscudo intent names, and gold intent names annotated in the original dataset.
4 Intent-Aware Contrastive Learning
The second supervised positive pair is between the input utterances and their gold utterances. We define gold utterances as randomly sampled utter- ances that share the same gold intent names the input utterances:
We aim to build encoder that produces simi- lar representations between utterances and the cor- responding intent names. In this section, we in- troduce the intent-aware contrastive learning ap- proach using triples of an utterance, gold intent, and pseudo intent from various dialogue datasets.
N Σ
Lgold ullerance
lxs, soldd
(5)
N
Our training objective is designed to align the representations of utterances and their intent names in the semantic embedding space. For this pur- pose, the InfoNCE. loss (van den Oord et al., 2018), which pulls positive pairs close each other and pushes away negative pairs. The loss for the i-th sample T; formulated as follows:
where gold is the gold utterance x;. Finally, the semi-supervised positive pairs are between the input utterances and their pseudo intent names:
1 N
Lpeendo
Σ ((xi, pacudo),
(6)
E(x;; 1) = exp [s]smm/((ss) So(v)))
(3)
Σx exp (iim(ff(z.))) fo
where yt pseudo denotes the pseudo intent name of x; constructed by the IRL tagger, described in Sec- tion 3.3
where y = (11, WW... MN) are pairs of the input with batch size of N, and sim denotes the cosine similarity between two embeddings. Again, fo(:) denotes any text encoder that represents in- tent names or utterances in the embedding space. Note that pairs that are not positive batch are treated as negative pairs.
Our final loss is a combination of these three losses as follows:
Lgold_intent + Ggold_utterance pseudor (7)
where À is the weight term of the semi-supervised loss term.
We here define three types of positive pairs, two of which are supervised and one is semi-supervised. The first of the supervised positive pairs is between the input utterances and their gold intent names annotated in the pre-training datasets. The equation used is follows:
Experiment
5.1 Baselines
To evaluate the effectiveness of our proposed PIE model, compare it with the following state- of-the-art approaches for few-shot IC tasks: Pro- toNet (Dopierre al., 2021a) and ProtAugment (Dopierre et al., 202 as fine-tuning methods, and SBERT PaERTparp (Ma et al., 2022) as pre-trained text encoder.
1 N
Cgold intent
Σ
(xx goold)
where gold is the gold intent name of z;.

---

ProtoNet is meta-training approach that fine- tunes encoders by using series of episodes con- structed on task-specific training sets. ProtAug- ment is an advanced method derived from Pro- toNet, which augments paraphrased utterances within episodes to mitigate overfitting caused by the biased distribution introduced by limited num- ber of training examples. The authors of ProtoNet and ProtAugment perform additional pre-training SERT-base-cased (1 (110M) using training utterances and the language model objective, and use their initial model. We refer to this model as BART inspired by task- addativee pre- trainingn (TAPT) (Gu- rurangan et al., 2020).
biguity, use the first-turn utterance of each dia- logue from these datasets. Furthermore, the num- ber of utterances between intents the raw datasets is highly imbalanced. To alleviate this imbalance, we set the maximum number of utterances per in- tent of the TOP and DSTC11-T2 datasets to 1000 and the SGD and MultiWOZ 2.2 datasets to 100. We then annotate the IRL labels on the utterances using the IRL tagger. Based on the IRL predic- tions, we filter utterances when no Action, Argu- ment, or Query labels are detected, because they are likely to lack information for interpreting user intents. Finally, we treat MultiWOZ 2.2 as the val- idation set for tuning the hyperparameters of the pre-training stage. Table 4 summarizes the statis- tics of the datasets.
SBERTPaapphne Paraphrasc is text encoder pre-trained on large-scale paraphrase text pairs (Reimers and Gurevych, 2019). Ma al. (2022) discover that this pre-trained text encoder can produce good ut- terance embeddings without any fine-tuning on task-specific datasets. Although the authors lever- age SBERT solely at the inference stage of Prototypical Networks, we conduct additional experiments by fine-tuning the encoder using Pro- toNet and ProtAugment baselines. Note that we reproduce the performance of SBERT Paraphrasc using sarrapprrrse-mppttbaae---e
5.3 Downstream Datasets
Intents
Dataset
Domains
Utterances
Train
Vahd
Test
Banking?7 HWU64
13.083
11,036
16
Liu54
25,478
Clinc150
22,500
Table 5: The statistics of four IC datasets.
10M), which has the same number of parameters as BERT-base- cased, for a fair comparison.
We evaluate PIE model and baseline models on four IC datasets (Table 5). Banking77 (Casanueva et al., 2020) an IC dataset in the banking domain. As there are many overlapping tokens between intent names (e.g. 'verify top up', 'top up limits', and 'pend- ing top up'), fine-grained understanding is required when correctly classifying intents for this dataset. HWU64 (Liu et al., 2019b) is dataset in 21 dif- ferent domains, such alarm and calendar for home assistant robot. Liu54 (Liu al., 2019b) a dataset collected from Amazon Mechanical Turk, and workers designed utterances for given intents. Clinc 150 (Larson et al., 2019) is a dataset that in- cludes a wide range of intents from ten different domains such as 'small talk' and 'travel'.
5.2 Pre-training Datasets
Gold
Pseudo Intents
Dataset
Utterances
Intents
(+x2)
31,111
61
23.711
DSTCI1-T2
4,459
148
3,304
Train
SGD Tota
3,561
44
2,647
39,117
252
29,577
Val
MuliWOZZ2 22
586
10
Table 4: Pre-training datasets for the PIE model.
We collect four dialogue datasets to pre-train and one to validate our encoder: TOP (Gupta et al., 2018)), TOPv2 (Chen et al., 2020), ASTC11-T2 (Gung et al., 2023), SGD (Rastogi et al., 2020), and WultiWOZ 2.2 (Zang al., 2020). Dialogues in SGD and MultiWOZ 2.2 datasets consist of multi-turn turt utterances, and these utterances are of- ten ambiguous when context around them is not given (e.g., 'Can you suggest something else?' is labeled as "LookupMusic".. To minimize this am-
Banking77
HWU64
Liu54
Clinc150
TOP (+v2)
0
2
2
DSTC11-T2
3
0
SGD
0
All
3/77
2/64
0/54
6/150
Table 6: Number of overlapping intent names between the pre-training data and downstream data.
'hnnps//uuginnffacccoommmcmm trnsstrrmrrsparpphrs---pnehaaaer

---

Banking77
HWU64
Liu54
Clinc150
Average
Method
Fine-tuning
K-0
K-1
Kan
K=1
KWD
K-1
K-0
R-1
K=n
K-
BERT
50.821:
12.1
52.7-2.6
50.7a41
51.6
SBERT
83.6 83.6g18
82.2
82.3-22
94.74a0
85.7
PIT: (Ouns)
86.1213
86.0217
85.6_1.9
96.2
88.5
--BETTAAP
27.7
41.0g15
38.3
53.6e1.5
51.7
60.0=1.1
39.5
51.8
39.3
51.6
L- Purptraee
86.9119
90.910d
83.611a
89.0
79.9.a.
89.8
94.3g11
97.7
86.2
91.9
L-PIE (Ouns)
88.3=22
92.4an7
87.7=26
92.2+16
83.8
91.7=14
96.5
98.3
89.1
93.7
L-BERT TAPP
85.7
91.5210
81.4g1
86.611.3
80.3
88.2.1.a
93.0g19
96.9
85.1
90.8
L-SBERTraagtaee
ProtoNet
90.9=1.
94,5
15.8428
91.1
83.7=4.8
92.1=16
97.1
98. 98.6ga2
89.4
94.0
L-PIE (Ours)
90.7g22
94.3
86.6ga
92.1
85.0ga.
92.4
97.3200
98.61a3
89.9
94.4
BERE
89.2g2.1
93.4205
87.0126
89.8
83.0.4.
90.9_0.9
95.3110
97.7.00
L-SBERTRaalaae
ProtAugment
92.3=1.1
94.8204
87.3
91.7+1.6
84.
92.5-1.4
98.5 98.5a02
90.2
94.4
L-PIE (Ours)
92.4g1.0
94.8204
88.8g30
92.4g1.6
86.0ga.3
92.9_1.3
97.620.
98.7g00
91.2
94.7
Table 5-way K-shot intent classification performance of pre-trained models with and without fine-tuning on four test sets. Averaged accuracies and standard deviations across five class splits reported. The 'L-' prefixes indicate the use of intent label names when creating prototypes, enabling zero-shot evaluation. Highest scores are boldfaced.
Before proceeding, we examine the number of intent names that overlap between the pre-training data and downstream data (Table 6). When com- paring the intent names, we first apply stemming ("restaurant reservation' + 'restaur reserve") and arrange the tokens in alphabetical order ('restaur re- serv' + 'reserv restaur') )) for each intent name. This approach aims to maximize the recall of overlap- ping intent names. Consequently, we find that only Il out of 345 intent names from the downstream data overlaps (c.g, 'reserve restaurant' intent in the pre-training data and 'restaurant reservation' intent in the downstream data).
for example, 27-way for Banking77 and 50-way for Clinc150. We set K, which is the number of examples per intent in an episode, to O and to experiment with zero- and one-shot IC. Finally, we treat intent labels as examples when creating proto- types for each intent. This enables experiments in the zero-shot setting and enhances performance in the few-shot setting. To denote the usage of labels as examples, we append 'L-' prefixes the method names (e.g., L-PIE).
5.5 Results
5-way K-shot intent classification Table 7 shows 5-way K-shot IC performance on four test sets. The results demonstrate that PIE achieves an average accuracy of 88.5%, surpassing SBERTPamphrse- which is considered the strongest baseline model, by 2.8% in the one-shot setting. This highlights the effectiveness of our pre-training strategy. Additionally, where intent labels are used as examples, our L-PIE model achieves 89.1 1% and 93.7% in the zero-shot and one-shot settings, respettvely consistently outperforming L- @BERTParaphraee by 2.9% and 1.8%. It is worth not- ing that the L-PIE model also significantly outper- forms L-BERTTAPT + ProtoNet, which fine-tunes an encoder on the target datasets, by a substantial margin of 4.0% and 2.9%. This shows that our proposed approach builds an effective intent clas- sifier that performs well even prior to fine-tuning on task-specific data. Our L-PIE model shows further improvement when fine-tuned with Pro- tAugment, outperforming the strongest baseline L-SBERT + ProtAugment by 1.0% in zero- shot IC.
5.4 Implementation Details
Here, we describe detailed information when pre- training our PTE model and employing for zero- and few-shot IC tasks.
We use paraphrase-mpar mmnet-base-ve the same encoder used in the SBERT Paraphrase baseline, as an initial model for further pre-training in our ap- proach. The hyperparameters are tuned based on the validation set described in Section 5.2. As result, we set the training epochs 1, the learning rate to le-6, the batch size to 50, and À to 2.
After pre-training, we apply the model to zero- and few-shot IC tasks in the 5-way and N-way settings. The baselines we compare with only ex- periment with a 5-way setting where the task is predicting the correct intent from among five candi- date classes. We further include an N-way setting, where N can be much larger than five because, in practice, it is often required assign more than five intents building TOD systems. When evalu- ating models on the N-way setting, we use all the intent classes in the test set as candidate intents,

---

Banking77N-2r
Liu54N-18
Clinc150x:50
Average
Model
Fine-tuning
K-0
K-1
K40
K=1
KWD
K-1
K-0
K-1
K=n
K-
BERT SBERT
28 A8.4
28 28.421.
32.1
26.3g00
28.8
62.3
58.0
68.8-3.1
76.7417
66.5
PIT: (Ouns)
65.3
66.042.
73.5-28
22.5
71.8
--BETTAAP
9.0g1.1
19.3
25.4g1a
33.4
21.
33.1
17.422
27.4
18.3
28.3
L- Purptraee
68.9 68.914.
76.1
61.4ga
69.012.4
66.724.
79.4_
76.3228
87. 21.
68.3
78.0
L-PIE (Ouns)
71.444
78.2+22
68.7
76.8
71.4464
82.6.8.2
83.
90.4
73.7
82.0
L-BERT TAPP
65.114.7
73.8
53.4g45
61.3
63.7
75.6_za
73.5
82.9
63.9
74.2
L-SBERTraagtaee (Ours)
ProtoNet
74.9=4.2
81.9119
26.0
74.4
71.8
82.9=8.5
84.3+14
90.9+10
74.0
82.5
2..
814422
67.2,
76.9
73.7.m.3
83.6.3.2
85.4.1
91.2g11
75.2
83.3
BERE
2..1
78.5122
63.1
69.4
69.5
80.5_1.
79.5126
86.5g11
L-SBERTRaalase Panglasse
ProtAugment
77, 47.4
82.8+1.7
67.5
75.5
12.7
83.6=a.0
84.521.
90.9a1.
75.4
83.2
L-PIE (Ours)
27.6
82.9
70.6256
77.4g15
74.6
84.2_a.1
86.5.18
91.8g10
77.3
84.1
Table N-way K-shot intent classification performance of pre-trained models with and without fine-tuning on four test set. Averaged accuracies and standard deviations across five class splits are reported. The 'L-" prefixes indicate the use of intent label names when creating prototypes. Highest scores are boldfaced. Fine-tuning is done in the 5-way setting due memory constraints.?
N-way K-shot intent classification We show- case the performance of our PIE model and the baselines in more challenging and practical sce- nario (Table 8). In this scenario, the intent for user utterances needs be classified among signifi- cantly larger number of intent classes (e.g., 10x for Clinc150), The results show that our L-PIE model achieves 73.3% and 82.0% in zero- and one-shot settings, respectively, outperforming the baseline L-SBERTParaphnee by 5.4% and 4.0%. These per- formance improvements are significantly higher than those observed in the 5-way K-shot IC task. This indicates that our PIE model performs well in practical scenarios, as stated above.
We leverage dialogue datasets for building the PIE model as described in Section 5.2. Here, perform an ablation study over the pre-training datasets N-way K-shot IC tasks (Table 9). The result shows that using the TOP (+TOPv2) dataset, which has 31K utterances, 61 gold intents, and 23K pseudo intents, improves the performance the most over L-SBERT Paraphrase (indicated as None Ta- ble 9). Specifically, there is an improvement of 4.8% and 3.7% in the zero- and one-shot settings, respectively. Although using other datasets, such as SGD DSTCI1-T2, does not improve the perfor- mance in comparison with using the TOP dataset, observe that merging them further improves the overall performance on downstream tasks.
6 Analysis
6.2 Pre-training Loss Ablation
6.1
Pre-training Corpus Ablation
HWU64N=25
Clinc150
Method
Average
K=0
K=1
K=0
K=1
Pre-training Data
K=0
K=l
L-PIE (Ours)
68.7
76.8
83.1
90.4
None
68.3
78.0
Cpscudo
66.9
75.7
81.4
89.7
Ggold_ Lgold_ Lntte_ intent
68.7
76.2
82.7
90.4
SGD
69.6
78.6
Cgold_utteran
68.7
75.9
82.7
90.0
DSTC11-T2
70.7
79.4
TOP (+TOPv2)
73.1
81.7
Table 10: Loss ablation in N-way K-shot intent classifi- cation with accuracy on HWU66 and Clinc1500.
DSTC11-T2
73.3
81.9
+ SGD
73.7
82.0
As described in Section 4, our intent-aware con- trastive loss comprise three sub-losses, Lgold intent.
Table 9: Pre-training data ablation of L-PIE in N-way K-shot intent classification on four datasets (Banking77, HWU64, Liu54, and Clinc150). 'None' indicates SBERT Parphhuse which used as the initial encoder before applying the intent-aware aware pre- peetrriinig method.
"When fine-tuning models in the N-way setting, we en- countered out-of-memory issues on a single 12GB GPU. To work around this limitation, we fine-tuned models in the 5-way setting and evaluated them in the N-way setting.

---

95
100
85
90
95
80
70
L-PI (Ours) PA
75
LIE (Ours) PA
80
L-PE (Ours) + PA
L-SBERT Laaaaea
Trananue
70
L-SBERT eeeeese.
LIBERTo
PA
LBBER PA
LEBRTTO Tour PA
(a) HWU64N=25
(b) Liu54Na18
(c) Clinc150xeh
Figure 2: Performance N-way K-shot intent classification with varying K. PA refers to ProtAugment.
100
85
80
80
90
(Ours)
70
(Qurs)
LPE (Ours]
60
LISEERTAa LSBRRT
PA
65
L.SBET
PA
75
LBBERT Toeaoon PA
LEETTS +
LIBERTsUr
PA
BERT
10
18
30
N
N
HWU64
Liu54
(c) Cline150
Figure 3: Performance on N-way O-shot intent classification with varying N.
goold utterance. and ppeudd- To the benefit of using these losses during pre-training, ablate each loss function in the N-way K-shot IC tasks on HWU64 and Cline150 (Table 10). The re- sults indicate that two sub-losses, Ggold intent and Ltternneeeeacee show relatively marignal improve- ments. However, it is noteworthy that Epecudo serves as the key sub-loss for PIE, highlighting the effectiveness of using pscudo intents. Specifi- cally, removing pseudd from the final loss results in up 1.8% and 1.1% degradation in performance in the zero- and one-shot settings, respectively.
more practical and realistic settings where many user intents are used for the TOD system and only few utterances are available.
6.4 Impact of Overlapping Intents
Banking77
HWU64Ka5
Clinc150y.sr
overlapping intents
72.0
69.6
82.8
Table II: N-way 0-shot IC performance of L-PIIE by removing overlapping intents from evaluation sets.
As shown in Table 6. there are few overlap- ping intent names between pre-training data and downstream data (except for Liu54). These coinci- dental overlaps can hinder an accurate evaluation of the generalization ability of our model. To un- derstand the impact of intent overlaps, we also mea- sure the performance using only non-overlapping intents (Table 1 1)). We observe that the impact is marginal enough to be neglected, and surpris- ingly, removing overlapping intents rather can lead to better performance on Banking77 and HWU64. Through further analysis, we discover that this is partly because of the bias towards pairs of utter- ances and intent annotated in pre-training datasets. For example, an utterance 'please play my favorite
6.3 Varying K and N
We visualize the performance of the PIE model in challenging N-way K-shot IC task settings where the number of example utterances K or the num- ber of candidate intent classes varies. Plots of performance at varying K (Figure 2) show that our model has consistently higher performance than the baselines, and the performance improvement of model is the largest when K is small (c.g., K=0). Plots of performance varying (Figure show that the performance improvement of our model increases as the number of intents N increases (i.e. increasing from N=5 to N=50). These visualiza- tions reveal that the PIE model can be utilized in

---

song' in pre-training data has an intent 'play music". Our model then incorrectly predicted 'play music' for a test utterance 'that song is my favorite', where the correct intent is 'music likeness'.
Acknowledgements
We gratefully acknowledge the members of AWS AI for providing valuable feedback on the project. We would also like to thank the anonymous review- ers for their insightful comments. The part of Mu- jeen Sung's graduate study and, accordingly, this work supported by National Research Founda- tion of Korea NVRF-20231AA233044777
7 Conclusions
In this work, we propose pre-training method that leverages pseudo intent names constructed using an IRL tagger in semi-superviser manner, followed by intent-aware pre- pre-training (PIE). Experiments on four intent classification datasets show that our model achieves state-of-the-ee performance on all datasets, outperforming the strongest sentence en- coder baseline by up to 5.4% and 4.0% in N-way zero- and one-shot settings, respectively. Our anal- ysis shows that PIE performs robustly compared to the baselines in challenging and practical settings with large number of classes and small number of support examples. In future work, we will explore the use of IRL and our PIE model in multi-label in- tent classification or sut-of-scope detection tasks.
References
Pavel Burnyshev, A. Bout, Anfrey Bout, Valentin Ma- lykh, and Irina Piontkovskaya. 2021. Infobert: Zero- shot approach to natural language understanding ing contextualized word embedding. Recent Ad- vances in Natural Language Processing.
lñigo Casanueva, Tadas Temčinas, Daniela Gerz. Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Pro- ceedings of the 2nd Workshop on Natural Language Processing for Conversational AI.
Junfan Chen, Richong Zhang, Yongyi Mao, and Jie Xu. 2022. Contrastnet: A contrastive learning framework for few-shot text classification. AAAI.
Limitations
Xilun Chen, Asish Ghoshal, Yashar Mehdad, Luke Zettlemoyer, and Sonal Gupta. 2020. Low-resource domain adaptation for compositional task-oriented semantic parsing. In EMNLP.
One limitation of our method is that while it lever- ages annotations from the IRL tagger, the detection of spans for certain labels, such as 'Problem," is not accurate enough (38.1% F1 score). This is likely due to the relatively short number of annotations of this type in the training set (45 annotations). To mitigate this limitation, we could consider annotat- ing more instances of this label or implementing techniques for handling imbalanced labels.
Thomas Dopierre, Christophe Gravier, and Wilfried Logerais. 2021a. neural few-shot text classification reality check. In EACL.
Thomas Dopierre, Christophe Gravier, and Wilfried Logerais. 2021b. Protaugment: Intent detection meta-learning through unsupervised diverse para- phrasing. ACI/JJNNLP.
Another limitation is that we currently treat all IRL labels equally when constructing pseudo in- tents. However, the importance of each label in interpreting intent can vary. To address this, we plan to investigate treating different labels differ- ently when pre-training the encoder (e.g. by giving more weight to 'Action' and 'Argument' labels and less weight to 'Slot' labels).
James Gung, Raphael Shu, Emily Moeng, Wesley Rose, Salvatore Romeo, Yassine Benajiba, Arshit Gupta, Saab Mansour, and Yi Zhang. 2023. Intent induction from conversations for task-oriented dialogue track at dste 11. DSTC11 Workshop at SIGDIAL x INLG 2023.
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. 2018. Semantic parsing for task oriented dialog using hierarchical representations. EMNLP.
Ethics Statement
Suchin Gururangan. Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy. Doug Downey, and Noah Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In ACL.
Our proposed method is for enhancing zero- and few-shot intent classification, and it does not raise any ethical concerns. We believe that this research has valuable merits that can lead more reliable task-oriented dialogue systems. All experiments in this study were carried out using publicly available datasets.
Jason Krone, Yi Zhang, and Mona Diab. 2020. Learning to classify intents and slot labels given handful of examples. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AL.

---

Dmitry Lamanov, Pavel Burnyshev, Katya Artemova, Valentin Malykh, Andrey Bout, and Irina Pio- ntkovskaya. 2022. Template-based approach to zero- shot intent recognition. In Proceedings of the 15th International Conference on Natural Language Gen- eration, pages 15-28.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara. Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ, 2.2 A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational Al.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan Kummerfeld, Kevin Leach, Michael Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of- scope prediction. EMNLP M-NNLP.
Zengfeng Zeng, Dan Ma, Haiqing Yang. 7hen Gou, and Jiamping Shen. 2021. Automatic intent- slot induc- tion for dialogue systems. Proceedings of the Web Conference 2021.
Yiwen Zhang. Caixia Yuan, Xiaojic Wang, Ziwei Bai, and Yongbin Liu. 2022. Learn adapt for general- ized zero-shot text classification, In ACI.
Han Liu, Xiaotong Zhang, Lu Fan, Xuandi Fu, Qimai
Li, Xiao-Ming Wu, and Albert Y.S. Lam. 2019a. Re- constructing capsule networks for zero-shot intent classification. In EMNI.P-ICCNIP.
Han Liu, Siyang Zhao, Xiaotong Zhang, Feng Zhang, Junjie Sun, Hong Yu, and Xianchao Zhang. 2022. simple meta-lest veeta-lernnigg paradigm for zero-shot intent classification with mixture attention mechanism. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, pages 2047-2052.
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019b. Benchmarking natural lan- guage understanding services for building conversa- tional agents. /WSDS.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019c. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint 1907:1197.1166
Tingting Ma, Qianhui Wu, Zhiwei Yu, Tiejun Zhao, and Chin-Yew Lin. 2022. On the effectiveness of sentence encoding for intent detection meta-learning. In NAACI.
Aaron Mueller, Jason Krone, Salvatore Romeo, Saab Mansour, Elman Mansimov, Yi Zhang, and Dan Roth. 2022. Label semantic aware pre- pre-training for few-shot text classification. ACI..
Martha Palmer, Daniel Gildea, and Nianwen Xue, 2010. Semantic role labeling. SLHLT.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghay Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guiee guided dialogue dataset. In AAAI.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In EMNLP-IJCNIP.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. NeurIPS.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 0807.03488.


Reviews:
[{"review_id": "WJgMSCp6Dq", "reviewer": "Reviewer_wRCq", "summary": "", "questions": "Is there any ways that the author could share the details of the annotated IRL datasets to help community to boostrap a tagger.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful reading, encouraging feedback and interest in our method and dataset.\n\nQ1: “Is there any way that the author could share the details of the annotated IRL datasets to help the community to bootstrap a tagger?” We will publicly release our IRL annotations to help readers gain a clearer understanding of the annotation descriptions outlined in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposed a novel and inspiring method for the IC task and achieve STOA performance. The method can potentially be useful for other NLP tasks as well.", "reasons_to_reject": "No obvious flaws.", "questions_to_authors": "Is there any ways that the author could share the details of the annotated IRL datasets to help community to boostrap a tagger.", "paper_topic_and_main_contributions": "The authors introduce a unique pre-training approach for text encoders, leveraging contrastive learning combined with intent pseudo-labels. The primary motivation behind this methodology is to produce embeddings highly conducive to Intent Classification (IC) tasks, potentially minimizing the dependency on manual annotations. The proposed method is called PIE (Pre-trained Intent-aware Encoder). The PIE model's design intends to synchronize the encodings of utterances with their associated intent designations. The procedure is detailed and appears to be executed in two primary stages: 1. Initial training of a tagger that pinpoints pivotal phrases within utterances, which are deemed essential for discerning intents. 2. Utilization of these identified phrases as instances for the pre-training of a text encoder, executed in a contrastive fashion. As per the experiments demonstrated, there is notatble improvement in accuracy with the PIE model recording a 5.4% and 4.0% augmentation in performance when compared to the erstwhile state-of-the-art pre-trained sentence encoder. This observation is consistent for the N-way zero- and one-shot settings evaluated across four distinct IC datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6kxo1WUjGn", "reviewer": "Reviewer_qFW1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thorough reading and comprehensive feedback! We’re glad you found our approach innovative and interesting and that you found our presentation to be clear.\n\nRTR 2A, 2B, 2C, 2D: On IRL annotations, we acknowledge that our role labeling scheme does not capture all nuances of semantics in intent utterances. However, because our main goal is to facilitate intent encoder pre-training through generation of pseudo labels, we believe a simplified representation that captures key information in each utterance is sufficient. We intentionally adopt a small/flat (relationless) tagging process to avoid annotation complexity, even though this sacrifices some representational capacity. Of course, a more sophisticated semantic parse could facilitate more robust representations, which we plan to explore in future work.\n\nThe low scores on the 'Problem' label, as discussed in the limitation section, stem from the limited problematic state utterances in the SGD dataset. Including IRL annotations on other datasets would be able to mitigate this. Despite this limitation, we still demonstrate the effectiveness of using our IRL tagger for constructing pseudo labels on zero- and few-shot intent classification datasets that include more utterances with problem states (like Banking 77).\n\nThanks for the suggestion to conduct further ablation studies to understand the impact of IRL tagger performance on downstream tasks. Though we agree this may lead to further useful insights, we selected RoBERTa over other encoders (BERT-base) based on IRL tagger validation set performance. Our focus in this work is on comparison of pre-training with or without IRL tags.\n\nRTR2E: We acknowledge that different datasets may have different granularities for defining intents (e.g. one FileClaim intent vs. multiple fine-grained FileAutoClaim / FilePetClaim intents) – understanding the effect of this potential noise in contrastive learning specific to intent encoder pre-training is left for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1: The idea of assigning roles to words using IRL for creating pseudo intents to pre-train the encoder is innovative and interesting.\n\nS2: This paper has good approach presentation and detailed result analysis.\n\nS3: This paper is well-written and has clear structure to follow.\n\n", "reasons_to_reject": "W1: In section 3, the authors annotate the intent roles with 6 types for obtaining the IRL taggers. However, the definitions of intent roles like Action, Argument, Request, etc., may not capture all nuances across various domains or applications. Additionally, these roles may overlap in some contexts, potentially complicating the tagging process.\n\nW2: In Table 2, while many IRL labels have high precision, recall, and F1 scores, the 'Problem' label has significantly lower scores, suggesting that this particular aspect of the model might struggle in practical applications.\n\nW3: By formulating IRL as a sequence tagging problem, the complexity of the relationships between different labels in an utterance might be overlooked. This could lead to suboptimal intent representations.\n\nW4: The IRL tagger uses RoBERTa-base as its foundation. While RoBERTa is a powerful model, the results might differ if another model was used. This dependency also means that improvements in RoBERTa (or discovery of issues in it) could directly affect the IRL tagger's performance. Probably an ablation study is needed for showing the IRL tagger's performance on different models and how the different IRL tagger performance may affect the IC task performance.\n\nW5: For the intent-aware contrastive learning part, the method assumes that randomly sampled utterances that share the same gold intent names can be treated as positive pairs. This might introduce ambiguity, especially if different utterances with the same intent name have nuanced differences in meaning. Also, the method treats pairs that are not positive in a batch as negative pairs. This simplistic approach might introduce noisy negative samples, affecting CL performance.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the problem of intent classification (IC) in task-oriented dialogue systems. Specifically, the main challenge it seeks to tackle is that IC models often have difficulty generalizing when trained without a sufficient number of annotated examples for each user intent. To mitigate this issue, the paper introduces a novel pre-training method for text encoders. This method utilizes contrastive learning combined with pseudo intent labels to produce embeddings better tailored for IC tasks, thereby decreasing the necessity for extensive manual annotations. Experiments results of this paper show it has achieved the state-of-art intent classification performance on multiple benchmark datasets.\n\nC1: The paper introduces an innovative algorithm that generates pseudo intent names from utterances gathered across multiple dialogue datasets. This addresses the problem of obtaining ample annotations for a diverse intent set.\n\nC2: The paper proposes a pre-training strategy that leverages intent role labeling (IRL) to identify key phrases in utterances which are essential for deciphering intents. The identified phrases are then used to construct examples that pre-train a text encoder contrastively. The outcome is the creation of a model called PIE (Pre-trained Intent-aware Encoder), devised to synchronize the encodings of user utterances with their respective intent names.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YR4EDo3H3e", "reviewer": "Reviewer_Tq8c", "summary": "", "questions": "1. Do the authors consider comparing with other zero-shot intent-classification models not only using Prototypical Networks, such as [1]?\n2. The expression of \"pre-training\" is a little bit mis-leading, since the proposed model still uses paraphrase-mpnet-base-v2 encoder as a backbone model.\n3. Table 8 shows N-way settings. It is confused that \"Fine-tuning is done with 5-way due to memory constraints.\" in caption of Table 8.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s careful reading, helpful feedback and suggestions!\n\nRTR3A: On lack of literature review, in addition to adding some references suggested by the reviewers, we will further enrich our final manuscript by adding more recent related work on intent classification and contrastive learning.\n\nRTR3B: On effectiveness in zero-shot setting, Table 8 demonstrates that our method outperforms the SBERT_paraphrase baseline by 5.4% in the zero-shot setting without fine-tuning. These zero-shot settings become feasible by treating intent names as prototype class representations.\n\nRTR3C: While the quality of pseudo intents may be noisy due to their automatic generation without human verification, we demonstrate that incorporating these pseudo intents during pre-training enhances the text encoder's ability to align utterances and intent class names, thereby improving zero-shot intent classification. Details regarding the size of the pre-training dataset are provided in section 5.2.\n\nQ3A:  “Did you consider comparing with other zero-shot intent-classification models not only using Prototypical Networks, such as [1]?” Prototypical Networks are a widely used and powerful method for zero- and few-shot intent classification [2, 3]. We observed that [1] also employs prototypical networks for zero-shot intent classification, as stated 'The class label description ... can be treated as the class prototype.' To provide a clearer insight, we will include experiments with other approaches, such as a cross-encoder and episodic meta-learning baseline [3], in our final manuscript.\\\n[1] A Simple Meta-learning Paradigm for Zero-shot Intent Classification with Mixture Attention Mechanism. (Liu et al., SIGIR’22)\\\n[2] A neural few-shot text classification reality check (Dopierre et al., EACL’21)\\\n[3] The Devil is in the Details: On Models and Training Regimes for Few-Shot Intent Classification (Mesgar et al., 2022)\n\nQ3B, Q3C: Thanks for your suggestions on presentation. To prevent confusion, we will update the term to 'pre-finetuning' in the final manuscript. Initially, we called our approach 'pre-training' due to our model being trained before being fine-tuned on downstream tasks. Regarding the Table 8 caption, when we tried fine-tuning models with the N-way setting, we ran into out-of-memory issues on a single 12GB GPU. To work around this, we used a 5-way approach for fine-tuning and an N-way approach for evaluating. We'll explain this more clearly in the Implementation Details to avoid confusion in the final manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a new intent-aware contrastive learning objective to leverage gold and pseudo intents and improves intent classification performance compared to baselines.\n2. The motivation of using intent role labeling to generate psudo-intents are well stated and the problems in zero-shot intent classification are identified.", "reasons_to_reject": "1. Lack of review and investigations on intent classification and contrastive learning. \n2. The effectiveness of proposed model is not strong enough as it is not demonstrated under the zero-shot settings for intent-classification model. Instead, this paper only compares to intent-classification models using Prototypical Networks.\n3. the quality and amount of pre-training datasets need to be studied.", "questions_to_authors": "1. Do the authors consider comparing with other zero-shot intent-classification models not only using Prototypical Networks, such as [1]?\n2. The expression of \"pre-training\" is a little bit mis-leading, since the proposed model still uses paraphrase-mpnet-base-v2 encoder as a backbone model.\n3. Table 8 shows N-way settings. It is confused that \"Fine-tuning is done with 5-way due to memory constraints.\" in caption of Table 8.", "paper_topic_and_main_contributions": "This paper proposes an algorithm to generate pseudo intent names from utterances across various dialogue datasets to create a pre-training dataset. PIE (Pre-trained Intent-aware Encoder) are proposed by utilizing intent-aware contrastive learning on both gold and pseudo intent names. The model outperforms the state-of-the-art work on N-way zero- and one-shot settings on experiments on four Intent Classification datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Liu, Han, Siyang Zhao, Xiaotong Zhang, Feng Zhang, Junjie Sun, Hong Yu, and Xianchao Zhang. \"A Simple Meta-learning Paradigm for Zero-shot Intent Classification with Mixture Attention Mechanism.\" In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2047-2052. 2022."}}, {"review_id": "ZA4t3QydMF", "reviewer": "Reviewer_QwHo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful feedback.\n\nRTR4A: Thank you for providing two relevant papers. The main difference between theirs and ours is that while the papers use templates or descriptions to enhance intent representation when fine-tuning, we employ pseudo labels constructed by an IRL tagger when pre-training. We will include these papers as related works in our final manuscript to clarify the difference. Prototypical Networks are considered the state-of-the-art method for few-shot intent classification [1]. Therefore, our focus is on comparing our work with it.\\\n[1] The Devil is in the Details: On Models and Training Regimes for Few-Shot Intent Classification (Mesgar et al., 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A novel method demonstrating the improvement in representational ability of the LLMs used as encoders in task-oriented dialog task.\n", "reasons_to_reject": "The proposed method seems to be pretty close to already published ones [1, 2]. It would be nice to see the comparison.\nAnother important point is that the authors do not compare their work with state of the art in the field.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The main idea of the paper is presentation of a novel PIE method which enables an encoder to better represent an intent.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. Lamanov, D., Burnyshev, P., Artemova, K., Malykh, V., Bout, A. and Piontkovskaya, I., 2022, July. Template-based Approach to Zero-shot Intent Recognition. In Proceedings of the 15th International Conference on Natural Language Generation (pp. 15-28).\n2. Burnyshev, P., Bout, A., Malykh, V. and Piontkovskaya, I., 2021, September. InFoBERT: Zero-Shot Approach to Natural Language Understanding Using Contextualized Word Embedding. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021) (pp. 208-215)."}}]

Author rebuttal:

Meta-review: Overall, the reviewers are positive about the paper, highlighting its novelty, innovative approach, and strong performance in intent classification. However, Reviewer 2 raises some concerns about the intent role labeling, the use of the RoBERTa-base model, and the treatment of positive and negative pairs in the contrastive learning. Reviewer 3 also mentions the lack of investigation on intent classification and contrastive learning and suggests comparing with other zero-shot intent-classification models. Reviewer 4 appreciates the novelty of the method but suggests comparing it with similar published work and the state-of-the-art.

In terms of pros, the paper is commended for proposing a unique pre-training approach for text encoders, leveraging contrastive learning and intent pseudo-labels. The method is considered inspiring and has the potential for other NLP tasks. The motivation behind the methodology is clearly stated, and the experiments show significant improvement in performance over the state-of-the-art pre-trained sentence encoder. Additionally, the paper is well-written, has a clear structure, and provides sufficient support for its claims.

In terms of cons, the concerns raised by Reviewer 2 regarding intent role labeling, the RoBERTa-base model dependency, and the treatment of positive and negative pairs in contrastive learning should be addressed. Reviewer 3 suggests conducting further investigation and comparing with other zero-shot intent-classification models. Reviewer 4 suggests comparing the proposed method with similar published work and the state-of-the-art.
Decision: Accept-Main