QAID: Question Answering Inspired Few-shot Intent Detection
Authors: Asaf Yehudai
Matan Vetzler
Yosi Mass
Koren Lazar
Doron Cohen
Boaz Carmeli
Venue: ICLR 2023 poster (2023)
Keywords: Intent Detection
Question Answering
Contrastive Learning
Passage Retrieval

Abstract: Intent detection with semantically similar fine-grained intents is a challenging task. To address it, we reformulate intent detection as a question-answering retrieval task by treating utterances and intent names as questions and answers. To that end, we utilize a question-answering retrieval architecture and adopt a two stages training schema with batch contrastive loss. In the pre-training stage, we improve query representations through self-supervised training. Then, in the fine-tuning stage, we increase contextualized token-level similarity scores between queries and answers from the same intent. Our results on three few-shot intent detection benchmarks achieve state-of-the-art performance.

Published as a conference paper at ICLR 2023
QAID: QUESTION ANSWERING INSPIRED FEW-SHOT INTENT DETECTION
Asaf Yehudai
Matan Vetzler
Yosi Mass
Koren Lazar Doron Cohen Boaz Carmeli
IBM Israel Research Lab Hebrew University of Jerusalem
(first. last @iimm. com, {yosimass, doronc, boazc) il ibm. com com
ABSTRACT
Intent detection with semantically similar fine-grained intents is a challenging task. To address it, we reformulate intent detection as a question-answering re- trieval task by treating utterances and intent names as questions and answers. To that end, we utilize question-answeriig retrieval architecture and adopt a two stages training schema with batch contrastive loss. In the pre-training stage, we improve query representations through self-supervised training. Then, in the fine- tuning stage, increase contextualized token- similarity scores between queries and answers from the same intent. Our results on three few-shot intent detection benchmarks achieve state-of-the-arr performance.
INTRODUCTION
Intent detection (ID) is the task of classifying incoming user query one class from set of mutually-exclusive classes, a.k.a. intents (Wang et al., 2014: Schuurmans & Frasincar 2019 Liu al. 2019a). This ability is cornerstone for task-oriented dialogue systems as correctly identifying the user intent at the beginning of an interaction is crucial to its success. However, labeled data is required for training and manual annotation costly. This calls for sample efficient methods, gaining high accuracy with minimal amounts of labeled data.
Recent works tackling few-shot ID have relied on large-scale pre-trained language models, such as BERT (Devlin et al. 2018). These works leverage task-adaptive training and focus pre-training model on a large open-domain dialogue corpus and fine-tuning for ID classification (Mehri al,
2020
Wu et al.) 2020a Casanueva et al. 2020 Zhang et al. 2021a).
Alternative approaches tried to learn query representation based on quere-to-query matching (hence- forth, Match-QQ systems) (Zhang et al. 2020 Mass al. 2020, Mehri et al., 2021). Zhang
et al. 2020); Mass et al.] (2020) adopt pairwise-encodigg systems with cross-attention to deploy K-N -.Nearst-Neighbor (K-NN) (Fix & Hodges (989) classification schema where training queries
are fully utilized for both training and inference stages. Nevertheless, those methods' downside is the processing time combined with the difficulty of scaling large number of intents (Liu et al.
2021ch.
The need efficiently compare an incoming query to a large of possible answers resides the core of any question answering (QA) retrieval system (henceforth, Match-QA systems) (Karpukhin al. 2020). Recently, KKhttab Zaharia (220)) introduced ColBERT, which allows faster training and inference by replacing the cross-attention mechanism used by Match- BQch--QQua systems (Zhang et al., 2020 Mass et al. 2020, Nogueira & Cho 2019) with fast contextualized token-level similarity mechanism dubbed late interaction.
In this work, we present a Question Answering inspired Intent Detection system, named QAID. We start by formulating the ID) task as
retrieval task by treating the utterances and the intent names as queries and answers, respectively. This reformulation allows us to introduce valuable additional signal from the intent names. Then, we adapts the efficient architecture of Col- BERT while replacing its triplet function loss with batch contrastive loss which was proven to be more robust (Khosla et al. 2020) and performs well in various tasks (Gunel et al.) 2021 Gao al. 2021a), including ID classification (Zhang et al. 20211). In contrast to ColBERT which compares query a pair of positive and negative documents, also include queries positive examples,

---

Published as a conference paper at ICLR 2023
and compare the queries both to their answers and to other queries from the same intent. This allows QAID to represent similarly both queries and answers of the same intent. Therefore, our training method assumes the settings of both Match-QQ and Match-QA. In inference, QAID relies on the token-level similarity (late interaction) mechanism between incoming query and all intent names for its predictions (Khattab & Zaharia 2020).
Our contribution is thus threefold. (1) We show that few-shot intent detection can be successfully
handled by QA systems when letting the intent name play the role of the answer. (2) We show how intent detection architectures can benefit from recent advancements in supervised batch contrastive training and late-in interanntaattinn scores. (3) We report state-of-the-ar results on three few-shot intent detection benchmarks.
2
METHOD
Our method addresses the few-shot intent detection task, in which we have C defined intents and the task is classify an incoming user query, q. into one of the C' classes. In our formulation, upon getting a new user query q. we need to retrieve the most suited intent name. We set balanced K-shot
learning for each intent (Mehri al., 2020 Casanueva
al. 2020 Zhang et al., 2020). i.e., the
training data containing K examples per inten
In the following section, we describe the structure of our QAID framework and its training stages. First, in Section 2.] we elaborate on the different components of QAID. Then, Section 2.2 we present the two training stages: the self-supervised contrastive pre-training in [2.2.] and the super- vised batch contrastive fine-tuning in 2.2.2 Lastly, in Section 2.3 we briefly touch on our decision formulate ID question retrieval task.
2.1 REPRESENTATION LEARNING FRAMEWORK
The main components of our framework are:
Data Augmentation module, Aug(.). For each input query, q, we generate two random aug- mentations, Aug(ç), each of which represents different view of the input, 9. For augmentation we use the combination of two simple and intuitive 'corruption' techniques (Gao et al. 202 Wu al., 2020b] Liu et al., 2021b); (i) randomly masking tokens from q (Devlin et al. 2018); (ii) dropping a small subset of neurons and representation dimensions. Technique (i) done before passing the query to the encoder and technique (ii) done in the forward prop-
agation through the encoder model.
Encoder model, Enc(.). which maps a query q, consisting 9I.... 4m tokens, to Enc(g) E RmxDE where Dg is the embedding dimension. In our experiments, it is either 768 or 1024.
Projection layer, (-)j(.), a single linear layer that maps vectors of dimension Dg to vectors of dimension Dp 128, followed by normalization to the unit hypersphere.
Token-level score, Score(-. :), given two queries u (u,,... Um)) and v (ui..., vy), the relevance score of u regarding to v, denoted by Score(u, (u, v). is calculated by the late interac- tion between their bags of projected contextualized representations, ie =(w) = Proj(Enc(un Namely, the sum of the maximum token-wise cosine similarity of their projected representations (Khattab & Zaharia 2020). Equation showw the formulation of this score.
Score(u, v) Σ max z(u); =(v), vE[m]
fE[(]
2.2 TWO-STAGE CONTRASTIVE TRAINING
In both stages, given a batch of input samples Q = (q',... "))."), we first apply Aug followed by the encoding and projection layer, denoted by the 2(-) function as described in the last section, and so we have XQ = = 2(Aug(Q) E 2p where the 2n result of the two random augmentations we applied to each query. In the self-supervised training, each two augmented queries are the only
In the rest of the paper refer the intent examples queries and use intents and classes interchangeably
2

---

Published as a conference paper at ICLR 2023
positive examples for each other while all other queries negative. In the supervised training phase, we also run the same process with the same encoder on the corresponding intent names, R²nxD,, XA together with Xq forms
A = (a'....a") resulting in X = =()ug(A)) training batch of 4n instances. In this supervised setting, all queries and intent names of the same intent are positive to each other while all others are negative.
2.2.1 PRE- TRAINING
We use a task-adaptive pre-training stage to overcome the few-shot constraint, as done by most goal- oriented dialogue works (Mehri al., 2020 /hhan et al., 2020). Our pre-training aims to facilitate
domain adaptation by two early- ctrly-stage objectives: (I) Incorporate token-level domain knowledge into the model (Mehri et al. 2020 2021 (2) Adopt queries' representations to the dialog domain through data augmentation techniques and self-supervised contrastive training (Zhang et al. 202 2022a).
Practically, in training batch containing 2n augmented queries, let [2n] be the index of an arbitrary augmented query. Then in the self-supe contrastive learning stage, the loss takes the following form:
Σ log exp(Scoreld's .q ) LE[2m] EeeA(o) exp Score( ,"t /²)/T)
c =
(2)
where is called the anchor/pivot, J(t) is the index of the second augmented sample deriving from the same source sample (a.k.a positive), A(t) [[2m] \ f), A(t) \ J(t) are called the negative and
E R is scalar temperature parameter that controls the penalty to negative queries (see step (a) Figure
In addition to the self-supervise contrastive training, we pre-train also on the masked language modeling (MLM) task (Taylor 1953), to further adjust sentence-level representation to the domain of the data. Moreover, this improves lexical-level representation which essential for token-level similarity. Hence define m as the average cross-entropy loss over all masked tokens.
Masked language modeling as auxiliary loss:
The overall loss for the pre-training phase is PT Lsels + mim where À is controllable hyperparameter.
2.2.2 FINE-TUNING
At the fine-tuning stage, we only have a limited number of examples for each intent, and intents may be semantically similar, making the classification task difficult. To address the data scarcity, we utilize the explicit intent names as unique examples that serve as answers our QA retrieval framework. This is similar to recent works that leverage label names for zero-shot few-shot text classification (Meng et al. 2020, Basile et al., 2021). The intent name is usually assigned by a domain expert when designing goal-oriented dialogue system. As such, it provides semantic description of the intent that aims to discriminate it from other intents. Consequently, intent names may provide a signal that is otherwise difficult to extract from small set of queries.
Additionally, we utilize the pre-trained model designed for queries' representations and continue fine-tuning it on the few-shot examples with the permanent representation learning technique (Khosla et al.) (2020) of supervised batch contrastive training (see step (b) in Fig. [.. In that way, each time our model pulls together two queries from the same class it simultaneously also pulls their intent names closer together as well. Formally our supervised contrastive loss has the form:
eapl Scorell' (''' Σ log
uup Σ
PP(L) rE[4m]
PEP(() EmeAte) exp(Score(c /²)
In this formulation 9 may represent either an augmented query or its intent name, and since each instance has four views: augmented queries and two augmented intent names, have total
3

---

Published as a conference paper at ICLR 2023
Foiss
Faise
CONTRASTIVE
CONTRASTNE
125-0 768-2
128-0 7180
125D
1250
788-0
785-
Li
qi
df 0
a
a²
q
(a) Superviso
Supervise
(c) Indoxing
(d)
Shared Weeights/Actiiatioons
function
Scoring
Figure 1: Schematic illustration of method. Hat and bar represent augmentation, the subscript is a ruining index, superscript is the class index. (a) Self-supervised contrastive training with data aug- mentation enhance quires' representations. (b) Supervise contrastive fine-tuning learn query- to-query and query-to-and guery-to-answer similarity. (c) Indexing the answers into Faiss index. (d) Compare incoming query against all answers in the index and predict the most similar one.
of 4n samples. A(t) [[4m]] \ t); P(t) is the group of all samples that positive to a and is defined as all the augmented queries or intent names derived from the same label as
Besides the supervised contrastive loss we also train with classification loss, L and an MLM
loss, _ In total, the fine-tuning loss is L FT L + A colass Amim Amim_mm L where Actass and Xmim are controllable hyperparameters.
Indexing and Inference. After fine-tuning, index the embeddings of all candidate answers (a.k.a intent names) outputted from the encoder into Faiss (Johnson et al.) (021). a library for large-scale vector-similarity search. see Khattab & Zaharia (2020) for more details. Then, inference time, we compare the incoming query representation with all answers in the index and retrieve the most similar (see step (c) and (d) Fig. [
2.3
WHY TO FORMULATE THE TASK AS A QUESTION ANSWERING RETRIEVAL TASK?
For few-shot intent detection model to be practical, must be computationally efficient in both training and inference phases, while possibly handling large number of intents (Qi et al. 2020 Here we are leveraging the recent success of dense passage retrieval for question answering that was shown to perform well and efficiently (Karpukhin al., 2020 Khattab Zaharia 2020). Those systems can handle large number of candidate answers by using (1) a dual-encoder framework with light comparison instead of the computationally demanding pairwise- encoding,, (2) an indexation that enables large-scale fast retrieval.
EXPERIMENTAL SETUP
3.1 DATASETS
We experiment with three widely studied few-shot intent detection datasets which represent the in- tent detection (ID) part of DialoGLUE benchmark These datasets present challenging few-shot ID tasks with fine- graigre intents that are semantically similar. Moreover, they facilitate the comparison with recent state-of-the-us basclines.
Clinc150 (Larson al., 2019) contains 22,500 personal assistance queries classified into 150 intents across 10 domains.
Banking77 (Casanueva et al. 2020 contains 13,242 online banking queries classified into 77 fine- grained intents in single domain.
For readily http:///ww..a.toaoo////eeeeeeeen
4

---

Published as a conference paper at ICLR 2023
HWU64 (Liu et al. 2019b) contains 11,106 personal assistant queries classified into 64 intents across 21 different domains. Table reports the data splits of each dataset.
Dataset
#Train
#Vaild
#Test
#Intents
#Domains
CLINC150
15000
3000
4500
150
10
BANKING77
8622
1540
3080
77
1
HWU64
8954
1076
1076
64
21
Table 1: Data statistics of the three intent detection datasets from DialoGLUE.
3.2 MODELS AND BASELINES
We experiment with RoBERTa-base and RoBERTa-large encoders from the Hugging Face trans- formers library with ColBERT architecture In the self-supervised pre-training stage, we utilize the training and validation sets of the six ID datasets from Zhang et al. (202Tb); ; Mehri et al. (2020)
2021). For fair evaluation, we exclude the test sets from our pre- pre-training following the observation of Zhang et al. (2021b). We fix the number of embeddings per query at m 32 same as Khattab & Zaharia (2020). For the MLM loss, we follow the masking strategy of Devlin et al. (2018). We also use this masking strategy to augment our input queries in addition to the representation-leeel augmentation through the encoder build in 10% dropout. For the contrastive loss, we use the im-
plementation of Khosla et al. (2020 For the late interaction score, we normalize the score by the number of tokens in the summation. We train our encoder for 20 epochs with a batch size of 64, learning rate of 1e temperature parameter of 0.07 (same as Khosla al. (020 and A 0.1 as recommended in the literature.
For the fine-tuning stage, we train our models on 5- and 10-shot splits available within the public dataset distribution. We train our model for 10 epochs starting from the pre-trained model check- point. We set the batch size to 32, where queries and answers are encoded separately by the same encoder. Answers are truncated by the longest one in the batch same as Khattab & Zaharia (2020). We apply the same masking schema queries and answers as done in the pre-training. We set the temperature to 0.07. We also set Aclass and Amdm. to 0.1 and 0.05, respectively, making Laup the main summand in 2 FF,, We used those parameters they were recommended in the literature and shown to perform best in hyperparameter tuning. Following previous works, we ran the experiments with five different seeds and report the average accuracy.
For our Faiss Index implementation, we use IVFScalarQuantizer ("InVerted File with Scalar Quan- tizer:"). To improve memory efficiency, every embedding is represented by 8 bytes. In our work, we the full retrieval option that effectively retrieves all candidate answers, In cases of many intents, one can deploy fast retrieval of top candidate answers.
0
3.3 BASELINES
We start by categorizing the baselines by three main representation and prediction methodologies. A Classifier architecture learns both query representation and a linear classification head via cross- entropy loss. The classification head contains single vector per class that is implicitly trained to represent the class and enable prediction at inference time. Match-QQ architecture learns query representation via query-to-quent similarity matching it learns increase the similarity between embeddings of queries from the same class while simultaneously decreasing the similarity between queries from disparate classes. During inference, an input query matched against large collection of training queries, and the intent in which its queries are the most similar to the input is predicted. A Match-QU architecture learns to match queries to answers. The model learns to increase the similar- ity between queries and their corresponding answers while simultancously decreasing the similarity
hhaps://ittub..co//llbbbi htnp://ccooooloooooeeeeeeSSoooee
"htps://gituub.comm// http://hw....c///ccccocceeeeett/
5

---

Published as a conference paper at ICLR 2023
between queries and other answers. At inference time, incoming query is matched against all possible answers, and the most similar is predicted. In these terms, the pretraining of QAID is based on Match-QQ, its fine-tuning involves both Match-QQ and Match-QA, and its prediction is based on Match-QA, We experimented with prediction methods that include also Match-QQ, but found it preform slightly worse than inference with only Match-QA. We will elaborate more in Section $4.1
3.3.1 BASELINE MODELS
We compare our approach and results against strong baseline models reported in the literature. In the rest of the section, we discuss these models in more detail and align them with the paradigms mentioned above. Notably, some baseline models mix and match components across architectures.
Classifier: We fine-tune RoBERTa-base encoder with a feed-forward classification head our classifier baseline.
CoIBERT (Khattab & Zaharia 2020): Contextualized Late Interaction over BERT (CoIBERT) is state-of-the-an passage search and retrieval system. ColBERT provides a Match-QA baseline and is the basis for the QAID architecture. For training we use 20 triplets (query, pos answer, neg answer) for each query, with hard negatives, namely, we run the query against all answers using bm25 (Robertson & Zaragoza 2009) and select the negatives from the most similar answers.
USE+CONVERT (Casanueva et al. 2020y: USE Yang et al. 2019) is large multilingual dual- encoder model pre-trained in 16 languages. CONVERT (Casanueva et al.) (020) an intent detection model with dual encoders that are pre-trained on 654 million (input, response) pairs from Reddit.
CONVEBERT (Mehri al. 2020); a BERT-base model which has been trained on large open- domain dialogue corpus. CONVEBERT achieved improvements over vanilla BERT architecture and state-of-ther offthh-art results on a few task-oriented dialogue tasks.
(Mehri et al., 2021]: a CONVEBERT-bare model trained to im- prove similarity matching of training examples, i.c., Match-QU -Q. Additionally, the model trains with observers for transformer attention and conducts task aask-adaptive self-supervised learning with mask language modeling (MLM) on the intent detection datasets. Combined represents the best MLM+EXanple++bserveen setting in the referenced paper.
CONVEBERT+Combingg
DNNC (Zhang et al., 2020): Discriminative Nearest Neighbor Classification (DNNC) model is trained find the best-matched example from the training set through similarity matching (Match-QQ). The model conducts data augmentation during training and boosts performance by pre-training on three natural language inference tasks.
CPFT (Zhang et al. 2021b): Contrastive Pre-training and Fine-Tuning (CPFT) is a two-stage intent-detection architecture. During the first stage, the model learns with a self-supervised con- trastive loss on a large set of unlabeled queries. In the second stage, the model learns with su- pervised contrastive loss to pull together query representation from the same intent (Match-QO) The inference is done via a classification head that added and trained during the second stage.
4 RESULTS
Table 2 lists the results on the three datasets described in Section 3.1 QAID with RoBERTa-base achieved the best results across all datasets and shots. Notably, increasing the model size from RoBERTa-base to RoBERTa-large resulted in additional significant improvement across all datasets. For 5-shot, QAID with RoBERTa-base improves over CPFT, which achieved the best results re- ported so far, by more than 4 points on the BANKING77 dataset which is translated to 30.64% in error rate reduction (ERR). Similarly, QAID achieves ERR of 8.5% and 18. 18.9 over CPFT for the CLINC150 and HWU64 respectively. We attribute our improvement to three key differences be- tween our method and CPFT. (I) Our problem formulation kept the class representation the same during training and inference. In other words, we didn't train classification layer for inference. (2) Incorporating answers as data points contributes an additional discriminating signal. (3) The token-level late interaction score is shown perform better our ablation experiments demon- strate in Section 4.J. Moreover, our standard deviations (std) are consistently lower than those of DNNC and CPFT with the highest std of 0.15 and average std of 0.07. We believe the reason for

---

Published as a conference paper at ICLR 2023
CLINC150
BANKING77
HWU66
Model
5
10
5
10
5
10
Classifier (RoBFRTa-base)
87.68
91.22
74.46
83.79
73.52
82.62
ColBERT
82.03
88.10
72.71
79.25
74.98
81.78
USE+CONVERT (Casanueva et al. 2020
90.49
93.26
77.75
85.19
80.01
85.83
CONVBERT MMehn et al.
2020)
92.10
83.63
83.77
CONVBERT Combined
Mehri et al. 2021
93.97
85.95
86.28
DNNC
(Zhang al.
2020
91.02
93.76
80.40
86.71
80.46
84.72
CPFT
Zhang et al.
2021b
92.34
94.18
80.86
87.20
82.03
87.13
QAID
(RoBERTa-bare base)
93.41
94.64
85.25
88.83
85.52
87.98
QAID largeRTa-large)
94.95
95.71
87.30
89.41
87.82
90.42
Table 2: Accuracy results on three ID datasets in 5-shot and 10-shot settings. Baseline results are from the original papers except for RoBERTa Classifier and ColBERT.
the low std in the results is the combination of batch contrastive loss with data augmentation and the fine-graid one-grained inte-interaction similarity score. Accordingly, our improvements are significant ac- cording to an unpaired t-test with a p-value of 1e le= An additional advantage of our method is its efficiency. QAID pre-training run-time takes about two hours and it has to run only once for all of our targets. Our fine-tum thee-tuning takes only ten minutes on one NVIDIA V100 GPU, compared to three hours of fine-tuning of DNNC. Another important aspect of results is the effect of scaling from NOBERTa-base to RoBERTa-large, which resulted in significant improvements in both 5 and 10- shot scenarios across all datasets, aligned with results showing larger models generalize better from small data (Bandel et al. 2022). Moreover, in some cases scaling the model was more beneficial than additional examples. Namely, RoBERTa-large in 5-shot surpasses RoBERTa-base in 10-shot.
4.1 ABLATION TESTS
In this section, we describe several ablation studies demonstrating the importance of our method components and the main factors that contribute our improvement over ColBERT
We present our ablation results in Table 3 We start by analyzing the improving effect of the pre- training (PT) and batch contrastive training on COlBERT. We can see that both stages boost the performances considerably across all settings. It is noticeable that the pre-training (row ColBERT with PT) improves more in the 5-shot than the 10-shot setting with deltas of 4.59 and 2.00 points on average, respectively. This result is consistent with the observation that a model with better query representation essential when only few examples are available (Choshen et al., 2022). Batch contrastive training (row ColBERT with batch contrastive) improves performance in most settings, with an average improvement of 5.74 points over ColBERT. We attribute this improvement major enhancements that batch contrastive training introduces. The first is the shift from a model that learns only Match-QA to a model that learns both Match-QA and Match-QQ. The second is the improved technique of batch contrastive loss over triplet loss that allow process many positive and negative examples at once and intrinsic ability to perform hard positive/negative mining (Khosla al. 2020). We note that this change has minor effect on the training time relies on representations calculated in the batch and has no effect on the inference time.
In addition, we study some modifications of QAID to understand their effect. To further investigate the effect of batch contrastive we train QAID with N-pairs loss (row QAID - N-pairs loss) with in-batch negatives, a widely used loss in retrieval models, e.g. DPR and RocketQA (Karpukhin et al., 2020) Qu et al. (02T). In this setting, each query has one positive example, which in our case is the intent name, and multiple irrelevant (negative) examples, either different queries or intent names. This method differs from our supervised batch contrastive loss which allows many positive examples. Our results show that replacing QAID supervised batch contrastive loss with N-pairs loss leads to a decrease of more than I point average. These results support our claim that retrieval models can benefit from adopting supervised batch contrastive loss. When we conducted fine-tuning training with and without auxiliary tasks (MLM and classification losses), we found
7

---

Published as a conference paper at ICLR 2023
that the auxiliary tasks increased QAID accuracy by 0.43 average. Interestingly, the increase was more pronounced as the dataset contained fewer domains, 0.72, 0.48, and 0.10 of average improvement on Banking77, Clinc150,, and HW respectively. We also examine the performance of our method without pre-training. row QAID w/o PT. Results indicate that our method achieves average improvement of 3.53 points compared CPFT without pre-training (CPFT w/o PT) in Table 3 This result emphasizes the superiority of our proposed method as a fine-tuning method. Additionally, better understand the role of the data augmentation module in our training, we conduct an experiment where we did not apply the data augmentation module. Results, QAID w/o data augmentation, show that by augmenting the data we improve the results by about a third of point average. In the 5-shot setting, the improvement is about 0.75 points average, and in the 10-shot setting, the effect is inconsistent. Those results can indicate that data augmentation is more beneficial where less data is available.
We experiment with replacing the similarity score in QAID with the cosine similarity of the CLS token (row QAID Cosine Similarity) instead of the token-level late interaction (Khattab & Za- haria 2020) (row QAID). We can see that using the token-level late interaction achieves higher results across all datasets and shots. We ascribe this improvement the fine fine-grained nature of the late-interaction score that enables detailed token-level comparison. This score presents an efficient alternative to the costly cross- cross-attention scoring that most Match-QQ methods use.
Finally, we discuss our inference method. We experiment with indexing and predicting based on both queries and answers, i.e., using Match-QQ and Match-QA in the inference stage we do in training. Inference based only on Match-QA achieve slightly better (0.07) results on average, with an average improvement of 0.18 and 0.17 on Banking77 and Clinc150, respectively, and an average decrease of 0.14 on HWU64. These results indicate that our training method achieves answer representation that reflects the distribution of both training queries and answers. Therefore, allowing a more efficient inference that relies only on the answers' representations.
CLINC150
BANKING77
HWU64
Model
10
10
5
10
ColBERT
82.03
88.10
72.71
79.25
74.98
81.78
ColBERT with PT
89.31
90.85
75.73
80.42
81.20
84.88
ColBERT with batch contrastive
89.92
93.17
81.41
86.68
80.36
85.54
CPFT w/o PT (Zhang et al. 2021b)
88.19
91.55
76.75
84.83
76.02
82.96
QAID N-pairs loss
92.38
93.70
84.47
87.25
84.19
87.13
QAID w/o PT
90.52
93.26
81.61
86.20
80.45
86.48
QAID w/o data augmentation
93.21
94.64
84.34
88.46
84.37
88.46
QAID cosine similarity
92.73
93.71
84.11
87.24
84.04
87.70
QAID
93.41
94.64
85.25
88.83
85.52
87.98
Table 3: Accuracy results of our ablation experiments.
RELATED WORK
5.1 FEW-SHOT INTENT DETECTION
Task adaptive pre- pre-training is common strategy to face the data scarcity problem in few-shot intent detection classification. Predominant approach for task adaptive pre-training models leverages self- supervised mask language modeling training on large dialogues datasets (a few hundred million dialogues) and on the domain itself to tackle few-shot intent detection (Casanueva et al.) 2020) Mehri et al. 2020 2021). [Shnarch et al. (2022) showed that unsupervised clustering helps better than MLM pre-training. DNNC (Zhang et al. 2020 pre-trains their system annotated pairs from natural language inference (NLI) leveraging BERT (Devlin et al. 2018) pairwise encoding. Then they model intent detection as a kNN problem with k = where the pre-trained NLI model leams to predict the similarity score for pair queries. However, this model is computationally expensive as it fully utilized training examples in both training and inference.
8

---

Published as a conference paper at ICLR 2023
The CPFT work, suggested by Zhang al. (2021b), shares some similarities with approach. when the two main ones are: the two-stage training process and the use of batch contrastive loss. However, have several key differences where extend upon their method. Those differences make our method more effective and efficient, especially when the task involves a large number of intents. Firstly, reformulate the few-shot ID classification task retrieval task. To that end, we adopt an efficient Dual-Encoder-basee retrieval architecture- ColBERT (Khattab & Zaharia 22020), and late-interaction similarity score. Moreover, we treat the intent names as the answers we wish to retrieve. Secondly, we adjust the training method to learn both Match-QQ and Match-QA simi- larities. Finally, we adopt a retrieval- saaseeval-based inference based on the similarity between the incoming query and the intent names, therefore we are not required to train additional classification head.
Zhang al. 2022b) design two stage training with batch contrastive loss and add explicit regular- ization loss directing the feature space towards isotropy. They report high S-way few-shot results on the same benchmarks we use. Nevertheless, when evaluating their model accuracy the results are much lower than ours (about 16% and 9% lower on Banking77 and HWU64 respectively).
5.2 INTENT NAME
The idea of exploiting class names was proposed in the setting of zero and few-shot classification by few past works (Meng al. 2020 Yin al.) 2019 Basile al. (021)) Yin et al. (2019)
propose to formulate text classification tasks as a textual entailment problem (Dagan et al. 2005).
This mapping enables using a model trained on natural language inference (NLJ) as zero-sha shot text classifier for a wide variety of unseen downstream tasks (Gera et al. (022). Zhong et al. (2021) map the classification tasks to a question-ant -anwwerng format, where each class is formulated as a question and given as a prompt, and the decoder probabilities of the "Yes" and "No" tokens correspond to positive or negative prediction of the class.
In our work, cast the classification problem to the task of question answering retrieval and treat a much larger number of classes than these works tackle, which is usually up to twenty.
5.3 BATCH CONTRASTIVE LEARNING
Batch contrastive training was shown to achieve improved representation and perform better than contrastive losses such as triplet, max-margin, and the N-pairs loss (Khosla et al., 2020). Gunel al. (202T); Gao et al. al. (00Tbb suggest incorporating batch contrastive learning to train the encoder in natural language processing tasks. Gao et al.] (2021b) designed a simple contrastive learning framework through dropout augmentation. They trained on NLI data to achieve state-of-the-art results on unsupervised and full-shot supervised semantic textual similarity (STS) tasks (Agirre) jet al. 2012: 2015 2016). Liu et al. (2011a) suggest MirTorBERT, a self-supervised framework with two types of random data augmentation: randomly erase mask parts of the texts during tokenization, and representation level augmentation through built-in encoder dropout. We differ from those works we target the few-shot intent detection task. Moreover, adjust the late interaction score from Khattab & Zaharia (2020) to achieve cross-attention-like similarity scores. We also showed that class names can serve as an additional augmentation that can be the base for inference prediction.
6 CONCLUSIONS
In this paper, we present QAID, Question Answering inspired Intent Detection system, that mod- els the few-shot ID classification a question-answe answering retrieval task, where utterances serve as questions and intent names as answers. We train QAID with a two-stage training schema with batch contrastive loss. Results show that replacing ColBERT triplet loss with batch contrastive loss leads to a considerable improvement. We assert that a contributing factor to this effect is the shift to learning Match-QQ and Match-QA representations. We leave for further research to investigate this effect on retrieval tasks. Moreover, our results show that incorporating token-level similarity scores in contrastive loss outperforms the common cosine similarity score without a notable increase in training time. We encourage future research to utilize this type of contrastive loss in other tasks and investigate its effect. Finally, our results on three few-shot ID benchmarks show that QAID achieves of-thoo-the-rrt performance.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We thank Leshem Choshen and Ariel Gera for their helpful feedback as we pursued this research.
REFERENCES
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. SemEval-2012 task A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computa- tional Semantics - Volume I: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop Semantic Evaluation (SemEval 2012), pp. 385-393, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics. URL https: TEEDs://acLaanhhoooog..oooo 0wtp://////11111111111
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Wei- wei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), ,, pp. 253-263, Denver, Colorado, June June 2015. Association for Computational Lin- http:h/ppp///cllaanooooooo.oo///
guistics. doi: 1188653/1//1152000
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, Ger- man Rigau, and Janyce Wiebe. 2e06 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-22) pp. 497-511, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v1/$16- 1081. URL httph:tts//// hattps://aclantholl.. aolottooooooo
htg/s16-108)
Elron Bandel, Yoav Goldberg, and Yanai Elazar. Lexical generalization improves with larger models and longer training. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 4398-4410, Abu Dhabi, United Arab Emirates, December 2022, Association for Computa- tional Linguisties. URL URLhttps:://an
anthology. org/2022.22222 fwwningsnnnnineeeee eerinnggs-mmnllp 323
Angelo Basile, Guillermo Pérez-Torró, and Mare Franco-Salvador. Probabilistic ensembles of zero- and few-shot learning models for emotion classification. In RANLP, 2021.
Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders. arXiv preprint arXiv:2003.04807, 2020.
Leshem Choshen, Elad Venezian, Shachar Don-Yehia, Noam Slonim, and Yoav Katz. Where to start? analyzing the potential value of intermediate models. arXiv preprint arXiv:2211.00107. 2022.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Springer, 2005.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 11X.v:1810.04805, 2018.
Evelyn Fix and Joseph Lawson Hodges. Discriminatory analysis. nonparametric discrimination: Consistency properties. International Statistical Review/Revue Internationale de Statistique, 57 (3):238-22) 247, 1989.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence
embeddings. ArXiv, abs/2104.08822 2021a.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint 2 2021b.
Ariel Gera, Alon Halfon, Eyal Shnarch, Yotam Perlitz, Liat Ein-Dor, and Noam Slonim. Zero-shot text classification with self-training. In Proceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pp. 1107-1119, Abu Dhabi. United Arab Emirates, Decem- ber 2022. Association for Computational Linguistics. URL https: / hology. 2002. emn 1p-main 1
10

---

Published as a conference paper at ICLR 2023
Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. Supervised contrastive learning for pre-trained language model tne-ttning. ArXiv, abs/2011.01433, 2021.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535-5444 2021. doi: 10.109//BDD7AA221992225577
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.00 04000, 2020.
Omar Khattab and Matei Zaharia. Colhert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 39-48, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:18664 18611188667, 2020.
Stefan Larson, Anish Mahendran, Joseph J Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, al. An evaluation dataset for intent classification and out-of-scope prediction. arXiv preprint arXiv: 2019.
Fangyu Liu, Ivan Vulic, Anna Korhonen, and Nigel Collier. Fast, effective, and self-supervised: Transforming masked language models into universal lexical and sentence encoders. In EMNLP, 2021a.
Fangyu Liu, Ivan Vulic, Anna Korhonen, and Nigel Collier. Fast, effective, and self-supervised: Transforming masked language models into universal lexical and sentence encoders. In EMNLP, 2021b.
Jiao Liu, Yanling Li, and Min Lin. Review of intent detection methods in the human- human-machine dialogue system. In Journal of Physics: Conference Series, volume 1267, pp. 012059. IOP Pub- lishing, 2019a.
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. Benchmarking nat- ural language understanding services for building conversational agents. arXiv preprint arXiv: arXiv:1903.055: 1903.0556.,,, 2019b.
Zhiwei Liu, Ziwei Fan, Yu Wang, and Philip S. Yu. Augmenting sequential recommendation with pseudo-prior items via reversely pre-training transformer. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021c
Yosi Mass, Boaz Carmeli, Haggai Roitman, and David Konopnicki. Unsupervised faq retrieval with question generation and bert. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 807-812, 2020.
Shikib Mehri, Mihail Eric, and Dilek Hakkani-Tur. Dialoglue: natural language understanding benchmark for taik-orie dialogue. arXiv preprint arXiv:2009. 13550, 2020.
Shikib Mehri, Mihail Eric, and Dilek Z. Hakkani-Tür. Example-driven intent prediction with ob- servers. In NAACL, 2021.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. Text classification using label names only: A language model self-training approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9006 9017, Online, November 2020. Association for Computational Linguistics. doi: 10. 18653/v1/
2020.emnlp-main.722 URL htt.s: //acl antho conthology.oo orwp:/200.0000000 emm p-ma i 724
Rodrigo Nogueira and Kyunghyun Cho. 1rXiv:/901.044 2019.
Passage re-ranking with bert.
arXiv preprint
11

---

Published as a conference paper at ICLR 2023
Haode Qi, Lin Pan, Atin Sood, Abhishek Shah, Ladislav Kune, Mo Yu, and Saloni Potdar. Bench- marking commercial intent detection services with practice-driven evaluations. arXiv preprint arXiv:2012.03929, 2020.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 202| Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5835-5847, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021 1221.naacl-meg mall-mmain46666 URL. https:::./............ /ac antho logy. org/2021 www.cancaaa.m....mm ma n 466
Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends Information Retrieval, 3(4):333-389, 2009. ISSN 1554-0669. doi:
10.1561/15000000099 URL http:://xx..ae doi. 10 wtt./////11111111111111000
Jetze Schuurmans and Flavius Frasincar. Intent classification for dialogue utterances. IEEE Intelli- gent Systems, :22 08, 2019.
Eyal Shnarch, Ariel Gera, Alon Halfon, Lena Dankin, Leshem Choshen, Ranit Aharonov, and Noam Slonim. Cluster & tune: Boost cold start performance in text classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 7639-7653, Dublin, Ireland, May 2022. Association for Computational Linguis- URL
tics. doi: 36653//120222.alllonng/ acl-longe 526
Wilson Taylor. "cloze procedure": A new tool for measuring readability. Journalism quarterly, 30(4))415-433, 1953.
Fang Wang, Zhongyuan Wang, Zhoujun Li, and Ji-Rong Wen. Concept-based short text classifica- tion and ranking. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, pp. 1069-1078, 2014.
Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. TOD-BERT: Pre-trained natural language understanding for taik-oriented dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 917-929, Online, Novem- ber 2020a. Association for Computational Linguistics. doi: 10.18653/v /22p2/000.0mmmmmi.... URL httthtts//aaaaaathooooae http:///caaaacolooooor/////////e 2020. wwnle-mailai... 66
Zhuofeng Wu, Sinong Wang. Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. Clear: Contrastive learning for sentence representation. ArXiv, abs/2012.154666 2020b.
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, et Multilingual universal sentence encoder for semantic retrieval. arXiv preprint arXiv:1907.00400 2019.
Wenpeng Yin, Jamaal Hay, and Dan Roth. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat- ural Language Processing (EMNLL-IJCNLPP, pp. 3914-3923, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1DDI--1100 1404. URL https: 7/aclanthology: http:/thhooooo...oo//11/11-11
Haode Zhang, Yuwei Zhang, Li-Ming Zhan, Jiaxin Chen, Guangyuan Shi, Xiao-Ming Wu, and
Albert Lam. Effectiveness of pre-training for few-shot intent classification. arXiv preprint a7Xiv:2109.05782, 2021a.
Haode Zhang, Haowen Liang, Yuwei Zhang, Li-Ming Zhan, Xiao-Ming Wu, Xiaolei Lu, and Albert Y. S. Lam. Fine-tuning pre-trained language models for few-shot intent detection: Supervised pre-training and isotropization. In NAACL, 2022a.
Haode Zhang, Haowen Liang, Yuwei Zhang, Liming Zhan, Xiao-Ming Wu. Xiaolei Lu, and Albert Y. S. Lam. Fine-tuning pre-trained language models for few-shot intent detection: Supervised pre-training and isotropization, 2022b. URI.https:: wtt.tttss:///xiiiiii wrg.
httgg//bbb... abs 220..220502000 07208
12

---

Published as a conference paper at ICLR 2023
Jian-Guo Zhang, Kazuma Hashimoto, Wenhao Liu, Chien-Sheng Wu, Yao Wan, Philip S Yu, Richard Socher, and Caiming Xiong. Discriminative nearest neighbor few-shot intent detection by transferring natural language inference. arXiv preprint 0rXiv22010.13009 2020.
Jianguo Zhang, Trung Bui, Seunghyun Yoon, Xiang Chen, Zhiwei Liu, Congying Xia, Quan Hung Tran, Walter Chang, and Philip Yu. Few-shot intent derection via contrastive pre-training and fine-tuning. arXiv preprint vrXiv22109006444 2021b.
Ruiqi Zhong, Kristy Lee, Zheng Zhang. and Dan Klein. Adapting language models for zero- shot learning by meta-tuning on dataset and prompt collections. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2021, pp. 2856-2878, Punta Cana, Dominican Re- public, November 2021. Association for Computational Linguisties. doi: 10. 10.18653/v122011. 244... URL https: / hwtp://aaaaaooocoooo////////////
244
13


Reviews:
[{"review_id": "pDkRT_lJwzy", "reviewer": "Reviewer_ZHT4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors reformulate the problem of intent detection as a question-answering task by treating utterances and labels as questions and answers. A two stage training schema is employed by utilizing question-answering retrieval architecture and batch contrastive loss. The first of the training stages is for learning better query representations, while the second stage is for optimizing the contextualized token-level similarity scores between queries and answers from the same intent. State-of-the-art results are reported on three few-shot intent detection benchmarks.\n", "summary_of_the_review": "Overall, this is an interesting paper and addresses the challenging intent detection task in a question-answering setting. \n", "strength_and_weaknesses": "Strengths\n\nS1: It is an interesting idea to treat ID problem as a question-answering task.\n\nS2: Utilization of Token level domain knowledge in a self-supervised fashion along with queries adaptation to dialogue domain is interesting.\n\nS3: Masked language modeling has also been employed as auxiliary loss.\n\nS4: Intent names have been employed for generating extra supervision signals.\n\nS5: There are detailed experiments in this work, with an interesting insight that ColBERT triplet loss with batch contrastive loss leads to a considerable improvement.\n\nS6: The paper is well written and easy to follow.\n\n\nWeaknesses\nW1: Why a balanced dataset is the requirement for this work. If the dataset is imbalanced, how will the proposed method behave?\n\nW2: Typos: In abstract: “self supervise manner.”\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZQPM7xongN", "reviewer": "Reviewer_uX1q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes QAID, Question Answering inspired Intent Detection system, which models the intention detection classification as a question-answering task. The model uses two stages of training: a pretraining for better query representation and finetuning on few-shot labels of query and answers (name of intents). Certain choices of model, such as token-level similarity, batched contrastive learning, and inference with answers only, are verified by detailed ablation studies. The proposed method achieves SOTA  performance on three intention detection dataset from DialoGLUE.", "summary_of_the_review": "The paper is inspired by the development of ColBERT in QA task and propose a variant of the model for the task of intent detection. The proposed model makes several adaptations including batch contrastive loss and  signal from the intent names. The proposed method achieves SOTA performance on three few-shot ID benchmarks. And the ablation study proves the necessity of various components in the system. ", "strength_and_weaknesses": "Strength: \n\n- The paper uses and adapts techniques from QA and answer retrieval to the task of ID. The method is effective and efficient on three benchmark datasets when compared with various baselines.\n- The detailed ablation study verifies the necessity of stages and component of proposed framework.\n- The paper is well-written and easy to follow.\n\nWeakness:\n- Given the number of intents being only a few hundreds at most, is Faiss really necessary for inference?\n- n has two meanings in section 2.1: one for batch size and another for number of tokens.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NX_ZWP3fkMZ", "reviewer": "Reviewer_rUR4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper reformulates intent detection as a retrieval task, by treating utterances and intent names as questions and answers respectively. The authors leverage a dual-encoder based retrieval architecture for few-shot intent detection, with late-interaction scores (as used in ColBERT) and batch contrastive training. ", "summary_of_the_review": "Overall the paper shows good (and statistically significant) improvements over existing methods for few-shot intent detection. However, the novelty in the paper is limited as it mainly involves combining existing techniques in a retrieval-based formulation (such as batch contrastive loss and late-interaction scores) with methodologies for few-shot intent detection proposed in CPFT (such as self-supervised contrastive pre-training and fine-tuning)", "strength_and_weaknesses": "Strengths:\n\nFormulating as a retrieval task helps make use of current SOTA techniques in retrieval such as late interaction scores and batch contrastive loss. The idea of using a retrieval-based dual encoder for this task is interesting from an efficiency perspective too, since the intent names (which are the answers) can be pre-computed during inference. This overcomes the high latency of cross-attention based approaches for which the processing time scales with the number of intents.\n\nWeaknesses:\n\n1) It's not entirely clear why augmentation is needed and what is the motivation for / benefit from using the data augmentation module\n\n2) Since CPFT seems to be most similar to this paper, the authors need to highlight better what the main differences are with CPFT. From reading the paper, the impression I get is that the authors just incorporate the few-shot intent detection methodologies used in CPFT (such as self-supervised contrastive pre-training and fine-tuning) into a dual-encoder based retrieval formulation (by using ColBERT)\n\n3) It is not surprising to see benefits from using late-interaction scores and batch contrastive training (which is one of proposed contributions). ColBERT/ColBERT-V2, which are based on late-interaction scores, have already been shown to outperform cosine similarity based matching in dual encoder models. Further, batch contrastive loss is already a popularly used training technique for dual-encoder based retrieval models (such as DPR, RocketQA)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4ABXukfMw8C", "reviewer": "Reviewer_73BF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a Question-Answering inspired Intent Detection System (QAID). The paper treats the intent identification task as a Question-Answering retrieval task by treating the utterances and the intent names as queries and answers respectively. QAID adapts ColBERT architecture from prior work and replaces the loss function with batch contrastive loss. The novelty of this work is in converting intent id task to Question-Answering form and using the intent names as answer, and using supervised batch contrastive training for finetuning LM on this task. ", "summary_of_the_review": "Overall, this paper builds on top of prior work (especially ColBERT and CPFT) but uses intent names for retrieval. The paper shows the benefit of combining these techniques and presents SOTA results and shows good ablations. ", "strength_and_weaknesses": "Strengths: \n\n1. The paper reports SOTA results in three benchmark intent identification tasks for both 5-shot and 10-shot setting. \n\n2. The paper shows ablations to support the claims on supervised batch contrastive training and late interaction scores. \n\n3. The paper is well written and easy to understand \n\n \nWeakness: \n\n1. The ablations can be improved. The paper can show ablation to understand the component of each component and the impact it has on the intent id task. The missing ablations are: QAID – batch contrastive,  QAID – intent names (not sure if this is possible) and QAID – data augmentation. But it would be good to show results by removing one component at a time. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster