RSVP: Customer Intent Detection via Agent Response Contrastive and Generative Pre-Training
Authors: Yu-Chien Tang
Wei-Yao Wang
An-Zi Yen
Wen-Chih Peng
Venue: EMNLP 2023 Findings (2023)
Keywords: Intent Detection
Task Adaptive Fine-Tuning
Contrastive Learning
Question Answering
TL;DR: We present RSVP, a novel two-stage framework integrating agent reponses to address the challenging intent detection problem in real-world task-oriented dialogue sysemts.

Abstract: The dialogue systems in customer services have been developed with neural models to provide users with precise answers and round-the-clock support in task-oriented conversations by detecting customer intents based on their utterances. Existing intent detection approaches have highly relied on adaptively pre-training language models with large-scale datasets, yet the predominant cost of data collection may hinder their superiority. In addition, they neglect the information within the conversational responses of the agents, which have a lower collection cost, but are significant to customer intent as agents must tailor their replies based on the customers' intent. In this paper, we propose RSVP, a self-supervised framework dedicated to task-oriented dialogues, which utilizes agent responses for pre-training in a two-stage manner. Specifically, we introduce two pre-training tasks to incorporate the relations of utterance-response pairs: 1) Response Retrieval by selecting a correct response from a batch of candidates, and 2) Response Generation by mimicking agents to generate the response to a given utterance. Our benchmark results for two real-world customer service datasets show that RSVP significantly outperforms the state-of-the-art baselines by 4.95% for accuracy, 3.4% for MRR@3, and 2.75% for MRR@5 on average. Extensive case studies are investigated to show the validity of incorporating agent responses into the pre-training stage.

RSVP: Customer Intent Detection via Agent Response Contrastive and Generative Pre-Training
Yu-Chien Tang, Wei- Yao Wang, An-Zi Yen, Wen-Chih Peng Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan Commytyc.cs1@@eyee csw.encc........mooo edu. tw, sf1638 cs05@nctu. edu. tw, azyenênycu. edu. tw, wcpengcsênycu. edu. tw
Abstract
like manipulation approach product...
The dialogue systems in customer services have been developed with neural models to provide users with precise answers and cound-the-clee support task-oriented conversations by de- tecting customer intents based on their utter- ances. Existing intent detection approaches have highly relied on adaptively pre-training language models with large-scale datasets, yet the predominant cost of data collection may hinder their superiority. In addition, they glect the information within the conversational responses agents, which have lower col- lection cost, but are significant customer in- tent as agents must tailor their replies based on the customers' intent. In this paper, we propose RSVP, self-supervised framework dedicated to task-oriented dialogues, which utilizes agent responses for pre-training two-stage man- ner. Specifically, we introduce two pre-training tasks to incorporate the relations utterance- response pairs: 1) Response Retrieval by se- lecting a correct response from batch of can- didates, and 2) Response Generation by mim- icking agents generate the response given utterance. Our benchmark results for two real-world customer service datasets show that RSVP significantly outperforms the state-of- the-art baselines by 4.95% for accuracy, 3.4% for MRR 3, and 2.75% for MRR @5 av- erage. Extensive case studies are investigated show validity incorporating agent re- sponses into the pre-training !
Intent: ProductMeripulsttio
First need press the button the device
to tum it on...
Figure 1: An example of the customer utterance (blue), customer intent (orange), and the agent re- sponse (red).
for high-quality shopping experiences, providing precise answers efficiently through customer ser- vices has become increasingly vital for companies. Recently, companies, especially e-commerce com- panies, are developing their own Al models (Tao et al., 2018; Chen et al.. 2019; Song et al.. 2022) to automate repetitive tasks and reduce the bur- den agents. The common approach is to frame this problem intent detection task (Liu et al., 2019a), which involves classifying customer's utterance into one of the pre-defined intents.
Previous methods tackling intent detection tasks with deep neural models (E et al.. 2019; Zhang et al., 2019) able learn semantic represen- tation from customer utterances. However, these methods rely on large-scale high-quality labeled data requiring expensive and time-consuming an- notations (Abro et al., 2022). Recent studies on intent detection primarily focus on pre-training lan- guage models with public dialogue datasets and fine-tuning them for downstream intent detection tasks (Casanueva et al., 2020; Mchri al., 2020; Mehri and Eric, 2021; Zhang et al., 2021a). These methods have proven to be effective, but they heav- ily depend on the availability of additional public pre-training datasets, where large amounts of un- labeled data, especially when the language is not
1 Introduction
Task-oriented dialogue systems aim assist users in finishing specific tasks, and have been deployed in wide range of applications, e.g., tour guides (Budzianowski et al., 2018), medical applications (Levy and Wang, 2020), and especially in the e- commerce domain such as booking flights and ho- tels (EI Asri et al., 2017). With the rapidly ex- panding market and customers' growing demands
"odee is available at https//ggthab.com//mmmmyyygy/

---

in English, can be hard to acquire and collect due
responses, we expect that the adapted model can infer the intent better for each incoming utterance. Our contributions can be summarized as follows:
the high cost (Liang al., 2021). Furthermore, most of these studies only utilize customer utter- ances to detect intent, while neglecting the use- fulness of metadata (e.g., agent responses) from in-house customer services systems, which can be efficiently collected with customer utterances and without the requirement of annotation cost. How- ever, how to effectively utilize the agent responses benefit intent detection is an unknown yet chal- lenging problem, as the agent responses cannot be obtained before detecting the intent of customer requests during real-time deployment.
We propose a novel intent detection approach for customer service by integrating the agent response the pre-training objectives. Our method is flexible for the intent detection prob- lem and can also be deployed to variety of domains with task-oriented dialogue systems in similar contexts.
We designed two pre-training tasks, namely Response Retrieval and Response Generation, that enable the language model learn the implicit intent of customer utterances with agent responses and benefit from the recent advancements in contrastive learning.
In light of this, we aim to incorporate agent re- sponses with customer utterances in the training set to produce fine-grained knowledge. We hypothe- size that if the model is able to provide an accurate response for given utterance, it must have inferred the hidden intent behind the utterance. As shown in Figure 1, a customer may raise question about the manipulation of product. The agent response shows the concrete solution, "Press the button on the device", to the customer utterance due to the agent's understanding of the intent being the manip- ulation approach to the product. If the pre-trained model understands that the agent's ss response will include the manipulation approach to the product, the intent detection model will be able to classify the utterance into the intent: ProductManipulattio..
Experimental results on customer intent de- tection benchmarks demonstrate the effective- ness of our methods without the need for ad- ditional pre-training datasets.
Related work
Existing intent detection approaches have shown promising results by leveraging the ability of trans- fer learning with two-stage pre-training and fine- tuning process. Specifically, these works (Hender- son et al., 2020; Mehri et al., 2020; Casanueva et al., 2020; Mehri and Eric, 2021) involve pre- pre-training models in a self-supervised manner with mask lan- guage modeling (MLM) on large public dialogue datasets, and fine-tuning them on the target intent datasets. Zhang et al. (2021a) presented a contin- ued pre-training framework, where they pre-trained the model with MLM and cross-yntropy losses from few publicly available intent datasets to im- prove the few-shot performance on cross-domain intent detection tasks.
In this work, we propose RSVP, two-stage framework composed of pre-training step and a fine-tuning step to utilize the agent responses to effectively learn the intent customer utterances by introducing two pre-training tasks to learn a much more reliable conversational text encoder which has been further pre-trained to predict an ac- curate response. The first task, Response Retrieval, follows dual encoder paradigm and models the customer utterance as a dense retrieval batch of agent responses. This reformulation enables the pre-trained language model to discriminate be- tween the correct response and the others. The second task, Response Generation, constructs a conversational question guestion-answering answering problem with the utterances and responses. By directly learning to answer the customer utterances, the pre-trained language model can improve the extracted seman- tic knowledge of the agent responses. Finally, the pre-trained language model will be used to fine- tune the intent detection task. As the pre- pre-training tasks have aligned the model better with the agent
Additionally, a line of studies has introduced pre-training methods incorporating a variety of relevant NL.P tasks, such as natural language in- ference (NLI), to leverage knowledge from other public datasets. Zhang et al. (2020) recast in- tent detection as sentence similarity task and pre-trained the language model NLI datasets with BERT (Devlin et al., 2019) pairwise encoding, and they came up with a nearest-neighhor frame- work to take advantage of the transfer learning. However, their system is computationally expen- sive due to the full utilization of training data in

---

Ulferance
Conversttoral
Response Retrieval
Dr Moves
response
wepuor ertereee (hoil
Pre-training stage
intent
Convenatioral
Grnemter
Response Genoraltio
AVB bdoyn ym Happee rilarnre dasds.
Baggage
stterace process
Fine- Fine-tuning stage
Intent Prediction
Intent
response process
Detector
copy weigats
Figure 2:
The overall framework of RSVP with two related pre vre-training tasks: Response Retrieval and Response Generation to explore the agent responses in the dialogue system and learn the customer intents.
both the training and inference stages. Zhang et al. (2021b) pre- pre-trained language model on intent de- tection datasets with self-supervised learning and fine-tuned it with supervised-contrastvy learning (Gunel al., 2021). Zhang al. (2022) designed an additional regularization loss term tackle the anisotropy problem, as they found that supervised pre-training may yield a feature space where seman- tic vectors fall into narrow cone. Yehudai et al. (2023) reformulated intent detection as question- answering task and used intent label names as the answer to the customer utterances. Our method shares some similarities with Vulić al. (2021),
of length H, and yi is from one of the pre-defined customer intents C. The goal is classify the cor- rect intent label for the corresponding utterance. As described in Section 1, denote the agent sponse r; = [r] ......rr r.]]]]]]]]]] of length 7 for each u; to incorporate the metadata in the model.
RSVP overview. We introduce a simple yet general framework RSVP with two pre-training tasks from the agent responses, which aims to mimic how the agents think when they encounter an incoming utterance and try respond to it. The first pre- training task, Response Retrieval, proposed match the relations between customer utterances and agent responses (Section 3.2). The second pre- training task, Response Generation, is designed to learn the structured contexts of the agent responses (Section 3.3). In these manners, the model is able to provide appropriate responses to utterances and has accordingly aligned with the thoughts of real agents. Afterwards, we employ this pre- trained model in the intent detection fine-tuning stage and train the classification task with intent labels each utterance. Besides, to leverage the recent success of consistency regularization (Verma et al., 2019), which has been proven to be effective in improving classification performance, we introduce additional contrastive loss term enhance its robustness (Section 3.4).
who proposed pre-training the language model on a public Reddit dataset (Henderson et al., 2019) with response ranking task, and then fine-tuned it on an intent detection task by contrastive learn- ing. In contrast, in order to take the customer ser- vice agent's response into consideration and relieve the large-scale data labeling effort, pre-train directly the customer service dialogue with our two pre-training tasks: Response retrieval, which introduces a response selection task by retrieval- based batch contrastive loss, and response genera- tion, which formulates mimicking agent responses by a text generation task.
3 Method
3.1 Preliminary
3.2
Pre- Pre-Training: Response Retrieval
Problem definition. Our method aims to build
The response retrieval task accurately retrieve the appropriate agent response for given customer utterance by formulating it as a question-answeringg (QA) dataset. The advantages applying the pre-
intent detector from annotated dataset D
{(1,, v1), (u2, 92), (wp M)))), where yi is
the intent label of utterance u, = [u]. u² u

---

training task are two-fold: 1) pre-training with agent responses does not require additional annota- tion or generic dataset as the existing work does (Wu et al., 2020; Vulić et al., 2021), and 2) the structure of the utterance-response pair, in which the response aims to provide related answer to the utterance and should thus share a similar represen- tation (Karpukhin et al., 2020), enables context- aware comprehension of the conversation. Specifi- cally, we retrieve the utterance embedding qi Rd and response embedding P; E IRd, where d denotes the vector dimension, with a shared conversational encoder (-), composed of a BERT-based encoder and a linear layer. We adopt pooling layer on top of the BERT-based encoder outputs to only con- sider the [CLS] token hidden representations and feed them to the linear layer with tanh activation function. The process can be briefly denoted as:
encoder-decoder architecture, we make the hypoth- esis that the BERT encoder has been pre-trained to do classification tasks with its hidden represen- tation and can accordingly perform better in the downstream intent detection task, yet T5 is pre- trained to directly generate text. Therefore, the pre-trained conversational encoder is adopted the encoder, and pre-trained BERT is used as the decoder. There are two adjustments being made for the decoder initialization with BERT. First, the self- attention mechanism is masked to make the model only focus on the left side of the context. Sccond, an mncoder-decoder cross-attentin layer will be added to the decoder. Formally, for each utterance ti and response r; the model learns to estimate the conditional likelihood P(rilus) via minimizing the cross-entropy loss:
Leen -log P(rllu))
= o(u),
(1)
Σ LogP(riln!"" .u)).
(4)
P. o(r)).
(2)
We note that both the encoder and decoder are jointly updated to make the conversational encoder learn how to respond to an incoming utterance. In this way, the encoder can simulate the real-world scenario where a new agent is learning come up with an appropriate response and can thus further learn to align with the style of agents" responses.
To reinforce the model to learn the embedding space where the utterances and their corresponding responses are more likely to have stronger seman- tic relatedness as compared to irrelevant ones, we utilize an in-batch contrastive loss for each pair f and corresponding Pt as:
3.4 Fine-Tuning: Intent Detection
Leetr Σ log( Ej=1 15577
We employ Multi- Layer Perceptron (MLP) as the intent detector on top of the pre-trained conversa- tional encoder. In this stage, the training utterances are the same as in the pre-training stage, and we exclude the responses during fine-tuning since the customer's ' intent should be detected before agents start to respond in the real world. Therefore, we discard the decoder and adapt the encoder in the fine-tuning stage. Specifically, the pre-trained con- versational encoder employed to retrieve the ut- terance sentence embedding d;- We predict the estimated probability P: E IR [R| of the intents by the softmax output of the MLP classifier as follows:
where n is the batch size, T is a temperature param- eter, and sim(.:)) denotes the cosine similarity be- tween two given embeddings. The corresponding response of an utterance is viewed as the positive sample and the other responses in the same batch the negative samples.
3.3 Pre- Training: Response Generation
The goal of Response Generation is to ask the model to generate an appropriate response based on the corresponding utterance. We hypothesize that the model is able to precisely classify the intent if it has the capability to respond to an utterance, as the agents also have to recognize the intent before they respond. It is worth noting that our primary focus is not on the quality of the generated responses but rather on the capability of the model to capture the underlying intents of the utterances.
P: = softmax (v)()))
(5)
where v-(-) denotes an MLP.
RSVP aims to optimize the cross-entropy loss:
Lee
2 log(p.)-
(6)
This sequence-to-sent tosuuuencee task can be achieved with an encoder-decoder architecture. While T5 (Raffel et al., 2020) much more common as an
However, only incorporating cross-entropy loss may be sensitive to real-world noises (c.g., mis- spellings in the text or label noise) for performance

---

degradation (Ghosh and Lan, 2021; Cooper Stick- land et al., 2023). In light of this, we integrate un- supervised learning with the cross-entropy loss to ensure the utterance embeddings remain consistent with their corresponding augmentations to further enhance the robustness. Inspired by the success of the contrastive learning (Gao et al., 2021), we adopt augmented functioo against qi twice to gel two views d =())) and Ø: =(q:) as a positive pair. Specifically, we choose a dropout (Hinton al., 2012) as 2 since the dropout technique, which per- forms an implicit ensemble different sub-models by simply dropping a certain proportion of hidden representations, has been proven to be effective as a regularization method (Wu et al., 2021; Liu et al., 2021). Combing it together with contrastive learn- ing further foree q and to be consistent with each other and have similar representations.
each dialogue into single utterance and response for experiments. TwACS is composed of conver- sations between customer service agents from dif- ferent airline companies and their customers on various topics discussed Twitter posts. We used the processed TwACS for our experiments. All results are averaged over 5 different random seeds.
4.2 Baselines
To evaluate the effectiveness of our proposed frame- work, we compare RSVP against multiple strong baseline approaches for intent detection tasks*: 1) Classifier: BERT-based encoder which is fine- tuned with MLP classification head and cross- entropy loss. 2) ConvBERT (Mehri al., 2020): a BERT-based model pre-trained on a large unlabeled conversational corpus (Henderson et al., 2019) with around 700 million dialogues. 3) DNNC (Zhang et al., 2020): discriminative nearest-neighbor model managing to find the best-matched sample from the training data through similarity match- ing. It pre-trains the language model with differ- ent labeled sources for NLI (Bowman et al., 2015; Williams et al., 2018; Levesque, 201 I; Wang al., 2018) and uses the pre-trained model for down- stream intent detection. 4) CPFT (Zhang et al., 2021b): a two-stage framework that pre-trained a BERT- model with self-supervised objectives in the first stage and fine-tuned with supervised contrastive learning in the second stage.
The unsupervised objective can be defined as:
Cunx_et Log( Σラ= c(arm(sin95)1+))
Finally, the objective function of RSVP is:
Cic Lee Acune
(8)
where a weight hyper-parameter and dis- cussed in Appendix A.
4 Experiments
4.1 Intent Detection Datasets with Responses
4.3 Quantitative Results
Most widely studied public intent detection bench- marks (e.g., Clinc (Larson et al., 2019), Bank- ing77 (Casanueva et al., 2020), and HWU64 (Liu et al., 2019b)) are not composed of user utterances and agent responses, which serve an essential factor in task-oriented dialogue services. In order to evaluate our proposed RSVP in scenarios as dis- cussed the Section 1, we collected real-world Chinese datasets from KKday² with 50,276 sam- ples, and adopted a public English dataset with 491 samples, TwACS (Perkins and Yang. 2019).
We use accuracy as the main evaluation metric. following (Zhang et al., 2021b). In addition, we also consider the scores of Mean Reciprocal Rank (MRR) @ (3, 5) evaluate the ability of each model to predict correct intent with a better rank, which can be served as another suggestion for agents using the intent detection system in real- world scenarios. Table 1 summarizes the results on the two datasets. We find RSVP consistently outperforms baseline methods across two datasets in terms of all evaluation metrics. Quantitatively, RSVP surpasses the state-of -orthe--rttttt model, CPFT, by 1.7% and 8.2% for accuracy on KKday and TwACS respectively. It also improves CPFT by 1.3% and 5.5% for MRR@) and 1.2% and 4.3% for MRR @5 on both datasets. These demonstrate
The KKday dataset annotated by 8 agents con- tains anonymous itterance-respone pairs for the e-commerce travel domain. We removed the URL and Emoji from the utterances and responses to preprocess the dataset. We note that the dialogue in KKday consists of multi-turn utterances and re- sponses; therefore, we concatenated all customer utterances and all agent responses respectively in
http://iiuubb.oomaaaapeeees
taalog-intent-intee induction
"The implementation details and dataset statistics are de- scribed in Appendix A.
"haps:/wwww.kdday.commmrm

---

KKday
TwACS
Model
Acc
MRR @
MRR @5
Acc
MRR@3
MRR@5
Classifier
51.76
67.16
69.02
59.60
69.93
71.59
+ Cunazel
$2.68
67.75
69.55
63.20
72.00
73.36
ConvBERT (Mchri et al., 2020)
47.88
61.96
63.74
57.20
67.93
69.50
+ +uun
47.91
61.77
63.85
58.40
67.80
69.04
DNNC (Zhang et al., 2020)
49.79
63.07
65.25
34.00
43.13
45.55
CPFT (Zhang ct al,, 2021b)
52.55
67.27
69.09
63.60
71.80
73.64
RSVP
53.42
68.15
69.92
68.80
75.73
76.83
Table 1: Results (x 100%) on two intent detection datasets. The highest results for each metric indicated in boldface, while the second best are underlined. Our improvements over the second-best baseline are significant with p 0.05 (p-values based on Pearson's x" test).
the effectiveness of incorporating the agent re- sponses into the intent detection task without the need for external annotated pre-training datasets. To delve into the impact of contrastive learning in the fine-tuning stage, we experiment the variants of Classifier and ConyBERT during training and add contrastive learning along with cross-entropy loss (+ (t-ssssed) al) as RSVP. We note that DNNC and CPFT are excluded due to the incorporation of mul- tiple losses; therefore, adding additional loss requires carefully adjusting the hyper-parameters It can be observed that the auxiliary unsupervised contrastive learning also increases Classifier accu- racy by 5. 1 on average, and ConvBERT slightly performs better on accuracy as well. Meanwhile, RSVP still outperforms these baselines equipped with contrastive learning, which showcases the strength of our model's capacity.
MultiWOZ 2.2
SGD
Model
F1
Acc
F1
Acc
CPFT
96.73
84.50
65.91
32.46
RSVP
98.27
91.04
72.88
45.77
Table Results (x100%) on two multi-label intent detection datasets. Our improvements over the second- best baseline are significant with < 0.05 (p-values based on Pearson's x" test).
probability 0.5) must exactly match the corre- sponding set of labels in ground truth.
In terms of all datasets and metrics, we can ob- serve that RSVP significantly outperforms the best- performing baselines, CPFT, which further show- cases the robustness of our method across various dataset scenarios. The results indicate that super- vised contrastive learning loss in the fine-tuning stage of CPFT may restrict them to multi-label context due to the difficulty of learning from the positive and negative pairs for each multi-intent in- stance. In contrast, we use unsupervised contrastive learning with the dropout function for augmenta- tion.
4.4 Results on Larger Datasets
To further illustrate the comparisons between RSVP and the baselines, we experiment with two additional datasets, MultiWOZ 2.2 (Zang et al., 2020) and SGD (Rastogi et al., 2020), which con- tain user utterances, agent responses, and user in- tents in conversation. It is noted that these two datasets were designed detect the active intent of each utterance rather than the whole dialogue as our settings. To provide fair comparisons as ours, we reformulate the customer intent in a dialogue as the set of active intent appearing in all of the utterances for these two datasets (i.e., multi-label classification task). The Fl score and accuracy are incorporated as the evaluation metries and the results are shown in Table 2. We note that the ac- curacy metrics may not be intuitive in multi-label classification tasks, and we consider evaluating more strictly with subset accuracy the set of labels predicted for a sample (with a model predicting
4.5 Ablation Studies
Pre-training Task. To understand the importance of each framework component and the main factors leading to our improvements, we conducted several ablation studies and summarize the results Table 3. First, we analyzed the effects of the pre-training stage by removing two pre-training tasks: response retrieval and response generation, respectively. We see that both tasks necessary to achieve bet- ter performances across all datasets and metries, in line with the fact that task-adaptive pre-training stage reinforces the model with the flexibility to ad- just its feature space to the target task for boosting the downstream performance (Gueta et al., 2023).

---

KKday
TwACS
Model
Acc
MRR@3
MRR@5
Acc
MRR @3
MRR@5
RSVP
53.42
68.15
69.92
68.80
75.73
76.83
w/o Response Retrieval
53.33
68.04
69.86
57.60
65.87
67.57
Response Generation
52.91
67.63
69.37
67.60
73.87
75.67
reverse pre-training tasks order
52.91
67.61
69.37
63.60
70.60
72.20
w/o Luna replace BERT with mT5
52.82
67.85
69.58
62.80
72.87
73.61
52.25
67.40
69.04
46.00
55.60
$7.66
Table 3: Results (x 100%) of ablative experiments.
RgV 10V7 vi top,, Oct
* NOVP 1OV7 w/ 1o5p, Des
REVI a RDVP RoIo 96%
(a) Accuracy
(b) MRR@3
(c) MRR@5
12p0 12VP vio Tasp. Can
a nOV RDVP A/o Terp. Ca5
ROVP e nSVP w/t feer. Can
(d) Accuracy
MRR@3
(f) MRR@5
Figure 3: Effects (x 100%) of the batch size on the Response Retrieval training. (a-c) report the results on KKday with varying numbers of negative samples respectively. (d-f) report the corresponding results on the TWACS dataset.
It is worth noting that removing the response re- trieval task results in an enormous drop in accuracy (-16.3%) on the TWACS. This is also consistent with the observation that model with better utter- ance representation is essential when only a few ex- amples are available (Choshen et al., 2022), given that TWACS is a relatively small dataset.
discarding the unsupervised contrastive learning loss in the fine-tuning stage and using only cross- entropy loss instead, which is widely used setting in classification tasks. Our results show that the ad- ditional Lums_ed of loss term leads an enhancement in the accuracy of 5.7% on average, supporting the claim that contrastive learning effective terms of boosting the accuracy and robustness in the classification task.
To investigate the effects of the pre-training or- der, we report the model variant which reverses the order the two pre-training tasks, which demon- strates that the proposed order is superior to the reversed one. We hypothesize that the response generation task is more difficult than the response retrieval task, which is similar to the customer vice scenario in which it is more difficult for novice agent directly come up with the appro- priate response instead of picking one from a few candidates. This aligns with prior work showing that presenting training tasks ordered by difficulty level benefits not only humans but also AI mod- els (Xu et ul., 2020).
Effects of Encoder-decoder architecture. We ex- periment with replacing the BERT encoder with the encoder part of mT5 (Xue et al., 2021) and us- ing the whole mT5 in the response generation task, adopting mt5-base from (Wolf et al., 2020) for the two datasets. The conversational embeddings and p are retrieved by the average encoder out- puts across all corresponding input tokens, given that b5-based models do not have [CLS] token. can be observed that mT5 is not as effective BERT across all datasets, especially on the TwACS with a drastic drop in accuracy (-49%). We make an assumption that the phenomenon comes from
Effects of el We also experiment with

---

Utterances
Generated Responses
Predicted Intent
May ask which SIM card you would recommend for my upcoming Europe trip?
Here the information regarding Lurope SIM card.
WIFI/SMM Card inquiries
which offers 4G high-speed internet access through eSIM.
May know when will the booking of
Hello, you can visir the product page again in Decemher
Date does not open yet
this aquarium tickets on 10/9 be uvailable?
to see is available for urdering (in January next year).
Could you confirm if my bickets have been
Hello, your order has been successfully placed. If you
Transaction
successfully hooked? The wehsite shows that
have any questions, please feel free to contact our
failed
I used a coupon but no order was generated.
customer service team at any time. Thunk you.
Table 4: Examples of utterances, generated responses, and predicted intent of KKday, All sentences have been translated into English.
4.7 Case Studies
-Kikky TwADS
-Klday Twikc
-hikky TwASS
Analysis of generated responses and predicted intents. We present a few randomly sampled utter- ances and their corresponding generated responses by the trained response generator with their pre- dicted intents in Table 4. Although the quality of generated response not the primary focus of this work, it can be seen that the generated response can effectively answer the utterance, which may open another direction to adopt this for automatic replies in the future. For instance, the model successfully recognizes the SIM Card recommendation intent of the utterance in the first example, tailors reply with product information for the customer, and cor- rectly predicts the customer intent: WIFI/SIM Card inquiries. This is supportive of our intuition that a model aligned with the thoughts of real agents can better understand the intent of customer utterances.
(a) Accuracy
(b) MRR@) @
(c) MRR@5
Figure 4: Impacts of different À in the fine-tuning stage.
the need for a larger dataset to transform a larger model like mT5 into a conversational encoder and response generator.
4.6 Parameters Analysis
Number of negative examples in Response Re-
trieval. To analyze the impacts of the number of negative examples in the response retrieval pre- training stage, we set batch size n E {4, 8, 12, 16) as summarized in Figure 3. We observe that in- creasing batch size leads to an enhancement in the performance in most cases. We ascribe the ob- served improvement to the increasing difficulty of the response retrieval task. By training the model distinguish the correct response from a larger set of unrelated negative samples, the model rein- forced to learn more accurate representations of the utterances and responses. As a result, it gains en- hanced capabilities to detect customer intents more effectively in the intent detection task.
Relations between intent boundaries and utter- ances. In order to further understand the effect of the RSVP framework, we visualize the utter- ance representations from the test set of KKday extracted by diffcrent training stages of the con- versational encoder: BERT, BERT after response retrieval (w/ Resp. Retr.), BERT after response generation (w/ Resp. Gen.), and BERT after full RSVP pre-training and fine-tuning (Full RSVP). Each sample point will be passed through cor- responding model and its high high-dimension embed- dings will be reduced to two-dimension point with iSNE (van der Maaten and Hinton, 2012). We randomly chose 5 intent classes for the analysis as shown in Figure 5. All the trained conversa- tional encoders exhibited clearer group boundaries compared to the vanilla BERT, indicating that the RSVP framework can extract better conversational representations for the downstream intent detection tasks. In addition, we notice that the Price intent cannot cluster as other intents in Figure (b) and (c). The reason may be that the variety of utter- ances asking about products price makes it more
Impacts of Luns_ct- We also study the impact of A for unsupervised contrastive learning in the fine- tuning stage. We set À c {0.2, 0.1, 0.6, 0.8) and report the results in Figure 4. RSVP performs sta- bly with different numbers of X, while the results show effects TwACS. This is consistent with the observation of Zhang al. (2021b) that classifica- tion performance may be sensitive to the weight of contrastive learning loss with limited training examples. Nonetheless, RSVP still performs better than all baselines in these cases.

---

grating the label names with responses, etc.
RSS Cadd ingurte
WIIS Card ingutte
Prze
Fas spetonse ane
Limitations
One limitation of the RSVP framework is that it has not dealt with the noisy labels introduced by the agents' annotation in the KKday dataset. This is likely due to being multi-label implicitly for the dataset, as in the real world the intent of customer utterance may belong to not only one intent class, while it is classified into merely one class. That makes the annotation inconsistent between utter- ances. Since we leverage agent responses in pre- training, the framework can be further enhanced by incorporating them into the calibration of intent labels to mitigate the limitation. Another limita- tion is that we have not considered the out-of-scope (OOS) intents. To address this, we plan to utilize nearest neighbor algorithm along with the response pre-training task to further discriminate against the OOS utterances and responses in future work.
-22
-40
-12
-10
-00
(a) Vanilla BERT
(b) Resp. Retr.
SER Gord mpako
Coroebiton splictto
40
-c0
20
-59
(c) w/ Resp. Gen.
(d) Full RSVP
Figure 5: LSNE plots of encoded utterances of KKday.
difficult to group together with the help of only one pre-taining task, yet it can show clearer boundaries with the integration of both objectives (full RSVP). Another interesting observation is the inseparable bonding of two intents Transaction failed and Can- cellation application since they share pretty similar utterances and responses. The results show that only learning to provide an appropriate response for an utterance is not enough for good classifi- cation performance, and fine-tuning stage can relieve the confusion between similar samples.
Acknowledgements
We thank the anonymous EMNLP reviewers for their insightful comments and feedback. This re- search was supported in part by KKday.
References
Waheed Ahmed Abro, Guilin Qi. Muhammad Aamir, and Zafar Ali. 2022. Joint intent detection and slot filling using weighted finite state transducer and bert. Applied Intelligence, pages 1-15.
5 Conclusion
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632 642, Lisbon, Portugal. Association for Compu- tational Linguistics.
This paper presents RSVP, a novel two-stage frame- work to address the challenging intent detection problem in real-world task-oriented dialogue sys- tems. RSVP leverages customer service agents' responses to pre-train with customer utterances, of- fering a refreshing view utilize the metadata in the dialogue systems. It allows us explore the im- plicit intent lying in both utterances and responses, and takes advantage of contrastive learning to im- prove the model's robustness. RSVP consistently outperforms other strong baseline models on two real-world customer service benchmarks, showcas- ing the effectiveness of this framework. We believe RSVP can serve fiexibly for task-oriented dialogue intent detection problems, and several interesting research directions could be further investigated within this framework, such as more metadata re- lated to the conversation, pre-training tasks inte-
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, lñigo Casanueva. Stefan Ultes, Osman Ra- madan, and Milica Gašić. 2018. MultiWOZ a large- scale multi-domain Wizard-of-Ok dataset for task- oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
lñigo Casanueva, Tadas Temčinas, Daniela Gerz. Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Pro- ceedings of the 2nd Workshop Natural Language Processing for Conversational AI, pages 38-45, On- line. Association for Computational Linguistics.

---

Shiqian Chen, Chenliang Li, Feng Ji. Wei Zhou, and Haiqing Chen. 2019. Driven answer generation for product-related questions in e-commerce. In Pro- ceedings of the Twelfth ACM International Confer- ence on Web Search and Data Mining, pages 411- 419.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. 2021. Supervised contrastive learning for pre-trained language model fine-tuning. In Interna- rional Conference on Learning Representations.
Matthew Henderson, Pawet Budzianowski, Iñigo Casanueva, Sam Coope, Daniela Gerz, Girish Kumar, Nikola Mrkšić, Georgios Spithourakis, Pei-Hao Su, Ivan Vulié, and Tsung- 'ssug--Hiien Wen. 2019, A reposi- tory of conversational datasets. In Proceedings of the First Workshop on NLP for Conversational AI, pages 1-10, Florence, Italy, Association for Computational Linguistics.
Leshem Choshen, Elad Venezian, Shachar Don- Yehiya, Noam Slonim, and Yoav Katz. 2022. Where to start? analyzing the potential value of intermediate models. CoRR, abs/221 1.0007..
Asa Cooper Stickland, Sailik Sengupta, Jason Krone, Suab Mansour, and He Hc. 2023. Robustification of multilingual language models real-world noise in crosslingual zero-shot settings with robust contrastive pretraining. In Proceedings of the 17th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 1375-1391, Dubrovnik, Croatia. Association for Computational Linguistics.
Matthew Henderson, lñigo Casanueva, Nikola Mrkšić, Pai-Hao Su, Ssung-Hsien Wen, and Ivan Vulić. 2020. ConveRT: Efficient and accurate conversational repre- sentations from transformers. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, pages 2161-2174, Online, Association for Computa- tional Linguistics.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Πya Sutskever, and Ruslan Salakhutdinov. 2012. Im- proving neural networks by preventing adaptation of feature detectors. CoRR, abs/1207. .0580.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Haihong F, Peiqing Niu, Zhongfu Chen, and Meina Song. 2019. novel bi-directional interrelated model for joint intent detection and slot filling. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 5467- 5471, Florence, Italy. Association for Computational Linguistics.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of- scope prediction. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1311-1316, Hong Kong, China. Association for Computational Linguistics.
Layla El AsTi, Hannes Schulz, Shikhar Sharma, Jeremic Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017. Frames: a corpus for adding memory to goal-oriented dialogue systems. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 207-219, Saar- brücken, Germany. Association for Computational Linguistics.
Tianyu Gao. Xingeheng Yao. and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Hector J. Levesque. 2011. The winograd schema chal- lenge. In Logical Formalizations of Commonsense Reasoning, Papers from the 2011 AAAI Spring Sym- posium, Technical Report SS- 1--16 Stanford, Cali- fornia, USA, March 21 21-23, 2011. AAAI
Sharon Levy and William Yang Wang. 2020. Cross- lingual transfer learning for COVID-19 outbreak alignment. In Proceedings of the 1st Workshop NLP for COVID- at ACL 2020, Online. Association for Computational Linguistics.
Aritra Ghosh and Andrew Lan. 2021. Contrastive learn- ing improves model robusiness under label noise. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR) Work- shops, pages 2703-2708.
Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. 2023. Knowledge is a region in weight space for fine-tuned language models. CoRR, ob4/2302.04863.
Kevin J. Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, and Lawrence Carin. 2021. Mixkd: Towards efficient distillation of large- scale language models. In ICLR. OpenReview.net.

---

Pangyu Liu, Ivan Vulić, Anna Korhonen, and Nigel Collier. 2021. Fast, effective, and self-uuper Transforming masked language models into universal lexical and sentence encoders. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1442-1459, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Chongyang Tao, Shen Gao, Mingyue Shang, Wei Wu. Dongyan Zhao, and Rui Yan. 2018. Get the point of my utterance! learning towards effective responses with multi-head attention mechanism. In IJCAI, pages 4418-4424
Laurens van der Maaten and Geoffrey E. Hinton. 2012. Visualizing non-metric similarities in multiple maps. Mach. Learn., 87(1):33-555 55.
Jiao Liu, Yanling Li, and Min Lin. 2019a. Review of intent detection methods in the human-machine dialogue system. In Journal of Physics: Conference Series, volume 1267. page 012059. IOP Publishing.
Vīkas Verma, Alex Lamb, Juho Kannala, Yoshua Ben- gio, and David Lopez-Paz. 2019. Interpolation consistency training for semi-supervised learning. In Proceedings of the Twenty-Eightn International Joint Conference on Artificial Intelligence, IJCAI-19 19, pages 3635-3641 International Joint Conferences Artificial Intelligence Organization.
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019b. Benchmarking natural lan- guage understanding services for building conversa- tional agents. arXiv preprint arXiv: 193.00556..
Ivan Vulić, Pei-Hao Su, Samuel Coope, Daniela Gerz, Pawel Budzianowski, lñigo Casanueva, Nikola Mrkšic, and Tsung-Hsier Wen. 2021. ConvFiT: Con- versational fine-tuning of pretrained language models. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 1151-1168, Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam. CoRR,
a.5/1771.05101..
Shikib Mehri and Mihail Bric. 2021. Exavple-driven intent prediction with observers. In Proceedings of the 2021 Conference of the North American Chap- ler of the Association for Compulational Linguistics: Human Language Technologies, pages 2999. Online. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, 2018. GLUB: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Shikib Mehri, Mihail Eric, and Dilek Hakkani-Tur 2020. Dialoglue: A natural language understanding benchmark for task-oriented dialogue. arXiv preprint arXiv:2009. 13570.
Hugh Perkins and Yi Yang. 2019. Dialog intent induc- tion with deep multi-view clustering. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLPP) pages 4016-4025, Hong Kong, China, Association for Computational Linguistics.
Adina Williams. Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long Pupers), pages 11 12-1, New Orleans. Louisiana. Association for Computational Linguis- tics.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac. Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-ar natural language processing. EMNLP (Demos), pages 38-45. Association for Com- putational Linguistics.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghay Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the AAA/ Conference on Artificial Intelligence, vol- ume 34, pages
Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong, 2020. TOD-BERT Pre-trained natural language understanding for task oriente di- alogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 917-929, Online, Association for Computational Linguistics.
Shuangyong Song, Xiangyan Chen, Chao Wang, Xi- aoguang Yu, Jia Wang, and Xiaodong He. 2022, two-stage user intent detection model on complicated utterances with multi-task learning. In Companion Proceedings of the Web Conference 2022, pages 197- 200.

---

Lijun Wu, Juntao Li, Yue Wang. Qi Meng. Tao Qin, Wei Chen, Min Zhang, Tie- Yan Liu, et al. 2021. R-drop: Regularized dropout for neural networks. Advances in Neural Information Processing Systems, 34:10890- 10905.
detection via contrastive pre-training and fine-tuning. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural I Language Processing, pages 1906-1912, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. 2020. Curriculum learning for natural language understand- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6095-6104, Online. Association for Computational Linguistics.
Jianguo Zhang, Kazuma Hashimoto, Wenhao Liu, Chien-Sheng Wu, Yao Wan, Philip Yu, Richard Socher, and Caiming Xiong. 2020. Discriminative nearest neighbor few-shot intent detection by trans- ferring natural language inference. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 5064- 5082, Online. Association for Computational Lin- guistics.
Linting Xue. Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou. Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. m'15: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ler of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
A Experimental Setup
Table 5 reports the intents of the four datasets, and Table 6 reports the statistics of the two datasets. Uttr. and Resp. denote the Utterance and Response respectively.
Asaf Yehudai, Matan Vetzler, Yosi Mass, Koren Lazar, Doron Cohen, and Boaz Carmeli. 2023, QAID: ques- tion answering inspired few-shot intent detection. CoRR, abs/2303.01593.
A.1 Implementation Details
We conducted RSVP experiments with the bert-base-chinese and roberta-base encoders from (Wolf et al., 2020) as the initial models for two datasets with different languages, respectively. The encoders are used for all models: 768- (68-dimensional Transformer layers with 12 attention layers. We used the pre-trained COnvBERT and DNNC' the initialized models of ConvBERT and DNNC. We adopted the training set during the pre-training stage (i.e., response retrieval and response genera- tion), and set the training epochs to 10 for the two pre-training tasks respectively. We set the temper- ature 7 to 0.8 and batch size n = 16 with a maxi- mum length of 512 tokens due to the hardware con- straints. During the fine-tuning stage, we trained our model for 15 epochs from the pre- pre-trained check- point with batch size of 10, and set À to 0.5, mak- ing Lee the main summand in ffz. We used the AdamW (Loshchilov and Hutter, 2017) optimizer during both the pre- mre-training and fine-tuning stages with the learning rate to 2e - and a dropout ratio of 0.1. All the experiments were conducted on an NVIDIA RTX 3090.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ 2.2 A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 109-117, Online. Association for Computa- tional Linguisties.
Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, and Philip Yu. 2019. Joint slot filling and intent detection via capsule neural networks. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 5259-5267, Florence, Italy. Association for Computational Linguistics.
Haode Zhang, Haowen Liang, Yuwei Zhang, Li-Ming Zhan, Xiao-Ming Wu, Xiaolei Lu, and Albert Lam. 2022. Fine-tuning pre-trained language models for few-shot intent detection: Supervised pre-training and isotropization.
Proceedings of the 2022 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies. pages 532-542 Seattle, United States. Association for Computational Linguistics.
Haode Zhang, Yuwei Zhang, Li-Ming Zhan, Jiaxin Chen, Guangyuan Shi, Xiao-Ming Wu, and Al- bert Y.S. Lam. 2021a. Effectiveness of pre-training for few-shot intent classification. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 11 14-1 120, Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Jianguo Zhang, Trung Bui, Seunghyun Yoon, Xiang Chen, Zhiwei Liu, Congying Xia, Quan Hung Tran, Walter Chang, and Philip Yu. 202 Few-shot intent
http:///ghhhbbbmm///ggg/////e/ee Detection

---

Dataset
Intents
Itinerary prod page). itinerary UI inquiries. Date doesn't open yet. Date can't be selected, Transaction failed. Booking method, Customer can't find the order, Product comparison, Refund, Coupon, campaign Inquiries, About the Ticket/Prod. (How/when use). Payment Customization inquiries, Payment Currency inquiries, How to redeem, Details Inquary(Meeting time/point,
in pred page).
use,etc.), WIFI/SIM Card inquiries, Multiple accounts, KKday points related rules, Price. Confirm booking details. Transaction procedure, Recommendation, Add/Change booking information, Cancellation application, Transpor tation area, Complaint. Receipt application. System error, Fraudulent, Pushing order, Complaint Fully booked (Change the sate/Cancellation). Log-in inquiries, Positive feedback, Deactivation
KKday
Campaign, Resend the voucher,
Baggage, BookFlight. ChangeFlight, CheckIn, CustomerService, FlightDelay, FlightEnter tainment, FlightFacilityy. FlightStaff, Other, RequestFeature. Reward, aeriinalaacilityy. Tematonlpeeaaiioon
TwACS
F FindRestarrant, BookRestaurant FindActraction, F Findlottll, BookHotel, BookTaxi,FindTraii
MultiWOZ
BookTaxi, FindTrain, BookTrain. FindBus. F FindHHospital tal. FindPolice
BuyBusTicket, FindBus, TransferMoney CheckBalance. GetEvents, GeblvailableTTim, AddEvent, FindEvents, BuyEventTickets,, GetEventDates, SearchönewayFlight,, SearchRoundtripf1lee ReserveOnewayflightt ReserveeRunntrriofLiigttss
SGD
FindApartment, ScheduleVisit. ReserveHotel. SearchHotel, BookHouse, SearchHouse.
FindMovies, PlayMovie, GetTimesforrovie, LookupSong, PlaySong, LookupMusic, PlayMedia.
GetCarsAvailable ReserveCar. ReserveRestaurant, FindRestaurants, GetRide. BookAppointment. FindProvider, FindAttractions, GetWeather
Table 5: Dialog intents of the four datascts.
Dataset
KKday
TwACS
# #Train
40,219
392
#Valid
5,029
49
#Test
5,028
50
#Intents
38
14
#Max Intent
22.167
107
#Min Intent
10
Avg Uttr. Length
44.48
40.35
Avg Resp. Length
185.84
60.78
Table 6: Statistics of the two intent detection datasets.


Reviews:
[{"review_id": "dbl4qBb00o", "reviewer": "Reviewer_v7zY", "summary": "", "questions": "- Accuracy on KKday dataset is low compared to TWaCS even though KKday is much larger in size compared to TWaCS. Is there any explanation for that?\n\n- Is there any plans to release KKday dataset?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your helpful review. We are excited that the reviewer finds our approach using additional metadata instead of public datasets interesting, and recognizes that our results in various languages are detailed. In addition, we are thankful that the reviewer rates our case studies visualization at various stages of training as insightful. Our point-to-point responses to your comments are given below.\n\n> [R1] For given examples, benefit or use of explicit intent detection is not articulated well\n\nThanks for your feedback! Intent detection methods are developed to consider different urgent importance and dispatching for the corresponding departments. For instance, website-related problems are often assigned to the engineering department, and product-related utterances may need to be solved by the R&D department. It can be quite time-consuming and high-cost for agents to determine the intents, especially for a large number of intent counts. Therefore, automated customer intent detection benefits companies by reducing human efforts in distributing customers’ inquiries and thus improving the efficiency of agent response. We will rephrase the example to the final version.\n\n> [R2, Q2] KKday dataset is not provided or publicly available and not described well in the paper. Is there any plans to release KKday dataset?\n\nWe discussed the KKday dataset in Section 4.1 and summarized the intent metadata and statistics (e.g., average lengths of utterances and responses) in Tables 4 and 5 respectively – The dataset consists of multi-turn utterances and responses for the e-commence travel domain, where the intent labels were annotated by humans. While the KKday dataset is confidential and requires to removal of personalized information, we indeed plan to release the dataset afterward to foster the research community. Meanwhile, we experimented with the public dataset – TwACS and additional datasets (MultiWOZ 2.2 [1] and SGD [2]) as suggested by Reviewers ChMc and KLoz for reproducibility. These quantitative results have showcased the effectiveness of our method.\n\n> [R3] Minor improvement in accuracy compared to a classifier for KKday dataset\n\nWe would like to highlight the significant performance of our RSVP and the classifier baseline. Quantitatively, RSVP outperforms the classifier baseline by 3.2% for accuracy, 1.5% for MRR@3, and 1.3% for MRR@5 in terms of the KKday dataset. In addition, although adding the proposed contrastive loss (i.e., $L_{uns\\\\_cl}$) improves the performance of the classifier, RSVP still surpasses the baseline, which also highlights the generic design of incorporating contrastive learning into different models using only the CE loss. Nonetheless, Our improvements over baselines are still statically significant with a p-value < 0.05. These comparisons on the TwACS dataset further demonstrate the significant improvement.\n\n> [Q1] Accuracy on KKday dataset is low compared to TWaCS even though KKday is much larger in size compared to TWaCS. Is there any explanation for that?\n\nThe KKday dataset can be considered as the more challenging real-world scenario compared with the TwACS dataset for the following reasons.\n- The KKday dataset has multi-label implicity for each utterance as mentioned in Section Limitations, where the intent of a customer utterance may belong to not only one intent, while it is classified into merely one class. This provides a unique challenge for RSVP and all the other baselines.\n- In addition, the higher number of labels (38 of KKday v.s. 14 of TwACS) also hinders the performance of each model. \n- Nonetheless, as stated in the caption of Table 1, our experimental results still show statistically significant improvements over baselines, leading to the fact that 1.7% accuracy improvements are helpful in practice.\n\n---\n### Reference:\n\n[1] Zang, Xiaoxue, et al. \"MultiWOZ 2.2: A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines.\" Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI. 2020.\n\n[2] Rastogi, Abhinav, et al. \"Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.\" Proceedings of the AAAI conference on artificial intelligence. 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Proposed an interesting approach that uses additional metadata like agents response instead of public conversational dataset to pre-train a model\n- Results in English (TWaCS) & Chinese (KKday) datasets shows that the proposed approach works on various languages\n- Detailed ablation study on various components of the framework and contrastive learning\n- Insightful visualization of few selected utterance representations at various stages of training\n- Code is provided as supplementary material", "reasons_to_reject": "- For given examples, benefit or use of explicit intent detection is not articulated well\n- KKday dataset is not provided or publicly available and not decribed well in the paper\n- Minor improvement in accuracy compared to a classifier  for KKday dataset", "questions_to_authors": "- Accuracy on KKday dataset is low compared to TWaCS even though KKday is much larger in size compared to TWaCS. Is there any explanation for that?\n\n- Is there any plans to release KKday dataset?", "paper_topic_and_main_contributions": "In this paper, the authors propose a two-stage framework (RSVP) for intent detection of customer utterances in a conversational system. The proposed framework consists of two pre-training tasks that use the agent's response along with the customer's dialog to help the model learn the implicit intent. After the pre-training steps, the model is fine-tuned to predict the intent. Experimental results on the KKday and TwACS datasets show that RSVP outperforms baselines. The authors also conducted ablation studies to validate their two-stage framework. Code to reproduce results and TwACS dataset are provided as part of supplementary materials.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u3kxKDFO25", "reviewer": "Reviewer_tLuo", "summary": "", "questions": "1. Results on more datasets.\n2. What will happen when right response doesn't represent the right intent detection?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive suggestions. We are excited that you recognize our novelty in using agent response with two pre-training tasks and find our paper to be meticulously structured. We have discussed more results with additional datasets and our point-to-point responses to your questions are listed below.\n\n> [R1, Q1] Only two datasets are not enough for experiments. More results on additional datasets.\n\nThank you for the suggestions! To further illustrate the comparisons between RSVP and the baselines, we experiment with two additional datasets as suggested by Reviewer ChMc and KLoz, MultiWOZ 2.2 [1] and SGD [2], which contain user utterances, agent responses, and user intents in a conversation. These two datasets were originally designed to detect the active intent of each utterance rather than the whole dialogue as our settings. To provide fair comparisons as ours, we reformulate the customer intent in a dialogue as the set of active intent appearing in all of the utterances for these two datasets (i.e., multi-label classification task).\n\nThe F1 score (f1) and accuracy (acc) are used as our evaluation metrics. We note that the accuracy metrics may not be intuitive in multi-label classification tasks, and thus we evaluate it more strictly with subset accuracy – the set of labels predicted for a sample (with a model predicting probability >  threshold 0.5) must exactly match the corresponding set of labels in ground truth. We compare the best-performing baselines (CPFT) with our RSVP and the results are presented in the following tables for two additional datasets. These results are run with 5 seeds. It can be observed that RSVP outperforms CPFT in all cases, which also validates the claims in our paper. We will add these results to the final version.\n\n1. MultiWOZ 2.2 [1]\n|   | f1 | acc |\n|---|---|---|\n| CPFT | 96.73 | 84.50 |\n| RSVP (ours) | 98.27 | 91.04 |\n| **Improvements** | **1.6%** | **7.7%** |\n2. SGD [2]\n|   | f1 | acc |\n|---|---|---|\n| CPFT | 65.91 | 32.46 |\n| RSVP (Ours) | 72.88 | 45.77 |\n| **Improvements** | **10.6%** | **41.0%** |\n\n> [R2, Q2] Right response doesn't represent the right intent detection. What will happen when right response doesn't represent the right intent detection?\n\nGreat question. This scenario may not happen for companies since the agents are responsible for resolving questions raised by customers, where these questions are asked based on customers’ intents. Thus, in our opinion, the mentioned scenario will probably happen for self-volunteering answers on social platforms (e.g., Stack Overflow) that may not address the questions instead of companies that aim to address customers’ inquiries to maintain customer engagement.\n\nFor this scenario, it is still possible to classify the correct intent by fine-tuning the pre-trained model with the corresponding intents, which can guide the model to change the importance of incorporating the knowledge of agent responses. Nonetheless, our focus is to support real-world customer services for companies.\n\n---\n### Reference:\n\n[1] Zang, Xiaoxue, et al. \"MultiWOZ 2.2: A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines.\" Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI. 2020.\n\n[2] Rastogi, Abhinav, et al. \"Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.\" Proceedings of the AAAI conference on artificial intelligence. 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Proposes a novel approach to use agent response and two specific pretraining task for improving the intent detection.\n2. Provides both qualitative analysis and quantitative metrics demonstrating this method improve the performance of intent detection.\n3. he paper is meticulously structured and articulated, making it exceptionally easy to follow and understand.", "reasons_to_reject": "1. Only two datasets (with a very small one) are not enough for experiments.\n2. Right response doesn't represent the right intent detection. There is no discussion about this.", "questions_to_authors": "1. Results on more datasets.\n2. What will happen when right response doesn't represent the right intent detection?", "paper_topic_and_main_contributions": "Existing intent detection methods ignore the use of agent response. In this paper, we propose RSVP, a self-supervised framework dedicated to task-oriented dialogues, which utilizes agent responses for pre-training, which includes response retrieval and response generation task. Experiment demenstrate the effectiveness of the method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PmEZSVbo3v", "reviewer": "Reviewer_ChMc", "summary": "", "questions": "Q1: At section 4.1  mentions that publicly available intent detection benchmarks usually don't come with dialogues, however it seems that there are existing work [1] for intent detection using large-scale publicly available dataset (SGD dataset) with agent responses [2]?\n\nNot that every dataset set need to be included, but I feel it'd be helpful if the authors could consider incorporating at least one larger-scale, publicly available dataset for comparison.\n\n[1] Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. AAAI 2019. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. \n[2] Li Zhang, Qing Lyu, and Chris Callison-Burch. AACL 2020. Intent Detection with WikiHow. ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive review comments. We are glad you find our proposed method intuitive and the performance solid and well supported by the experiments. Our response to the question is given below.\n\n> [Q1.1] At section 4.1 mentions that publicly available intent detection benchmarks usually don't come with dialogues, however it seems that there are existing work [1] for intent detection using large-scale publicly available dataset (SGD dataset) with agent responses [3]?\n\nThanks for the suggestions! We delve into these mentioned works and find that the Wikihow dataset has a key difference from our scenario, where the response is represented as their intent label. For example, the utterance may be “let check-in agents and flight attendants know if it’s a special occasion”, and the response would be “Get Upgraded to Business Class”; in this case, the response is also the intent label of the utterance. Therefore, the intent label is open-ended and is not a closed set.\n\n> [Q1.2] Not that every dataset set need to be included, but I feel it'd be helpful if the authors could consider incorporating at least one larger-scale, publicly available dataset for comparison.\n\nThanks for the constructive comments! We add the experiment to the MultiWOZ 2.2 [2] and SGD [3] datasets accordingly and present the results in the following tables. These two datasets were originally designed to detect the active intent of each utterance rather than the whole dialogue as our settings. To provide fair comparisons as ours, we reformulate the customer intent in a dialogue as the set of active intent appearing in all of the utterances for these two datasets (i.e., multi-label classification task). \n\nThe F1 score (f1) and accuracy (acc) are used as our evaluation metrics. We note that the accuracy metrics may not be intuitive in multi-label classification tasks, and we consider evaluating it more strictly with subset accuracy – the set of labels predicted for a sample (with a model predicting probability >  threshold 0.5) must exactly match the corresponding set of labels in ground truth. We compare the best-performing baselines (CPFT) with our RSVP and the results are presented in the following tables for two additional datasets. These results are run with 5 seeds. We can observe that RSVP still outperforms CPFT similar to our findings in the paper. We will add the experimental results and discussions to our final version.\n\n1. MultiWOZ 2.2 [2]\n|   | f1 | acc |\n|---|---|---|\n| CPFT | 96.73 | 84.50 |\n| RSVP (ours) | 98.27 | 91.04 |\n| **Improvements** | **1.6%** | **7.7%** |\n2. SGD [3]\n|   | f1 | acc |\n|---|---|---|\n| CPFT | 65.91 | 32.46 |\n| RSVP (Ours) | 72.88 | 45.77 |\n| **Improvements** | **10.6%** | **41.0%** |\n\n---\n### Reference:\n\n[1] Li Zhang, Qing Lyu, and Chris Callison-Burch. AACL 2020. Intent Detection with WikiHow.\n\n[2] Zang, Xiaoxue, et al. \"MultiWOZ 2.2: A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines.\" Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI. 2020.\n\n[3] Rastogi, Abhinav, et al. \"Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.\" Proceedings of the AAAI conference on artificial intelligence. 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method is intuitive and has solid performance and is well supported by it's experiments and evaluations.\n- The paper is well written and easy to follow.\n", "reasons_to_reject": "\nSee questions\n", "questions_to_authors": "Q1: At section 4.1  mentions that publicly available intent detection benchmarks usually don't come with dialogues, however it seems that there are existing work [1] for intent detection using large-scale publicly available dataset (SGD dataset) with agent responses [2]?\n\nNot that every dataset set need to be included, but I feel it'd be helpful if the authors could consider incorporating at least one larger-scale, publicly available dataset for comparison.\n\n[1] Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. AAAI 2019. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. \n[2] Li Zhang, Qing Lyu, and Chris Callison-Burch. AACL 2020. Intent Detection with WikiHow. ", "paper_topic_and_main_contributions": "This paper proposes RSVP, a 2-stage pretraining + fine-tuning framework for training intent detectors. Experiments on two customer service datasets show the proposed method achieve superiors performance compared to various baseline methods.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AfnWP0mvGI", "reviewer": "Reviewer_KLoz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are exceedingly grateful for your insightful comments and really appreciate your time in reading our paper! We are excited that you acknowledge the practical usage of our method in real-world dialogue systems and the experiments on both Chinese and English. Our point-by-point responses to your comments are listed below.\n\n> [R1] The novelty of the paper is limited compared to the existing body of work of using pre-training tasks to improve model performance on intent detection. The contribution of the paper over the works in Zhang et al. 2021 [1] and Vulić et al. 2021 [2] appear incremental. From Table 1, the improvement over CPFT for the larger KKday dataset also seems marginal.\n\nWe would like to highlight the differences compared with existing approaches. [2] incorporate an additional Reddit dataset for the pre-training, while we only use the in-house dialogue dataset related to the downstream intent detection task without the need for pre-training on other large-scale datasets. Besides, their response ranking task which uses the multiple negatives ranking loss is also different from our proposed response retrieval and response generation pretraining tasks whose goals are to learn agent knowledge before classifying customers' intents.\n\nWhile we share some similarities with [1] in terms of the usage of contrastive learning, our approach introduces key differences: (1) The existing work is not able to take advantage of the metadata (agent response), which is beneficial for learning fine-grained conversational text encoder. (2) The supervised contrastive learning loss in their finetuning stage may restrict them to a multi-label context, as it is difficult to define and sample the positive and negative pairs for each multi-intent instance. In contrast, we use unsupervised contrastive learning with the dropout function for augmentation.\n\nWe summarize the discussions of experimenting with additional datasets in the response to R3 to reduce repetitive responses.\n\n> [R2] The paper makes a strong assumption that if the model generates an accurate response it implies that the model has understood the intent. This may not hold true for several real word customer service contacts where the customer mentions an unknown intent, or the initial turns from agents ask clarifying questions to customers.\n\nIndeed, the agent response may still be in initial turn and the intent is unknown in practice. Therefore, we incorporate the responses from historical records into the pre-training stage to enable the model to mimic the agent providing a response, which can be viewed as the orientation when a new agent joins the company to learn customers’ intents from previous records. As the agent is able to infer the customer intent before answering the utterance, our model can also be equipped with this ability after the proposed pre-training tasks. During the real-world inference stage, our RSVP with such knowledge can classify intents solely based on the mentioned scenarios without agent responses.\n\n> [R3] I would also like to see evaluation on other benchmark dialogue datasets such as SGD and MultiWoZ which contain user-system responses and the active intents in the conversation.\n\nThanks for providing these constructive suggestions! We conducted experiments on additional two datasets (MultiWOZ 2.2 [3] and SGD [4]) with RSVP and the strongest baseline CPFT. These two datasets share a similar format and need to detect the intent of each utterance, not the whole dialogue as our settings. To address this issue, we reformulate the customer intent in a dialogue as the set of active intent appearing in all of the utterances for these two datasets. \n\nWe use the F1 score (f1) and accuracy (acc) as the evaluation metrics and show the results in the following tables respectively. We note that the accuracy metrics may not be intuitive in multi-label classification tasks, and we consider evaluating it more strictly with subset accuracy – the set of labels predicted for a sample (with a model predicting probability >  threshold 0.5) must exactly match the corresponding set of labels in ground truth. We can see that RSVP can outperform CPFT in all cases, which further supports the robustness of our method. The results are also aligned with our perspective in R1, “CPFT might be restricted to multi-label context due to the SCL loss term”. We thank the reviewer for this and will definitely include these results and analysis!\n\n1. MultiWOZ 2.2 [3]\n|   | f1 | acc |\n|---|---|---|\n| CPFT | 96.73 | 84.50 |\n| RSVP (ours) | 98.27 | 91.04 |\n| **Improvements** | **1.6%** | **7.7%** |\n2. SGD [4]\n|   | f1 | acc |\n|---|---|---|\n| CPFT | 65.91 | 32.46 |\n| RSVP (Ours) | 72.88 | 45.77 |\n| **Improvements** | **10.6%** | **41.0%** |\n\n\n---\n### Reference:\n\n[1] Zhang, Jianguo, et al. \"Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuning.\" Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.\n\n[2] Vulić, Ivan, et al. \"ConvFiT: Conversational Fine-Tuning of Pretrained Language Models.\" Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.\n\n[3] Zang, Xiaoxue, et al. \"MultiWOZ 2.2: A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines.\" Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI. 2020.\n\n[4] Rastogi, Abhinav, et al. \"Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 05. 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is generally well written and easy to follow. It also addresses a practical task of intent detection in real-world conversational systems.\n\nExperiments are conducted on two datasets of conversational intent detection on both Chinese and English.", "reasons_to_reject": "The novelty of the paper is limited compared to the existing body of work of using pre-training tasks to improve model performance on intent detection. The contribution of the paper over the works in Zhang et al. 2021 and Vulić et al. 2021 appear incremental. From Table 1, the improvement over CPFT for the larger KKday dataset also seems marginal.\n\nThe paper makes a strong assumption that if the model generates an accurate response it implies that the model has understood the intent. This may not hold true for several real word customer service contacts where the customer mentions an unknown intent, or the initial turns from agents ask clarifying questions to customers.\n\nI would also like to see evaluation on other benchmark dialogue datasets such as SGD and MultiWoZ which contain user-system responses and the active intents in the conversation.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes to use response generation and retrieval as pre-training tasks for building an intent detection model. The main hypothesis is leveraging agent responses to learn a good pre-trained model for intent detection task in task oriented conversational system. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Eeeh3pXjEy", "reviewer": "Reviewer_GcM6", "summary": "", "questions": "L288，pre-trained BERT is used as the decoder, could you explain how to use pre-trained BERT as the decoder?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your helpful comments. We are glad you find our approach of using response in our two pre-training tasks has impacts on intent detection task. Our point-by-point responses to your comments and questions are listed below.\n\n> [R1] This paper lacks innovation, and the improvement in experimental results is not significant. As for the transferability of the approach, it is unclear whether the pre-trained model would be helpful for intent recognition in other scenarios.\n\nWe would like to clarify the main novelty of our paper. 1) Our generic pretraining design allows us to utilize the in-house metadata in the dialogue systems instead of using additional large-scale datasets to pre-train the model. 2) RSVP can be swiftly adapted to a variety of domains with task-oriented dialogue scenarios in similar contexts (i.e., utterance and metadata). 3) As stated in the caption of Table 1, our improvements over other baselines are statistically significant with a p-value < 0.05. In addition, we have further included two additional datasets to validate our proposed approach. Experimental results in different language datasets can also support this and the analysis provides useful information (e.g., the effects of $L_{uns\\\\_cl}$ from L471 and the relations between intent boundaries and utterances in L542) to the community. \n\nTo further discuss the transferability and robustness of our approach, we experiment with two public dialogue datasets, namely MultiWOZ 2.2 [1] and the SGD [2] datasets. Both of them are initially designed for intent recognition tasks on each utterance, while we manage to detect the intents of the whole dialogue; therefore, we reframe the customer intent in a dialogue as the set of active intent appearing in all of the utterances for these two datasets, and thus the intent detection task will become a multi-label classification task. We consider the F1 score (f1) and accuracy (acc) as our evaluation metrics, and the quantitative results are listed in the following tables. We note that the accuracy metrics may not be intuitive in multi-label classification tasks, and we consider evaluating it more strictly with subset accuracy – the set of labels predicted for a sample (with a model predicting probability >  threshold 0.5) must exactly match the corresponding set of labels in ground truth. We can see that RSVP outperforms the strongest baseline CPFT across all cases, showcasing the superiority and transferability of our method.\n\n1. MultiWOZ 2.2 [1]\n|   | f1 | acc |\n|---|---|---|\n| CPFT | 96.73 | 84.50 |\n| RSVP (ours) | 98.27 | 91.04 |\n| **Improvements** | **1.6%** | **7.7%** |\n2. SGD [2]\n|   | f1 | acc |\n|---|---|---|\n| CPFT | 65.91 | 32.46 |\n| RSVP (Ours) | 72.88 | 45.77 |\n| **Improvements** | **10.6%** | **41.0%** |\n\n> [Q1] L288，pre-trained BERT is used as the decoder, could you explain how to use pre-trained BERT as the decoder?\n\nAs stated in L289-294, we have provided instructions on how to adjust pre-trained BERT for initializing a decoder: 1) Modify the original self-attention mechanism to employ causal masking, preventing attending to subsequent tokens during decoding. 2) Add an encoder-decoder cross-attention layer to the decoder. This adjustment approach provides the benefits of using pre-trained checkpoints for seq2seq tasks, leading to saved pre-training time [3]. We also conducted an ablation experiment on different encoder-decoder architectures (see Table 2: “replace BERT with mT5”), confirming that the method introduced in [3] achieved superior performance.\n\n---\n### Reference:\n\n[1] Zang, Xiaoxue, et al. \"MultiWOZ 2.2: A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines.\" Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI. 2020.\n\n[2] Rastogi, Abhinav, et al. \"Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 34. No. 05. 2020.\n\n[3] Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging Pre-trained Checkpoints for Sequence Generation Tasks. Transactions of the Association for Computational Linguistics, 8:264–280.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The approach of using response retrieval and response generation as pretraining tasks does have a certain impact on intent recognition.", "reasons_to_reject": "This paper lacks innovation, and the improvement in experimental results is not significant. \nAs for the transferability of the approach, it is unclear whether the pre-trained model would be helpful for intent recognition in other scenarios.", "questions_to_authors": "L288，pre-trained BERT is used as the decoder, could you explain how to use pre-trained BERT as the decoder?", "paper_topic_and_main_contributions": "This paper is about the problem of improving the accuracy of intent recognition using pretraining methods. The approach is to use response retrieval and response generation as pretraining tasks to enhance the accuracy of intent recognition.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper uses response generation and builds an intent detection using gold responses as retrieval as pre-training tasks for building an intent detection model. While it's a great paper, results on more datasets would strengthen the argument.
Decision: Accept-Findings