Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of ChatGPT
Authors: Xiaoshuai Song
Keqing He
Pei Wang
Guanting Dong
Yutao Mou
Jingang Wang
Yunsen Xian
Xunliang Cai
Weiran Xu
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Models
Evaluation
Out of Domain
Intent Recognition

Abstract: The tasks of out-of-domain (OOD) intent discovery and generalized intent discovery (GID) aim to extend a closed intent classifier to open-world intent sets, which is crucial to task-oriented dialogue (TOD) systems. Previous methods address them by fine-tuning discriminative models. Recently, although some studies has been exploring the application of large language models (LLMs) represented by ChatGPT to various downstream tasks, it is still unclear for the ability of ChatGPT to discover and incrementally extent OOD intents. In this paper, we comprehensively evaluate ChatGPT on OOD intent discovery and GID, and then outline the strengths and weaknesses of ChatGPT. Overall, ChatGPT exhibits consistent advantages under zero-shot settings, but is still at a disadvantage compared to fine-tuned models. More deeply, through a series of analytical experiments, we summarize and discuss the challenges faced by LLMs including clustering, domain-specific understanding, and cross-domain in-context learning scenarios. Finally, we provide empirical guidance for future directions to address these challenges.

Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of ChatGPT
Xiaoshuai Song'', Keging He"*, Pei Wang', Guanting Dong', Yutao Mou' Jingang Wang', Yunsen Xian', Xunliang Caî", Weiran Xu
Beijing University of Posts and Telecommunications, Beijing, China "Meituan, Beijing, China
(songxiaoshuai, wangpei, dongguanting myt, xuwei jobuptt edu. cn (hekeqing, wang i ingang, x xianyunsen, cai caixunlinn g j tuan. com
Abstract
OOD Intent Clusters
A
The tasks of out-of-domain (OOD) intent dis- covery and generalized intent discovery (GID) aim to extend a closed intent classifier to open- world intent sets, which crucial to task- oriented dialogue (TOD) systems. Previous methods address them by fine-tuning discrimi- native models. Recently, although some studies have been exploring the application of large lan- guage models (LLMs) represented by ChatGPT various downstream tasks, is still unclear for the ability of ChatGPT to discover and incre- mentally extent OOD intents. this paper, we comprehensively evaluate ChatGPT on OOD intent discovery and GID, and then outline the strengths and weaknesses of ChatGPT. Overall, ChatGPT exhibits consistent advantages under zero-shot settings, but still at a disadvantage compared fine-tuned models. More deeply, through series of analytical experiments, we summarize and discuss the challenges faced by LLMs including clustering, domain-specific derstanding, and cross-domain in-context Iearn- ing scenarios. Finally, we provide empirical guidance for future directions to address these challenges.'
IND Sappl
OOD Intent Discovery
Generaled Intent Discovery
OOD Unidbeled Susppees
Figure I: Illustration of OOD Intent Discovery and GID.
their intents, which facilitates identifying po- tential directions and developing new skills. The General Intent Discovery (GID) task (Mou et al., 2022b) further considers the increment of OOD intents and aims to automatically discover and in- crement the intent classifier, thereby extending the scope of recognition from existing IND intent set to the open world, as shown in Fig 1.
Previous work studied above OOD tasks by fine- tuning the discriminative pre-training model BERT (Devlin et al., 2018), 8). Recently, a series of pow- erful generative LLMs have been proposed one after another, such as GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022) and LLaMA (Tou- vTon et al., 2023). The emergence of LLMs has brought revolutionary changes to the field of natu- ral language processing (NLP). Given the superior in-context learning ability, prompting LLMs has become a widely adopted paradigm for NLP re- search and applications (Dong et al., 20226). Since these LLMs are trained on a large amount of gen- eral text corpus and have excellent generalization ability, this triggers the thinking of what benefits LLMs can bring and what challenges will LLMs face when applying them to open-scenario intent discovery and recognition?
1 Introduction
Traditional task-oriented dialogue (TOD) systems are based on the closed-set hypothesis (Chen et al., 2019; Yang et al., 2021; Zeng et al., 2022) and only handle queries within a limited scope of in-domain (IND) intents. However, users may in- put queries with oft-oindommin (OOD) intents in the real open world, which poses new challenges for TOD systems. Recently, a scries of tasks tar- geting OOD queries have received extensive re- search. OOD intent discovery (Lin et al., 2020; Zhang et al., 2021; Mou et al., 2022a) aims to group OOD queries into different clusters based
As one of the representative LLMs, ChatGPT, developed by OpenAI, has attracted significant at- tention from researchers and practitioners in a short period of time. While the NLP community has
"The first two authors contribute equally. Weiran Xu the corresponding author.
'Wo release our code at https://github.com mongriaashuai/00--vaall.a

---

been studying the ability of LLMs be applied to various downstream tasks, such as translation (Jiao
ter, which is available in the discriminant model.
et al., 2023), mathematies (Frieder et al., 2023),
What ChatGPT does not do well:
education (Malinka et al., e recogni- tion(Lei et al., 2023), its ability in OOD still not fully explored. Different from Wang et al. (2023) which evaluate the robustness of ChatGPT from the adversarial and out-of-distribution perspective, OOD intent discovery and GID focuses using IND knowledge transfer to help improve TOD sys- tems with OOD data. this paper, on the one hand, we focus on OOD intent discovery and GID the two scennscenario intent tasks to explore whether prompting ChatGPT can achieve good performance in discovering and incrementally recognizing OOD intents; on the other hand, we aim to gain insights into the challenges faced by LLMs in handling open-domain tasks and potential directions for im- provement.
For OOD intent discovery, ChatGPT performs far worse than the fine-tuned baselines under multi-sample or multi-category scenes, and is severely affected by the number of clusters and samples, with poor robustness.
For GID, the overall performance of ChatGPT is inferior to that of the fine-tuned baselines. The main reason is the lack of domain knowl- edge, and the secondary reason is the quality of the pseudo-intent set.
There are obvious recall errors both OOD discovery and GID. In OOD discovery, this is mainly due to the generative architecture of ChatGPT. In GID, recall errors are mainly caused by ChatGPT's lack of domain knowl- edge and unclear understanding of intent set boundaries.
To the best of our knowledge, we are the first to comprehensively evaluate ChatGPT's performance on OOD intent discovery and GID. In detail, we first design three prompt-based methods based on different IND prior to guide ChatGPT to perform OOD discovery in an end-to-end manner. For GID, we innovatively propose a pipeline framework for performing GID task under a generative LLM (Sec- tion 3). Then we conduct detailed comparative experiments between ChatGPT and representative baselines under three dataset partitions (Section 4). In order to further explore the underlying reasons behind the experiments, we conduct series of an- alytical experiments, including in-context learning under cross- donss- demonstrations, recall analysis, and factors that affect the performance of ChatGPT on OOD discovery and GID. Finally, we compare the performance of different LLMs on these OOD tasks (Section 5).
ChatGPT can hardly learn knowledge from IND demonstrations that helps OOD tasks and may treat IND demonstrations noise, which brings negative effects to OOD tasks.
In addition to the above findings, we further sum- marize and discuss the challenging scenarios faced by LLMs including large scale clustering, seman- tic understanding of specific domain and cross- domain in-context learning in Section 6 as well as provide guidance for future directions.
Related Work
2.1 Large Language Models
Our findings. The major findings of the study in- clude:
Recently, there are growing interest in leveraging large language models (LLMs) perform various NLP tasks, especially in evaluating ChatGPT in various aspects. For example, Frieder et al. (2023) investigate the mathematical capabilities of Chat- GPT by testing it on publicly available datasets, as well as hand-crafted ones. Tan et al. (2023) ex- plore the performance of ChatGPT on knowledge- based question answering (KBQA). Wang et al. (2023) evaluate the robustness of ChatGPT from the out-of-distribution (OOD) perspective. A ries works by Liu et al. (2023); Guo et al. (2023); Dong et al. (2022a, 2023) explore the impact input perturbation problems on model performance
What ChatGPT does well:
ChatGPT can perform far better than the non- fine-tuned BERT on OOD tasks without any IND prior, thanks to its powerful semantic understanding ability.
For OOD intent discovery, when there are few samples for clustering, ChatGPT's perfor- mance can rival that of fine-tuned basclines.
ChatGPT can simultancously perform text clustering and induce the intent of each clus-

---

give you seatences, , which recorted Set Flutt Please classly senteaces in iuto (Number of Clustars)
catagortes acconding to their intentions Y ou only need outyot the category number and the cormesponding sentence number in the following forma::
Cassgury I: 1,2.3,4,5
Direct chustering (DC)
jost can choose
delivery? comm
Nest, will first give you a set of intrrt categuries which will be recorded as See 1. Then I will give you another set of semtenoes without intrntion lohels,
recorted Set 2 first need leorn the imowledge Set and then the learned knowledge to classify the sentenoss Set 2 (Number Clustors)
categoriis according to their intentions. Y yu only need output the category number and the correponding sentence zumber in the following formst:
Canegury 1: 1,2,3,4,5
It should
intention set and intention set not overaap.
Set 11): pin, blocked, .....
Zero Shot Discovery (ZSD)
Set 2ID222 y
choose date for delivery? ?
Next will fistt give you a set sectences with intection labea, which will recceded as Set Then will give you another set of senteacess without intention labels, moondeed as Set 2. ou first need to learn the imowledge in Set and than use the learned knowlodge to classify the sentenoes in Set 2 into 5 Numbar of Ciustrin)
catagortes according their intentions. yu only need octyut category number Category 1: 1,2,3/4,5 ......
corresoonding sentence zumber the following format:
shoold be notted that the intention in sot and the intention in set do not overlap.
(p How unblock my
intertionginn blocked, ......
Few Shot Discovery (FSD)
Set 2 (D y: I. can I choose a date for delivery? .....
Figure 2: The different prompts for three methods of OOD intent discovery.
from small models to LLMs. In this paper, we aim to investigate the ability of ChatGPT to discover
3 Methodology
3.1 Problem Formulation
and increment OOD intents and further explore the challenges faced by LLMs and potential directions for improvement.
OOD Intent Discovery Given a set of labeled IND dataset DIND {[>:ND 1 1/, (ND) ) and unlabeled OOD dataset DOOD (1 ( y where all queries from DIND belong to a prede- fined intent set Y containing N intents, and all queries from DOOD belong to an unknown set y containing M intents?. OOD intent discov- ery aims to cluster M OOD groups from DOOD under the transfer of IND prior from DIND
2.2
OOD Intent Discovery
Unlike the simple text clustering task, OOD intent discovery considers how to facilitate the discovery of unknown OOD intents using prior knowledge of IND intents Lin et al. (2020) use OOD represen- tations to compute similarities weak supervision signals. Zhang et al. (2021) propose an iterative method, DeepAligned, that performs representation learning and clustering assignment iteratively while Mou et al. (2022c) perform contrastive clustering to jointly learn representations and clustering assign- ments. In this paper, we evaluate the performance of methods based ChatGPT about OOD discovery and provide detailed qualitative analysis.
General Intent Discovery GID aims to train a network that can simultancously classify a set of labeled IND intent classes IND containing N intents and discover new intent set YOOD con- taining M intents from unlabeled OOD set DOOD {(x,0D))m Unlike OOD discov- ery clustering that obtains M OOD groups, the ultimate goal of GID is to expand the network's classification capability of intent query to the total label set Y = INN U yOod containing N + M intents.
2.3 General Intent Discovery
Since OOD intent discovery ignores the fusion of IND and OOD intents, it cannot further expand the recogmition range of existing TOD systems. In- spired by the above problem, Mou et al. (2022b) propose the General Intent Discovery (GID) task, which requires the system to discover semantic concepts from unlabeld OOD data and then jointly classifying IND and OOD intents automatically. Furthermore, Mou et al. (2022b) proposes two frameworks for performing GID task under dis- criminative models: pipeline-based and end-to-end frameworks. In this paper, we propose a new GID pipeline under generative LLMs and explore the performance of ChatGPT in different scenarios.
3.2 ChatGPT for OOD Discovery
We evaluate the performance of ChatGPT on OOD intent discovery by designing prompts that include task instructions, test samples, and IND prior. We heuristically propose the following three methods based on different IND prior:
Direct clustering (DC): Since OOD intent dis- covery is essentially a clustering task, a naive approach is to cluster directly without utilizing any IND prior. The prompt is in the following
"Sttmatting M is out of the scope of this paper. In the following experiment, we assume that M is ground-truth and provide an analysis in Section 5.5. It should be noted that the specific semantics of intents in yooD are unknown.

---

Previous:
tion for generating intent descriptions, which are formally <OOD Discovery Prompt><Intent De- scribe Instruction>, input to ChatGPT, and obtain the intent description of each cluster. By aggregat- ing these intent descriptions, we obtain the OOD pscudo-intent set ypeadd Then, we incrementally add the pseudo-intent set to the existing IND intent set, ie.. Yjoint YIND U Y
Labrled IND Sumples
Unlabeled
Sample
Preidde Libelied
000 Sumple
OCD pacade-llbel indes cample
Skep., Ciasiiiatioon
New:
Primpi
Somples
Taei Ouary
peeudo
Next, we imput prompts for joint IND and OOD intent classification in the following form: <Classi- fication Instruction><Antent Set: joint >< z| E DIDD U DOOD
Step1 Obton
Sepp. Clvedfiction
Figure 3: Comparison of the discriminant and generative GID framework.
test >. According to the three differ- ent OOD discovery methods in Section 3.2, there are also three GID methods for ChatGPT: GID under Direct clustering (GID-DC), GID under Zero Shot Discovery (GID-ZSD), and GID un- der Few Shot Discovery (GID-FSD). should be noted that the GID-FSD method, we provide few labeled samples of IND as demonstrations. The complete prompts are provided in Appendix D.
test
format: <Cluster Instruction><NNumber of Clus- ters><Response Format><12
Zero Shot Discovery (ZSD): This method provides the IND intent set in the prompt as prior knowledge, but does not provide any IND samples. It can be used in scenarios where user privacy needs to be protected. The prompt is in the following format: <Prior: y YIND >CCuster Instruction><<Nubbe> of Clus- ters><Response Format><11000>
Experiment
4.1 Datasets
We conduct experiments on the widely used intent dataset Banking (Casanueva et al., 2020). Bank- ing contains 13,083 user queries with 77 intents in the banking domain. Due to the length limi- tation of ChatGPT's conversations, sample 15 categories from Banking as IND intents and considered three OOD category quantity set- tings. Specifically, the OOD category quantity is (IND/OOD=3:1), 10 (IND/OOD=3:2), and 15 (IND/OOD=1:1), respectively. For OOD intent dis- covery, we randomly sample 5 queries from the test set for each OOD class, where the ground-truth number of OOD classes is given as prior. For GID, we randomly sample 10 queries from the test set for testing. In addition, only when using FSD or GID-FSD method, we randomly sample queries for each IND class from the training set for demon- stration. We provide detailed statistics of datasets in Appendix A.
Few Shot Discovery (FSD): FSD provides sev- eral labelled samples for each IND intent in the prompt, hoping that ChatGPT can mine domain knowledge from IND demonstration and transfer it assist OOD intent clustering. The prompt is in the following format: <Prior: DIND & yIND><Cluster Instreetion><Number of Clus-
randomly
ters ters><Response Format><D Iest
According to the input, ChatGPT outputs the index of OOD samples contained in each cluster, in the form of <Cluster Index><OOD Sample Index>, We show the prompts of these methods in Fig
3.3 ChatGPT for GID
Previous discriminative GID framework first assign pseudo-label index to each OOD sample through clustering and then jointly training the classifier with labeled IND data. However, the classification of queries by generative LLMs depends on specific intent semantics rather than abstract pseudo-label index symbols. Based on this, we innovatively propose new framework that is suitable for gen- erative LLMs, which relies on LLMss to generate intent description with specific semantics as the pseudo-intent for each cluster, shown in Fig 3. In the first stage, on the basis of OOD intent discovery prompts, we add an additional instruc-
4.2 Baselines
For OOD intent discovery, we choose to use BERT directly for k-means clustering (MacQueen, 1965) and two representative fine-tuned methods:
DeepAligned (Zhang et al., 2021) is an im- proved version of DeepCluster(Caron et al., 2018). It designed a pseudo label alignment

---

IND/OOD=3:1
INDIOOD=312
INDIOOD=1: 1:1
Prior
Method
ACC
NMI
ARI
ACC.
NMI
ARI
ACC
NMI
ARI
BERT
52,00
41,58
15.36
36.00
42.33
5.616
29.33
45.26
2,499
W/O IND Prior
Cha)GPT(DC)
88.00
84.62
73.36
78,00
78.20
55.32
58.22
63.30
28.21
DeepAligned
100.0
100.0
100.0
78.67
82.18
61.01
74.67
80.83
55.35
DKT
93.33
91.71
84.82
80.67
82.59
64.53
76.45
83.23
60.92
With IND Prior
ChatGPT(ZSD)
92.00
87.25
80.16
67.33
68.72
39.32
50.67
60.73
21.25
ChatGPT(FSD)
74.67
64.77
45.92
56.67
63.56
31.47
49.78
60.98
20.72
Table I: Performance comparison on OOD intent discovery. Results are averaged over three random run (p < 0.01
under I-test).
INDIDODES1
IND.ODDEV2
INDIODD= F:]
O0D
ODD
F1
ACC
F1
ACC
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
Gigi
95.36
97.00
95.12
92.67
35.99
56,00
76.38
54.42
E26
96.13
95.50
97.00
97.00
96.35
95.88
95.21
93.33
78.69
30.5
88.
88,602
88.2
94.22
92.17
71.92
74.00
83.07
83.08
OwGPniGD--D DC)
67.41
68.44
70.36
75.33
67.96
70.17
62.15
64.67
57.53
61.33
60.30
63.53
63.06
66.44
59.72
62.00
61.39
64.22
OwGGnGGm-225
64.47
ChaNGPTGGDDFSD
72.77
79.1
20.27
17.33
39.65
63.67
68.74
74.89
50.75
52.000
61.54
65.73
68.29
74.89
52.57
$1.78
60443
63.33
Table 2: Performance comparison on GID.
strategy to produce aligned cluster assign- ments for better representation learning.
We only use the samples belonging IND and OOD intents in training set of Banking train all fine-tuned methods.
DKT (Mou et al., 2022c) designs a unified multi-head contrastive learning framework to match the IND pretraining objectives and the OOD clustering objectives. In the IND pre- training stage, the CE and SCL objective func- tions are jointly optimized, and in the OOD clustering stage, instance-level CL and cluster- level CL objectives are used jointly learn representation and cluster assignment.
4.3 Evaluation Metrics
For OOD intent discovery. We adopt three widely used metrics to evaluate the clustering results: Ac- curacy (ACC)', Normalized Mutual Information (NMI), and Adjusted Rand Index (ART). For GID, adopt metrics: Accuracy (ACC) and F1- score (F1), assess the performance of the joint classification results. Besides, we observe that all ChatGPT methods have the phenomenon that some samples are not assigned any cluster or intent (miss- ing recall), while some are assigned multiple clus- ters or intents (repeated recall). For samples with missing recall, we randomly assign cluster intent; for those with repeated recall, we only tain the first assigned cluster or intent. We provide detailed recall analysis in Section 5.3.
For GID, the baselines are as follows:
DeDDAligned-GID is a representative pipeline method constructed by (Mou et al., 2022b) based on DeepAligned, which first uses the clustering algorithm DeepAligned to cluster OOD data and obtains pseudo OOD labels, and then trains new classifier together with IND data.
4.4 Main Results
Table and Table respectively show the main re- sults of the ChatGPT methods and baselines under OOD discovery and GID for three dataset divisions.
E2E (Mou et al., 2022b) mixes IND and OOD data in the training process and si- multaneously learns pseudo OOD cluster as signments and classifies all classes via self- labeling. Given an input query, E2E connects the encoder output through two independent projection layers, IND head and OOD) head, as the final logit and optimize the model through the unified classification loss, where the OOD pscudo label is obtained through swapped pre- diction (Caron et al., 2020).
Next, analyze the results from three aspects: (1) Compare Method without IND Prior From Table 1, we can see that without any IND prior knowledge, i.e., direct clustering, ChatGPT's per- formance is significantly better than BERT un- der three dataset partitions, indicating ChatGPT's
*We use the Hungarian algorithm (Kuhn, 1955) to obtain the mapping between the prediction and ground truth clasees.

---

strengths in natural language understanding with- out using any private specific data. For example, under IND/OOD=3:1, ChatGPT(DC) outperforms BERT by 36.00% (ACC). As the number of OOD classes increases, the clustering metrics of Chat- GPT(DC) and BERT decrease rapidly, but Chat- GPT(DC) still achieves better performance than BERT, exceeding BERT by 28.89% (ACC) under IND/OOD=1:1.
WDO Dewwi
MD0 Doms
Figure 4: The impact of different demonstrations. W/O Demos means "no samples for demonstration". Under IND/OOD Demos, we demonstrate 3 labeled samples for each IND/OOD class.
(2) Compare ChatGPT with Finetuned BERT For OOD discovery, when the OOD ratio is rela- tively low, the optimal ChatGPT method is slightly inferior to fine-tuned baselines. However, as the OOD ratio increases, ChatGPT is significantly lower than fine-tuned model. We believe this is because as the OOD ratio increases, the number of clustered samples increases and more data brings more difficult semantic understanding challenges generative LLMs. However, discriminative fine- tuned methods encode the samples one by one and are therefore less affected by the OOD ratio.
through in-context learning. However, its OOD classification metric is as good as that of GID- DC. We think this because the quality of the pseudo-intent set induced by FSD is poor and the IND demonstration may be treated as noise. We leave the further analysis of demonstrations Sec- tion 5.1 and GID exploration in Section 5.2.
Qualitative Analysis
For GID, ChatGPT is significantly weaker than fine-tuned model both IND and OOD metrics. According to Table 2, on average in three scenarios, the optimal ChatGPT method is weaker than the optimal fine-tuned method by 17.37% (IND ACC), 20.56% (OOD ACC), and 23.40% (ALL ACC), respectively. We believe this is because ChatGPT is pre-trained on large-scale general training data, which makes it difficult to perform better than fine- tuned models on specific domain data.
5.1 In-context Learning from IND to OOD
In Section 4.4, we find that the IND prior does not bring positive effects to OOD tasks. To further explore the influence of different types of demon- strations, we compare the performance of OOD tasks under three demonstration strategies: W/O Demos, IND Demos, and OOD Demos. Specif- ically, for OOD discovery, we perform 15 OOD classes clustering; for classification, classification under 10 classes, where the OOD in- tent used the ground-truth intent set to avoid the influence of pseudo intent set.
test OOD
(3) Compare different ChatGPT methods For OOD discovery, DC generally achieves the best performance, while ZSD is slightly inferior, and FSD performs the worst. Although DC is slightly inferior ZSD in the 1ND/OOD=3:1 scenario, it significantly outperforms other ChatGPT methods in the other two scenarios. FSD almost performs the worst among the three methods. ZSD pro- vides additional prior knowledge IND categories, while FSD provides labeled IND samples as con- text. However, more IND priors actually lead to worse performance for ChalGPT.
The results are reported in Fig 4. Compared with W/O Demos, OOD Demos achieve signifi- cant performance improvement in both clustering and classification tasks. In contrast, IND Demos result in performance decline in clustering and almost no improvement classification. For OOD Demos, it can considered that the demonstra- tion and testing are of the same distribution, so ChatGPT can improve task performance through in-context learning. For IND Demos, the differ- ent distribution between demonstration and testing causes ChatGPT not only unable to bring perfor- mance gains through in-context learning but also regard demonstrations in-context noise that inter- feres with task performance. This shows that the distribution of demonstration text has great impact on the effect of in-context learning, as
For GID, GID-FSD performs best on IND clas- sification, while GID-DC performs best on OOD intents. Comparing GID-ZSD and GID-DC, the difference lies in the pseudo-intent set used. GID- ZSD is on average 6.22% (ALL ACC) behind GID- DC, indicating the importance of the pseudo-intent set. For GID-FSD, due to the IND demonstration, IND classification ability is significantly improved

---

CCC aasoveetyy
Dixceery
12
10
Take
Figure The impact of different intent sets on GID. The pscudo-intent set comes from ChaGGPT(GID-SD)
Figure 6: The incorrect recall ratio of ChatGPT. We average the statistics of three ChatGPT methods.
also mentioned in (Min et al., 2022). It should be noted that since both IND and OOD classes come from the banking domain, the fine-tuned model can improve OOD task performance through knowl- edge transfer from IND samples. However, this fails on ChatGPT, indicating that current in-context learning of LLMs lacks deep mining and transfer demonstration knowledge (i.e., from IND to OOD) capabilities.
100
80
60
NMI
ChatGPT|DC)
ChatGPTIZSD)
ChatGPT(FSD)
DeepAlignes
5.2 Reasons for limiting GID performance
DKT
Since ChatGPT performs GID in pipeline man- ner, we analyze ChatGPT's performance separately in generating pseudo intent sets and performing joint classification. We show a set of pseudo intent sets Table can be seen that the three sets of pseudo intents are roughly similar to the real intents in semantics but show some randomness in granularity. Taking intent of ID 5 as an exam- ple, run-1 explains the payment method as "google pay apple pay", which is consistent with the granularity of the real label; run-2 expands it to "different methods", and run-3 further broadens to "payment related", with coarser granularity.
3
10
15
20
Number Samples Per Class
Figure 7: The impact of the number of samples in each cluster on OOD discovery under IND/OOD=3:1.
unique problem of generative LLMs. Fig shows the statistics of ChatGPT's incorrect recall. For OOD discovery, as the OOD ratio (clustering num- ber) increases, the proportion of missing recall and repeated recall both increase significantly. For ex- ample, under IND/OOD=1:1, the probability of missing and repeated recall reaches 24.15% and 4.44% respectively, which has seriously damaged task performance. Since clustering tasks require inputting all samples into ChatGPT simultaneously, more samples bring more difficult task understand- ing and processing to ChatGPT, resulting higher incorrect recall rates. For GID, the proportion of incorrect recall is almost unaffected by the OOD ratio, as GID is performed on a sample-by-sample basis. Furthermore, find that incorrect recall on GID is mainly due to the lack of domain knowledge. This results in ChatGPT being unable clearly identify intent set boundaries and may proactively allocate a query to multiple intents or refuse to allocate the query to predefined intent sets. As mentioned in Scction 4.3, ChatGPT has the problem of missing and repeated recall, which is a
Next, we use the OOD ground-truth intent set to replace the OOD pseudo intent set and further add artificial descriptions for each IND&OOD in- tent, as shown in Fig 5. Compared with using pseudo intent set, using the ground truth intent set can only bring slight performance improvement, while adding intent descriptions can significantly improve classification performance. This shows that the main reason for limiting ChatGPT's fur- ther improvement in GID task is the lack of domain knowledge, and the secondary reason is the quality of the pseudo intents.
5.3 Recall Analysis
"We provide relevant cases in Appendix E.

---

ID Ground Truth Intent
Pseudo Intent (run-1)
Pscudo Intent 2 (run-2)
Pscudo Intent (run-3)
card, delivery estimate
Inquiress shout delivery time schadule
Delivery and shipment related questions.
Delivery related inquiries
Requests for cancellation, urgent card
Transaction cancellation
2 cancel irammer
needs, transaction roversion
reversion rolated queations.
Accoomb/card related inquiries
3
Inquires about idemity verification process
Identity verification related questions
Identity verification inquiries
Questions related fees charged for transactions or withdrawals
Fee related questions, especially related cash wilbdrawals.
4
Fee related inquiries
Issues related lopping up accounts with mobile payment services snch as Google Pay Apple Pay.
questiions related lop-ups and adding money to an account using different methods.
5
apple_pay pay. par_eppya_rr___eeae www.ggglee_ppp
Payment related inquiries
Table 3: Pseudo-intens set generated by three random runs of ChaGPT(GGD-DC) under IND/OOD=3:1.
3XDI00D&:11
IND:OOD-1:1
OOD Discovery
GID
A
K(Prol) Enor
K(Pred)
11
Fror
K(Pred)
11
Frve
Model
ACC
NMI
ARI
ACC:
IND
GOD
ACC
ALL.
ACC:
ChaGGT::DC
DeepAligned! -3)
16
14
23
16
text-davinci-002
38.00
39.11
9.577
31.56
28.67
30.40
text-davinci-003
71.33
72.12
49.84
59.56
63.33
61.07
The results of estimating the number of clusters, where K' is hyperparameter for DeepAligned. The real number of clusters are 5, 10,1,5, respectively.
Table 4:
Claude
70.00
84.27
62.24
56.67
70.00
62.00
ChaiGPT
78.00
78.20
55.32
68.44
75.33
70.17
Table 5: Comparison of different LLMs.. All LLMs use DC and GID-DC methods under IND/OOD=3:2.
5.4 Effect of Cluster Sample Number
perparameter can achieve good results, but the ro- bustness of the hyperparameter is poor. Therefore, ChatGPT is more suitable for estimating a small number of clusters.
We explore the effect of the number of clustered samples on ChatGPT by changing the ground-truth number of each OOD intent. As shown in Fig 7, ChatGPT has poor robustness to the num- ber of samples. The clustering performance first reaches optimal effect between 5 and 10 sam- ples per class, and then drops rapidly. In contrast, discriminative fine-tuned methods exhibits good robustness. We believe this is because when there are too few samples, it's difficull for ChatGPT to discover clustering patterns. And when there are too many samples, ChatGPT needs to process too many samples at the same time, leading to more difficult clustering.
5.6 Comparison of Different LLMs
In this section, we evaluate the performance of other mainstream LLMs and compare them with ChatGPT. Text-davinci-002 and text-davinci-003 belong to InstructGPT and text-davinci-003 is an improved version of text-davinei-002. Compared with GPT-3, the biggest difference of Instruct- GPT is that it is fine-tuned for human instructions. In addition the GPT family models, also evaluate a new LLM Claude developed by An- thropic As shown in Table 5, ChatGPT performs better than text-davinci-00s and text-davinei-O00 because ChatGPT is further optimized based on text-davinci-003: Claude shows competitive per- formance with ChatGPT on OOD discovery, but is weaker on GID. In addition, we try to evaluate GPT-3 (davinci) and find that GPT-3 fails to per- form the tasks, which illustrates the importance of instruction tuning.
5.5 Estimate the Number of Cluster K
In the experiments above, the number of OOD classes assumed be ground truth. How- ever, in real-world applications, the number of OOD clusters often needs to be estimated auto- matically. We use the same estimation algorithm DeepAligned as a baseline following (Zhang et al., 2021; Mou et al., 2022a). For ChatGPT, we re- move the ground- truth cluster number from the prompt and count the estimated cluster number based the result. The results are reported in Table 4. When the number of clusters is small, ChatGPT can obtain more accurate estimates. How- ever, as the number of clusters increases, ChatGPT performs worse than the baseline and is prone to overestimate. For the baseline, an appropriate hy-
5.7 Effect of Different Prompt
Thorough prompt engineering is crucial to mitigate the variability introduced by different prompts. To address this, we devise three additional variations
hhups://platfrm wwwpoop..../.oco///eeeoooooo com/docss/ooells hhtps:/wwww.athrppp.coo//////ppp

---

Prompt/Baseline
0OD Discovery
GID
6.2 Semantic understanding of specific domains
ACC
NMI
ARI
IND
00D
ACC
ACC
Original
58.22
28:21
62.00
Paraphrase
58.67
66.80
30.10
69.33
62.00
65.67
In Section 5.2, find that the main reason for the limited performance of LLMs on GID is the lack
verbosity
60.89
68.74
34.85
68.67
60.67
64.67
Simplification
53,78
64,20
25.26
65.33
58.67
62.00
Averuge
$7.89
66.2
29.50
67.44
60.83
64.14
of semantic understanding of specific domains, To improve the performance of general LLMs in spe- cific domains, one approach through fine-tuning LLMs, which often requires high training costs and hardware resources. Another approach is to inject domain knowledge into prompts to enhance LLMs. Section 5.1 and 5.2 show that providing demonstration examples describing label sets can significantly improve performance. but long prompts will increase the inference cost of each query. How to efficiently adapt LLMs to specific domains without increasing inference costs is still an area under exploration.
Dospaligned(-Cil -GID)
74.67
80.83
55.35
91.50
77.33
84.42
Table 6; Results on different prompt of DC/GID-DC
under IND/OOD=1:1.
(Paraphrase, Verbosity, Simplification) for Chat- GPT (DC/GID-DCC beyond the original prompt and conduct experiments with an IND/OOD ratio 1:1.7 The results are shown in Table 6 In sum- mary, different prompts led to slight fluctuations in the experimental outcomes, but the results still sup- port existing conclusions. For example, the results of ChatGPT(DC) are still lower than the baseline Deepaligned.
6.3 Cross-domain in-context learning
6 Challenge & Future Work
In some practical scenarios, such as the need to perform new task expand business scope, there is often a lack of demonstration examples directly related to the new task. We hope to improve the performance of new tasks by leveraging previous domain demonstrations. However, previous exper- iments show that cross-domain in-context learn- ing has failed in current LLMs. meaningful but challenging question is how in-context learning with IND demonstrations performs well in OOD tasks? A preliminary idea is to use manual chains of thought to provide inference paths from IND demonstration samples to the labels, thereby pro- ducing more fine-grained domain domain-specific knowl- edge. These fine- tne-grained intermediate knowledge may help generalize to OOD tasks.
Based on above experiments and analysis, we sum- marize three challenging scenarios faced by LLMs and provide guidance for the future.
6.1 Large scale Clustering
Experiments show that there are three main rea- sons why LLMs is limited in performing large- scale clustering tasks: (1) The maximum length of input tokens limits the number of clusters. (2) When the number of clusters increases, LLMs will have serious recall errors. (3) LLMs have poor robustness to the number of cluster samples.
There have been some work attempts to solve the sequence length constraints of transformer- models, such as (Bertsch et al., 2023). Another feasible approach is to summarize the samples into topic words before clustering. For recall problem, post-remediation method is first screen the sample index of the recall error after cluster- ing, and then prompt LLMs complete or delete the sample allocation through multiple rounds of dialogue. For robustness, a possible method is to first estimate the optimal number of cluster, select a small portion of seed samples from the original sample set and cluster them, and then classify the remaining samples into seed clusters.
7 Conclusion
In this paper, we conduct comprehensive evalu- ation of ChatGPT on OOD intent discovery and GID, and summarize the pros and cons of Chat- GPT in these two tasks. Although ChatGPT has made significant improvements in zero or few-shot performance, our experiments show that ChatGPT still lags behind fine-tuned models. In addition, we perform extensive analysis experiments to deeply explore three challenging scenarios faced by LLMs: large-scale clustering, domain-specift understand- ing, cross- in-context learning and provide guidance for future directions.
"These prompt are also shown in Appendix D.

---

Limitations
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint 22Xiv:2004.02311.
In this paper, we investigate the advantages, dis- advantages and challenges of large language mod- els (LLMs) in open-domain intent discovery and recognition through evaluating ChatGPT on out-of- domain (OOD) intent discovery and generalized intent discovery (GID) tasks, Although we conduct extensive experiments, there are still several direc- tions to be improved: (I) Given the paper's focus on large-scale language models like ChatGPT, it's worth noting that ChatGPT is only accessible for output, which makes it challenging thoroughly investigate and analyze its internal workings. (2) Although we perform three different data splits for each task, they all come from the same source dataset, which makes their intent gramularity consis- tent. The analysis of different intent granularity is not further explored in this paper. (3) Although we ensure that all experiments on ChatGPT in this pa- per are based on the same version, further updates of ChatGPT may lead to changes in the results of this paper.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova, 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing, arXiv preprint w:xiv:880.008000
Guanting Dong, Daichi Guo, Liwen Wang, Xuefeng Li, Zechen Wang. Chen Zeng, Keqing Hc, Jinzheng Zhao, Hao Lei, Xinyue Cui, Yi Huang, Junlan Feng, and Weiran Xu. 2022a. Pssat: A perturbed se- mantic structure awareness transferring method for cetturbation-robuue slot filling.
Guanting Dong, Jinxu Zhao, Tingfeng Hui, Daichi Guo, Wenlong Wang. Boqi Feng, Yueyan Qiu, Zhuoma Gongque, Keqing He, Zechen Wang, and Weiran Xu. 2023. Revisit input perturbation problems for llms: A unified robustness evaluation framework for noisy slot filling task. In Natural Language Processing and Chinese Computing, pages 682 694, Cham, Springer Nature Switzerland.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhi- fang Sui. 2022b. A survey for in-context learning. arXiv preprint (txxi:2301.002...
References
Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew R Gormley. 2023. Unlimiformer: Long- range transformers with unlimited length input. arXiv preprint ar5iv:2305.01625.
Simon Frieder, Luca Pinchetti, Ryan- Grif- fiths, Tommaso Salvatori, Thomas Lukasiewiez, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023. Mathematical capabilities of chatgpt. arXiv preprint arXiv:2301.1386)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al, 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Daichi Guo, Guanting Dong, Dayuan Fu, Yuxiang Wu, Chen Zeng. Tingfeng Hui, Liwen Wang. Xuefeng Li, Zechen Wang, Keqing He, Xinyue Cui, and Weiran Xu. 2023. Revisit out-of-vocabulary problem for slot filling: A unified contrastive framework with multi- level data augmentations. ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5,
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. 2018. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pages 132-149.
WX Jiao, WX Wang, JT Huang, Xing Wang, and ZP Tu. 2023. Is chatgpt a good translator? yes with gpt-4 as the engine. arXiv preprint arXiv:2301.087.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. 2020. Unsupervised learning of visual features by contrast- ing cluster assignments. Advances in neural informa- tion processing systems, 33:9912-99244
Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quar- terly, 2(1-2))83-99 97.
Stefan Larson, Anish Mahendran, Joseph J. Peper. Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjaa Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of- scope prediction. In Proceedings of the 2019 Confer- ence Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
lñigo
Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulié. 2020. Efficient intent detection with dual sentence encoders. arXiv preprint 20v:2000.00800
Qian Chen, Zhu Zhuo, and Wen Wang. 2019. Bert for joint intent classification and slot filling. arXiv preprint arXiv: 1902. 10909.

---

pages 1311-1316, Hong Kong, China, Association for Computational Linguistics.
Short Papers), pages 46-53, Dublin, Ireland, Associ- ation for Computational Linguistics.
Shanglin Lei, Guanting Dong, Xiaoping Wang, Keheng Wang, and Sirui Wang. 2023. Instructerc: Reform- ing emotion recognition in conversation with re- trieval multi-task Ilms framework, arXiv preprint arXiv:2309. 11911.
Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. 2023. Evalu- ation of chatgpt as a question answering system for answering complex questions. arXiv preprint arXiv:2303.0799 077992.
Ting-En Lin, Hua Xu, and Hanlei Zhang. 2020. Dis- covering new intents via constrained deep adaptive clustering with cluster refinement. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8360-8367.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Tiiic Hambro, Faisal Azhar, et 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1397.
Jiachi Liu, Liwen Wang, Guanting Dong. Xiaoshuai Song. Zechen Wang. Zhengyang Wang. Shanglin Lei, Jinzheng Zhao, Keqing He, Bo Xiao, and Weiran Xu. 2023. Towards robust and generalizable training: An empirical study of noisy slot filling for input pertur- bations.
Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng. Yidong Wang, Linyi Yang, Hao- jun Huang, Wei Ye, Xiubo Geng, et al. 2023, On the robustness of chatgpt: An adversarial and out-of-distributiin perspective. arXiv preprint arXiv:3302.12095.
MacQueen. 1965. Some methods for classification and analysis of multivariate observations. In Proc. Sth Berkeley Symposium on Math., Stat., and Prob. page 281.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. 2021. Generalized of-ot-diistrbuttinn detection: A survey. arXiv preprint arXiv:21 10, 11334.
Kamil Malinka. Martin Perešíni, Anton Fire, Ondfej Hujńáá, and Filip Januš. 2023. On the educational impact of chatgpt: Is artificial intelligence ready obtain university degree? arXiv preprint
Weihao Zeng, Keging He, Zechen Wang, Dayuan Fu, Guanting Dong, Ruotong Geng, Pei Wang, Jingang Wang, Chaobo Sun, Wei Wu, and Weiran Xu. 2022. Semi-supervised knowledge-groune for task-oriented dialog systems. In Proceedings of the Towards Semi-Supervies and Reinforced Task- Oriented Dialog Systems (SereTOD), pages 39-47, Abu Dhabi, Beijing (Hybrid). Association for Com- putational Linguistics.
pre-training
arXiv:2303.114 11146.
Sewon Min, Xinxi Lyu, Ari Holtzman. Mikel Artetxe, Mike Lewis. Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages I 11048-11 1948-110664, 1064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Hanlei Zhang, Hua Xu, Ting-En Lin, and Rui Lyu. 2021. Discovering new intents with deep aligned clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14365-14373.
Yutao Mou, Keqing He, Pei Wang, Yanan Wu, Jingang Wang, Wei Wu, and Weiran Xu. 2022a. Watch the neighbors: unified k-nearest neighbor contrastive learning framework for OOD intent discovery. In Proceedings of the 2022 Conference Empirical Methods Natural Language Processing, pages 1517-1529, Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguistics.
A Datasets
The original dataset Banking contains 77 classes and is class-imbalanced. Its training set has 9003 samples, the validation set has 1000 samples, and the test set has 3080 samples. The average token length of the samples is 1 11.91 with the longest being 79, We show 15 IND classes and 15 OOD classes sampled from the 77 class in Table 7,
Yutao Mou, Keging He, Yanan Wu, Pei Wang. Jingang Wang, Wei Wu, Yi Huang, Junlan Feng, and Weiran Xu. 2022b. Generalized intent discovery: Learning from open world dialogue system. In Proceedings of the 29th International Conference on Computational Linguistics, pages 707-720, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
B Implementation Details
For all baselines, we use the pre-trained BERT model (bert-base-uncased,,, with 12-layer trans- former) as the backbone, and freeze all but the last transformer layer parameters to achieve better performance and speed up the training procedure as suggested in (Zhang et al., 2021). In addition,
Yutao Mou, Keqing He, Yanan Wu, Zhiyuan Zeng. Hong Xu, Huixing Jiang, Wei Wu, and Weiran Xu. 2022c. Disentangled knowledge transfer for OOD intent discovery with unified contrastive learning. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 2:

---

IND Class
0OD Class
E Cases of Incorrect Recall
pin, Blocked
card delivery cstimee
not updated
pending cand cand__aammmee payment
verify. identity
We provide cases of incorrect recall in OOD dis- covery and GID in Figures 11 and 12, respectively.
verify source funds
withdrawhl .charge
disposable_ccm card limits
applc_pay hrw__or_ooppc_pgee
cand_about_n to sapire
pending_Jog lop_up
direct, debit, payment, resogniedd
request refund
top_up failed
card. linking
card payment feenent_ee_ccaaaree
trunsfer_nt received herpvdddby_reeiiiieee recipient
card, anival
declined, card, payment
card payment pott_ reangnised
ger disposable virmal, card
uctivate_my my .curd
cunduacepianee
cunsfer_timiing
get physical_cmnt cand
getting spare
contactioss, not working
compromisod_cco card
Table 7: Sampled 15 IND and 15 OOD classes for the experiments. For IND/OOD=3:1 and IND/OOD=3:2, the first and 10 OOD classes used respectively.
DOD Discovery
Mathod
ACC
NMI
ARI
Melhed
ACC:
ACC:
ArT:
ALL
DeepAligned
55.27
DeepAligned
DKT
91.78
96.97
95.16
ETE
99.11
97.78
98.44
ChanGPIDDC)
80.89
84.02
62.70
86.00
82.67
81.33
ChaGGPTFFD)
56.00
64.15
26.32
ChaGPEGGID-FSD)
73.33
Table 8: The reults on CLINC under IND/OOD=1:1.
we keep the hyperparameters consistent with those in the official open-source code of all baselines. For ChatGPT-based methods, we perform the ex- periments by calling OpenAl's official API, and the version of ChatGPT we used is gpt-3.5-turbo- 0301. For hyperparameters such as temperature in the API, we keep the default value of OpenAl un- changed. To reduce randomness, we average results over three random runs for all methods of ChatGPT and baselines.
C Results the CLINC Dataset
Furthermore, we undertake an exploration of ex- perimental outcomes utilizing an alternative widely employed dataset, CLINC (Larson et al., 2019), in an IND/OOD ratio of 1:1, as presented in Tabel 8. Despite the disparate label granularity and domain between CLINC and Banking Datasets, the experi- mental results from CLINC align with and bolster the generalizability our conclusions drawn from the Banking dataset.
D Details on Prompts
As we perform the GID task on ChatGPT in the manner of a pipeline, list the complete prompts we used in each stage of the three GID methods in Table 9. addition, we present the prompt variants ChatGPT(DC) in Table 10.

---

Method
Stage
Prompt
Next I will first give you a set sentences, which will be recorded as Set 1. First, please classify the sentenees into (Number of Clasters) caregories according their intentions. You only to outpat the category numher and the corresponding sentence number in the following format:
GID-DC
Catgary 1.2.34455
Then, you noed to summarize intent of each class from Category Category 5.
Set I () can chouse date for delivery? ......
Below predefined of intent culegories, recorded as Set 1:1. pin_ blocked...... Please classify following sentence into Set according to its intention, just response corresponding category number-ctest sample>
NeaL I will lirst give you a set of intent categurries which will be recorded as Set 1. Then will give you snother set sentences without intention labels, recorded as You lizst need learn knowledge in Set use the leamned lowgedge to classify the sentences in Set 2 into 5 (Number of Clusters) categories according to their intentions. You only need ourput category number and the corresponding Cutegory I: 1.2,3.4.5
number in the following format:
GID-ZSD
should he noted that the intention in set and the intention in set do not overlap. Then, you need summarize intent of each class from Category Category 5. Set I( y pin, blocked:
Set 20 D y: 1. can I choose date for delivery? ......
Relow a predefined set of intent categories, recorded as Set I:1. pin blockad...... Please clussily the following senience into Set according its intention, just respouse the corrsponding calegury number ctest sample>
2
Next, will first give you set sentences with intention another ser of sentences without intention lahels, recorded as Set You first need to leam the knowledge in Set I, and then use learned knowledge classily sentences in into (Number Clusters) calegories according their intentions. You only need to ourput the category number and the corresponding sentence number in the following format: Category 1,2,3,4.5...
which
recorded
Then will give you
GID-FSD
should be noted that the intention in set and the intention in set do not overlap. T Then, you need to summarize the intent of each class from Category to Category 5. (D Set (D Jemg): I How do unblack my card? intention:pin blocked; .............. Set (DDBP)
choose date delivery? ......
Next, will first give you predefined set intent categories, which will be recorded as Set Then will give you another of sentences with intention lahels, recorded as Ser 2.
2.
Set I:1. pin blocked.......
Set 2:sentence: How unblock my card? intention pin blocked; ..... You first need to leam the knowledge in Set 2, and then use the learned knowledge to classify the following sentence into Set according intention, just response corresunning calegory: sample>
Table 9: The complete prompts used in the three GID methods.
Type
Prompt
Next, I will first give you a set of sentences, which will be recorded as Sct Plcase classify the sentences in Set into categories according to their intentions. You only need output the category number and the corresponding sentence number the following format:
Original
Category 1: 1,2,3,4.5
I will provide you with collection of sentences, noted Set 1. Your lack is calegurize the sentences Set into distinel groups based on their underlying intentions. Your output should include the calegory number along with the corresponding sentence number, formatted as follows:
Paraphrase
Category 1: 1, 2, 3, 5, and
Next, be presenting you with a compilation sentences, collectively labeled "Set 1", Your task to categorize these sentences into distinct groups according to their underlying intentions, Upon completing the task, your response is anticipated to take the form of structured enumeration. Your response should consist of the assigned category number along with the respective sentence numbers following this format: Category 1: I. 2. 5...
Verbosity
Next, I'll provide sentences number and sentence number in this format:
Please categorize them into groups based on intentions, Output the category
Simplification
Category 1: 2, 4,
Table 10: The three prompt variants of ChatGPT(DC).

---

Missing Recall
Repeated Recall
Input
Response
Category 1: 2, 10, 30, 31, 50
Category 2: 4, 25, 27, 34, 44
Category 3: 5, 6, 15, 16, 36
Please classify sentences in into categories
Category 4: 1,
20,26.29,
Category 5: 7, 27
35,40,41,
2.27
according their intentions.
42,46,48
Category 6: 8, 18, 28,
Category 7: 9, 22, 32, 39,
Category 8: 11, 12,
Category 9: 13, 23, 33,
Category 10: 14, 19, 24, 37, 45,
Table 11: Cases of incorrect recall in OOD Discovery. The index marked with an underline denotes the repeatedly recalled sample.
Type
Input
Response
Ground-Truth Intent
How many days until the money will be in account?
Missing Recall
It not fit any category Set 1.
transfer, transfer_timing (IND)
The senlence does not match any intents in as it is not related to issues with payment management. could potentially under "purchase delivery inquiry" category.
would cancel payment. purchased something several days ago and still have not received
request_refund refund (OOD)
top, up failed. Note: sentence could under 3. pending__care parmm_p or 11.
don't know where this transaction
Lud__payment_nee _no recognised (IND)
Repeated Recall
came from?
card payment not recognised, depending on the context of the conversation.)
12. activate activate_my_cam mard: Issues related to card payme
My credit card was declined.
declined card payment (OOD)
Table 12: Cases of incorrect recall in GID.


Reviews:
[{"review_id": "Xh12DRZj3p", "reviewer": "Reviewer_s6Wk", "summary": "", "questions": "A. What's the variance of results across repeated runs in Table 1? \n\nB. In section 4.4 (2) \"compare ChatGPT with Finetuned BERT\", which rows in Table 1 display these results? There's only one \"BERT\" row which seems to be under the zero-shot setting.\n\nC. To what extent do your main findings apply to models other than ChatGPT (in Table 5)?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and valuable feedback on our manuscript. We are encouraged by your recognition, especially since you believe that this paper presents some intriguing experimental observations, comprehensive qualitative analysis, and clear writing. We sincerely apologize for any unclear presentation and hope this response can resolve your concerns.\n\nQ1: The authors provide conjectures for some of these observations, but didn't verify them.\n\nA1: Given the paper's focus on large-scale language models like ChatGPT, it should be noted that ChatGPT is only available for output, which makes it challenging to thoroughly investigate and analyze its internal workings. Nevertheless, we are committed to conducting experiments and  analysis from an external perspective. We have put forth our best efforts to design experiments rigorously and to seek possible explanations by comparing outputs in different scenarios, aiming to understand the reasons behind the model's behavior. For instance, regarding the issue of in-domain demonstrations leading to lower performance, we provide an initial explanation in section 4.4(3) and  further summarize underlying reasons through comparative experiments and connections to prior research [1] in section 5.1. We fully comprehend your expectations for more in-depth explanations. With recent open-sourcing of LLMs like llama2-70B, we're enthusiastic about delving into internal mechanisms for future studies.\n\n[1] Rethinking the role of demonstrations: What makes in-context learning work?\n\nQ2: There is no variation of prompt phrasing and exemplar choice. \n\nA2: We value your feedback and recognize that introducing these variations can further enhance the generalizability of the research findings. We devise three additional variations (Paraphrase, Verbosity, Simplification) for ChatGPT (DC/GID-DC) beyond the original prompt, and conduct experiments with an IND/OOD ratio of 1:1, as depicted in the results table below:\n\n|                   |       | Discovery |       |         | GID     |         |\n| ----------------- | ----- | --------- | ----- | ------- | ------- | ------- |\n| Prompt/Baseline   | ACC   | NMI       | ARI   | IND ACC | OOD ACC | ALL ACC |\n| Original          | 58.22 | 65.30     | 28.21 | 66.44   | 62.00   | 64.22   |\n| Paraphrase        | 58.67 | 66.80     | 30.10 | 69.33   | 62.00   | 65.67   |\n| verbosity         | 60.89 | 68.74     | 34.85 | 68.67   | 60.67   | 64.67   |\n| Simplification    | 53.78 | 64.20     | 25.26 | 65.33   | 58.67   | 62.00   |\n| Average           | 57.89 | 66.26     | 29.60 | 67.44   | 60.83   | 64.14   |\n| Deepaligned(-GID) | 74.67 | 80.83     | 55.35 | 91.50   | 77.33   | 84.42   |\n\nOn the whole, although the introduction of variations in prompt phrasing induces slight fluctuations in ChatGPT's performance, our conclusions remain robust, including observations such as \"ChatGPT underperforms compared to the baseline.\" Regarding exemplar choice, we examine the impact of different demonstration strategies in Section 5.1 and Figure 4. We will present the design of these prompt variations in the revised version.\n\nQ3: Only a single dataset is used in evaluation.\n\nA3: We recognize concerns about the universality of research outcomes from a single dataset. To address this, we undertake an exploration of experimental outcomes utilizing an alternative widely employed dataset, CLINC[2], in an IND/OOD ratio of 1:1, as presented below:\n\n|              |       | Discovery |       |                  |         | GID     |         |\n| ------------ | ----- | --------- | ----- | ---------------- | ------- | ------- | ------- |\n| Method       | ACC   | NMI       | ARI   | Method           | IND ACC | OOD ACC | ALL ACC |\n| DeepAligned  | 94.22 | 95.27     | 90.21 | DeepAligned-GID  | 98.67   | 94.22   | 96.44   |\n| DKT          | 97.78 | 96.97     | 95.16 | E2E              | 99.11   | 97.78   | 98.44   |\n| ChatGPT(DC)  | 80.89 | 84.02     | 62.70 | ChatGPT(GID-DC)  | 86.00   | 82.67   | 84.33   |\n| ChatGPT(ZSD) | 65.33 | 71.11     | 39.15 | ChatGPT(GID-ZSD) | 88.00   | 72.00   | 80.00   |\n| ChatGPT(FSD) | 56.00 | 64.15     | 26.32 | ChatGPT(GID-FSD) | 90.00   | 56.67   | 73.33   |\n\nDespite the disparate label granularity and domain between CLINC and Banking datasets, the experimental results from CLINC align with and bolster the generalizability of our conclusions drawn from the Banking dataset.\n\n[2] An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction\n\nQ4: No code or data is provided.\n\nA4: Although key information regarding the reproducibility of our results is presented in Figure 2  and Appendices A, B, and D of the paper, we sincerely apologize for not furnishing a more extensive compilation of data and code support. We hereby commit to supplying a comprehensive set of data and code subsequent to the conclusion of the blind review process, thereby ensuring facile reproducibility of our experimental outcomes by fellow researchers.\n\nQ5: The variance of results in Table 1.\n\nA5: In Table 1, we conduct three repeated experiments to mitigate the errors induced by randomness. The table below presents Table 1 with variance values:\n\n|              |              | IND:OOD=3:1  |              |              | IND:OOD=3:2 |             |             | IND:OOD=1:1 |             |\n| ------------ | ------------ | ------------ | ------------ | ------------ | ----------- | ----------- | ----------- | ----------- | ----------- |\n| Method       | ACC          | NMI          | ARI          | ACC          | NMI         | ARI         | ACC         | NMI         | ARI         |\n| BERT         | 52(0)        | 41.58(0)     | 15.36(0)     | 36(0.89)     | 42.33(0.99) | 5.616(0.95) | 29.33(0)    | 45.26(0.31) | 2.499(0.19) |\n| ChatGPT(DC)  | 88(3.27)     | 84.62(5.04)  | 73.36(7.33)  | 78(3.27)     | 78.2(2.54)  | 55.32(5.58) | 58.22(2.26) | 65.3(0.71)  | 28.21(1.83) |\n| Deepaligned  | 100(0)       | 100(0)       | 100(0)       | 78.67(0.94)  | 82.18(0.93) | 61.01(2.08) | 74.67(0)    | 80.83(0.14) | 55.35(0.64) |\n| DKT          | 93.33(2.49)  | 91.71(5.52)  | 84.82(2.44)  | 80.67(0.94)  | 82.59(1.93) | 64.53(3.79) | 76.45(1.66) | 83.23(1.49) | 60.92(2.25) |\n| ChatGPT(ZSD) | 92(5.66)     | 87.25(9.02)  | 80.16(12.21) | 67.33( 3.40) | 68.72(2.81) | 39.32(8.00) | 50.67(6.06) | 60.73(5.19) | 21.25(8.17) |\n| ChatGPT(FSD) | 74.67(10.50) | 64.77(11.62) | 45.92(13.73) | 56.67(3.40)  | 63.56(2.30) | 31.47(3.98) | 49.78(1.26) | 60.98(0.91) | 20.72(1.12) |\n\nWe observe that the ChatGPT methods exhibit higher variance compared to the baselines. This phenomenon could be attributed to the inherent characteristics of language generation in LLMs.\n\nQ6: Which rows in Table 1 display the “Finetuned BERT” results?\n\nA6: The term 'fine-tuning BERT' refers to two baselines with IND prior, namely DKT and DeepAligned.\n\nQ7: The extensibility of findings in other models.\n\nA7: We appreciate and value your questions regarding the extensibility of our findings to other models. Given our paper's primary focus on ChatGPT, we omitted some model results in certain experiments due to space and research priorities. However, as one of the forefront advanced large-scale models currently available, ChatGPT to a significant extent can serve as a representative of contemporaneous LLMs. Additionally, we present and analyze the performance of text-davinci-002, text-davinci-003, and claude models in Section 5.6 and Table 5. In forthcoming revisions, we intend to supplement the experimental outcomes of additional models in the appendix, thereby ensuring the comprehensiveness and applicability of the study.\n\nQ8: Typos Grammar Style And Presentation Improvements.\n\nA8: We will enhance our grammar and expression in the revised version based on your feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents some intriguing, and sometimes counterintuitive, behavioral test observations, e.g. in-domain demonstrations can actually lead to poorer performance than having no demonstrations at all when tested on OOD samples. \n- The paper conducts a quite comprehensive qualitative analysis, covering a wide range of aspects, including an ablation study, the effect of LMs,  recall analysis, etc.\n- The presentation is clear and relatively easy to follow.", "reasons_to_reject": "- While the paper presents some interesting behavioral test observations, I would love to see a deeper exploration into the underlying reasons for them. For instance, why do in-domain demonstrations result in subpar performance compared to no demonstrations? The authors provide conjectures for some of these observations, but didn't verify them. Therefore, the current takeaways of the paper look more on the shallower side and don't really look particularly insightful.\n- There is no variation of prompt phrasing and exemplar choice. It will be interesting to see how generalizable the current findings are to such variations.\n- Only a single dataset is used in evaluation, which may raise questions about the generalizability of the findings.\n- No code or data is provided, which can make it hard to reproduce the results.", "questions_to_authors": "A. What's the variance of results across repeated runs in Table 1? \n\nB. In section 4.4 (2) \"compare ChatGPT with Finetuned BERT\", which rows in Table 1 display these results? There's only one \"BERT\" row which seems to be under the zero-shot setting.\n\nC. To what extent do your main findings apply to models other than ChatGPT (in Table 5)?", "paper_topic_and_main_contributions": "The paper delves into the evaluation of ChatGPT's capabilities in out-of-domain (OOD) intent discovery and generalized intent discovery (GID). The authors have found that while ChatGPT exhibits decent performance in zero-shot scenarios, it doesn't quite match up to finetuned models. Through a series of analysis experiments, the paper sheds light on the strengths and challenges associated with ChatGPT on these tasks.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Some parts of the paper reads a bit too dry, like a technical report, especially when the use of acronyms is too dense (like in Section 4.4 (3)). You could just call them \"zero-shot\" and \"few-shot\" instead of using the acronyms.\n\n- It would be good if Figures 4 and 5 can be made colorblind-friendly.\n\n- Line 8: \"some studies has been ...\"\n\n- Line 496: \"ChatGPT is difficult to discover ...\" -> \"It's difficult for ChatGPT to discover ...\" \n"}}, {"review_id": "KuTkNxkRXZ", "reviewer": "Reviewer_TNLF", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and valuable feedback on our manuscript. We are encouraged by your recognition of the first ChatGPT empirical study on two tasks,  the proposed heuristic prompts and  comprehensive qualitative analyses in this paper. We sincerely apologize for any unclear presentation and hope this response can resolve your concerns.\n\nQ1: More analyses and validations on prompts and the prompts for ZSD and FSD may potentially mislead ChatGPT.\n\nA1: I fully comprehend your concerns. Thorough prompt engineering is crucial to mitigate the variability introduced by different prompts. To address this, we devise three additional variations (Paraphrase, Verbosity, Simplification) for ChatGPT (DC/GID-DC) beyond the original prompt, and conduct experiments with an IND/OOD ratio of 1:1, as depicted in the results table below:\n\n|                   |       | Discovery |       |         | GID     |         |\n| ----------------- | ----- | --------- | ----- | ------- | ------- | ------- |\n| Prompt/Baseline   | ACC   | NMI       | ARI   | IND ACC | OOD ACC | ALL ACC |\n| Original          | 58.22 | 65.30     | 28.21 | 66.44   | 62.00   | 64.22   |\n| Paraphrase        | 58.67 | 66.80     | 30.10 | 69.33   | 62.00   | 65.67   |\n| verbosity         | 60.89 | 68.74     | 34.85 | 68.67   | 60.67   | 64.67   |\n| Simplification    | 53.78 | 64.20     | 25.26 | 65.33   | 58.67   | 62.00   |\n| Average           | 57.89 | 66.26     | 29.60 | 67.44   | 60.83   | 64.14   |\n| Deepaligned(-GID) | 74.67 | 80.83     | 55.35 | 91.50   | 77.33   | 84.42   |\n\nIn summary, different prompts led to slight fluctuations in the experimental outcomes, but the results still support existing conclusions, confirming their robustness. Regarding the issue you mentioned about prompts potentially misleading ChatGPT for ZSD and FSD, we extend our sincerest apologies. Throughout the experimental procedure, we explicitly underscored at the terminus of the prompts, \"It should be noted that the intention in set 1 and the intention in set 2 do not overlap.\" Regrettably, this point was inadvertently omitted in Figure 2. We acknowledge this oversight and will correct it in the revised version.\n\nQ2: The analyses are not conclusive to root cause the poor performance and some statement is too strong.\n\nA2: Given the paper's focus on large-scale language models like ChatGPT, it's worth noting that ChatGPT is only accessible for output, which makes it challenging to thoroughly investigate and analyze its internal workings. In Section 4.4(3) and 5.1, we externally analyze the model's output and find that the performance decline is due to \"discrepancies in distribution between demonstrations and tasks\" and \"the treatment of demonstrations as noise\". With recent open-sourcing of LLMs like llama2-70B, we're enthusiastic about delving into internal mechanisms for future studies. With regard to the assertion that \"current in-context learning of LLMs lacks deep mining and transfer demonstration knowledge capabilities,\" as you rightly pointed out, different approaches may yield different outcomes. In the revised version, we will employ a more balanced statement, emphasizing potential limitations of the specific transfer learning methods employed in the experiments, while simultaneously acknowledging the potential for success with alternative approaches.\n\nQ3: The challenges in real-world scenarios could be explained earlier.\n\nA3: In order to align with the sequence of previous work[1] [2], we conduct experiments involving a simplified configuration in Section 4.4 and explore more realistic scenarios in Section 5.5. We apologize for any inconvenience caused by the shortcomings in our explanations and will rectify this aspect in the revised version.\n\n[1] Discovering New Intents with Deep Aligned Clustering \n\n[2] Watch the Neighbors: A Unified K-Nearest Neighbor Contrastive Learning Framework for OOD Intent Discovery\n\nQ4: Typos Grammar Style And Presentation Improvements.\n\nA4: Thank you very much for your suggestions. We will enhance our grammar and expression in the revised version based on your feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper claims to be the first to comprehensively evaluate ChatGPT’s performance on OOD intent discovery and GID. Given the popularity of LLMs and the trend in industry to try ChatGPT in any NLP task, such evaluation against strong baselines and calling out the limitations could be very insightful for practitioners. \n- The heuristic prompts proposed can serve as reasonable starting points for practitioners who do want to adopt ChatGPT to handle OOD queries. \n- The qualitative analyses are informative in showing how sensitive ChatGPT can be w.r.t. prompts, the IND:OOD ratio, and the number of clusters.", "reasons_to_reject": "- It’d be good to see more analyses and validations on how the prompts from Section 3.2 are engineered. We know that ChatGPT can be very sensitive to prompts. Without seeing sufficient due diligence in prompt engineering, it’s difficult to conclude whether ChatGPT’s relatively poor performance in Section 4.4 is mainly due to suboptimal prompts or inherent limitations of LLMs. Reading Figure 2, it seems that the prompts for ZSD and FSD could potentially mislead ChatGPT into thinking that the test set should have the same set of intents instead of new ones. It could help explain the counter-intuitive result that ChatGPT is doing worse with more IND knowledge. \n- Related to the previous point, it’d be good to see more thorough investigations on why ChatGPT is doing worse with more IND knowledge. This topic is briefly covered in Section 5.1, but the analyses are not conclusive to root cause the poor performance, and the statement “current in-context learning of LLMs lacks deep mining and transfer demonstration knowledge capabilities” is too strong and not sufficiently supported (just because the specific transfer learning approach experimented didn’t work it doesn’t mean other approaches won’t). \n- The evaluation setting assumes that the OOD test set is composed of only OOD queries with a known number of new intents. It’s a simplified setting compared to real world scenarios where the test set could be a mixture of IND and OOD queries with an unknown number of new intents. Section 5.5 does address the issue of estimating the number of clusters, but the challenges in real world scenarios compared to the experiment setup could be explained earlier.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper mainly focuses on evaluating ChatGPT on out-of-domain (OOD) intent discovery and General Intent Discovery (GID). It proposes a few heuristic prompts using various levels of information from in-domain (IND) data as the context for ChatGPT, and benchmarks these variations against the more standard fine-tuned approaches. The experiments show that ChatGPT significantly outperforms BERT baseline without prior IND knowledge, but is generally weaker than baselines given IND knowledge. The qualitative analyses further dove into potential factors impacting ChatGPT. Lastly, the authors summarize the main challenges with performing clustering tasks with LLMs and give general suggestions to mitigate them.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 012 – “extend”\n- Table 2, last row – “FSD”\n- Line 515 – “an”\n- Line 499 – “leading to”\n- Line 588 – If “previous experiments” refer to the ones not covered by this paper, you should add some references to support this claim. \n- Line 814 – as a nitpick, you can’t completely “avoid randomness” by averaging over a few runs. You can “reduce” or “mitigate the impact of” randomness."}}, {"review_id": "rZBOb7NOt3", "reviewer": "Reviewer_x2rM", "summary": "", "questions": "- How are ground-truth labels on OOD samples determined? What criteria is used for this? \n- Did you check whether the categorizations by ChatGPT or GPT-4 also made sense and if it was more or less aligned with human preference compared to supervised baselines?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and valuable feedback on our manuscript. We are encouraged by your recognition of the first ChatGPT empirical study on two tasks, the challenges for LLMs in clustering and the comprehensive evaluation in this paper. We sincerely apologize for any unclear presentation and hope this response can resolve your concerns.\n\nQ1: LLMs like ChatGPT aren't designed for text clustering and its poor performance isn't exciting.\n\nA1: We acknowledge that the original design of LLMs such as ChatGPT was not intended for clustering tasks, leading to certain limitations in this regard. However, as general language models, we contend that investigating the capabilities of LLMs in clustering tasks still holds value. Our research has unveiled numerous intriguing phenomena and findings in the realm of clustering, with the subpar performance of ChatGPT in clustering being merely a fractional aspect of the study. Furthermore, in-depth explorations into the reasons behind the underperformance of LLMs is undertaken in Section 5.1 through 5.5, and a comprehensive summary of the challenges faced is provided in Section 6.1, along with guidance for prospective avenues of future research.\n\nQ2: The uselessness of LLMs for text clustering tasks because of high computational overhead and low efficiency.\n\nA2: We understand your concerns regarding the performance and efficiency of LLMs in large-scale data clustering. Indeed, LLMs may incur certain computational overheads when dealing with vast datasets; however, this does not imply that scalability is ineffectual for clustering purposes. Recent studies like [1] and [2] have shown LLMs excel in aiding \"user-perspective-based clustering\" and \"clustering granularity determination.\" These works also combine LLMs and small models to improve efficiency and performance. Since these are not within the scope of this paper's discussion, it is not elaborated upon herein. The focal point of our research resides in the empirical evaluation of ChatGPT's role in clustering (Section 4.4), a comprehensive analysis of the inherent drawbacks of LLMs in clustering (Section 5.1 to 5.5), and a holistic synthesis of the challenges faced along with proposed solutions (Section 6.1). By addressing these aspects, our endeavor aims to draw attention to overcoming these issues in future research endeavors.\n\n​\t[1] ClusterLLM: Large Language Models as a Guide for Text Clustering \n\n​\t[2] Goal-Driven Explainable Clustering via Language Descriptions\n\nQ3: Only very straightforward prompting, only ChatGPT-3.5 and few samples for testing.\n\nA3: We devise three additional variations (Paraphrase, Verbosity, Simplification) for ChatGPT (DC/GID-DC) beyond the original prompt, and conduct experiments with an IND/OOD ratio of 1:1, as depicted in the results table below:\n\n|                   |       | Discovery |       |         | GID     |         |\n| ----------------- | ----- | --------- | ----- | ------- | ------- | ------- |\n| Prompt/Baseline   | ACC   | NMI       | ARI   | IND ACC | OOD ACC | ALL ACC |\n| Original          | 58.22 | 65.30     | 28.21 | 66.44   | 62.00   | 64.22   |\n| Paraphrase        | 58.67 | 66.80     | 30.10 | 69.33   | 62.00   | 65.67   |\n| verbosity         | 60.89 | 68.74     | 34.85 | 68.67   | 60.67   | 64.67   |\n| Simplification    | 53.78 | 64.20     | 25.26 | 65.33   | 58.67   | 62.00   |\n| Average           | 57.89 | 66.26     | 29.60 | 67.44   | 60.83   | 64.14   |\n| Deepaligned(-GID) | 74.67 | 80.83     | 55.35 | 91.50   | 77.33   | 84.42   |\n\nIn summary, different prompts led to slight fluctuations in the experimental outcomes, but the results still support existing conclusions, confirming their robustness. \n\nRegarding the issue pertaining to the number of test models and test samples, we seek to clarify certain misconceptions. The primary subject of investigation in this paper is ChatGPT. While, due to considerations of both space and research focus, we have not presented results for all models in each analytical experiment, we compare and discuss the text-davinci-002, text-davinci-003, and claude models in Section 5.6. As for the query sample quantity, as expounded upon in lines 284-299, owing to the constraints imposed by ChatGPT's input length, we have had to sample 5-10 test samples for each class and a maximum of 30 classes. We deem this to be an acceptable and understandable approach.\n\nQ4: How are ground-truth labels on OOD samples determined? What criteria is used for this?\n\nA4: Following prior work [3] and [4], as well as in accordance with the descriptions in Section 4.1 and Appendix B, we partition a subset of samples from a labeled dataset Banking and utilize them as unlabeled OOD samples by masking their original labels. Consequently, the ground-truth labels for these OOD samples correspond to the original intent labels from the Banking dataset. Furthermore, as indicated in footnote 2, we establish the mapping between the predicted labels and the ground-truth labels using the Hungarian algorithm. \n\n​\t[3] Discovering New Intents with Deep Aligned Clustering \n\n​\t[4] Watch the Neighbors: A Unified K-Nearest Neighbor Contrastive Learning Framework for OOD Intent Discovery\n\nQ5: Did GhatGPT's categories align with human preferences better than supervised baselines?\n\nA5: We appreciate your inspiring insights that comparing LLMs with baselines from a human preference perspective. Based on experimental results, we take into consideration the following two aspects.  In terms of classification accuracy, when utilizing the label distribution of the dataset as a proxy for human preference, ChatGPT performs less effectively than the fine-tuned baselines (Section 4.4).  From the perspective of inductive clustering/categories, supervised baselines can only provide label indexes, whereas LLMs like ChatGPT can generate labels with specific semantics, aligning more closely with human preferences. Additionally, in Section 5.2, we conduct case studies of the quality of labels generated by ChatGPT. Overall, ChatGPT demonstrates remarkable semantic performance while exhibiting a certain level of granularity randomness.  Although the primary focus of this paper is not on human preferences, we acknowledge the significance of human preferences and do not disregard them. We are keen on delving further into this aspect in future endeavors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is the first to study ChatGPT for OOD Intent Discovery and General Intent Discovery task.\n- This paper point out several challenges about using ChatGPT for clustering tasks.\n- This paper provide comprehensive evaluation results of various setting.", "reasons_to_reject": "The contributions of this paper are somewhat limited. Firstly, LLMs like ChatGPT are not designed to excel at text clustering tasks. Finding out that ChatGPT does not perform well on clustering is not particularly exciting. Secondly, I remain unconvinced about the usefulness of LLMs for text clustering tasks, because using LLMs on large-scale data clustering is of high computational overhead and low efficiency.  Finally, only very straightforward prompting strategies are investigated in this paper, and only one model of ChatGPT-3.5 is tested, and only one dataset with 5-10 sampled queries are used for evaluation.", "questions_to_authors": "- How are ground-truth labels on OOD samples determined? What criteria is used for this? \n- Did you check whether the categorizations by ChatGPT or GPT-4 also made sense and if it was more or less aligned with human preference compared to supervised baselines?", "paper_topic_and_main_contributions": "This paper investigate the use of ChatGPT on OOD Intent Discovery and General Intent Discovery tasks. OOD Intent Discovery aims to cluster a group of OOD samples into multiple (usually pre-defined number of) categories. Experiments turn out that ChatGPT still underperform fine-tuned methods on both tasks. This paper also list some challenges of ChatGPT on the studied task, and suggest several future directions.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper evaluates ChatGPT for the tasks of out-of-domain intent discovery and generalised intent discovery. They show that while ChatGPT performs well in zero-shot settings. However, it still does not outperform finetuned models. 

The reviewers were positive about this work and appreciated that this is the first work to evaluate ChatGPT's capabilities in out-of-domain (OOD) intent discovery and generalized intent discovery (GID). The following concerns were raised by the reviewers:

1. Lack of variations in prompts which may affect the performance of ChatGPT - the authors address this concern partially in the rebuttal and try experiment with more variations in the prompts. 
2. Only one dataset was used - the authors have addressed this by providing additional results on other datasets
3. No thorough analysis of the results and no investigation on why ChatGPTs' performance is poor - the authors mention this is beyond the scope of the paper but I do not fully agree.
4. Models other than ChatGPT have not been tried - Given that CHatGPT was state of the art at the time this work was undertaken and considering the costs of experiments with multiple LLMs I think it okay to keep the scope limited to GPT.

Reviewer x2rM had raised some other concerns but those have been clarified. 

Overall, I find this paper to be interesting and worth publishing. I would request the authors to address the following in the final version:

1. Add all additional results shared during the rebuttal phase to the main body or appendix of the final paper, as appropriate.
2. Honor their commitment made during the response period of making the code, data and evaluation scripts publicly available.
3. Address some of the concerns related to investigating the reason for poor performance.
Decision: Accept-Main