ClusterLLM: Large Language Models as a Guide for Text Clustering
Authors: Yuwei Zhang
Zihan Wang
Jingbo Shang
Venue: EMNLP 2023 Main (2023)
Keywords: text clustering
large language model
sentence relation
entropy-based sampling
TL;DR: We utilize sentence relations to query instruction-tuned large language models, and then leverage the feedbacks to improve clustering quality and determine cluster granularity

Abstract: We introduce ClusterLLM, a novel text clustering framework that leverages feedback from an instruction-tuned large language model, such as ChatGPT. Compared with traditional unsupervised methods that builds upon "small" embedders, ClusterLLM exhibits two intriguing advantages: (1) it enjoys the emergent capability of LLM even if its embeddings are inaccessible; and (2) it understands the user's preference on clustering through textual instruction and/or a few annotated data. First, we prompt ChatGPT for insights on clustering perspective by constructing hard triplet questions $<$does A better correspond to B than C$>$, where A, B and C are similar data points that belong to different clusters according to small embedder. We empirically show that this strategy is both effective for fine-tuning small embedder and cost-efficient to query ChatGPT. Second, we prompt ChatGPT for helps on clustering granularity by carefully designed pairwise questions $<$do A and B belong to the same category$>$, and tune the granularity from cluster hierarchies that is the most consistent with the ChatGPT answers. Extensive experiments on $14$ datasets show that ClusterLLM consistently improves clustering quality, at an average cost of $\sim$\$0.6 per dataset.

CLUSTERLLM: Large Language Models as a Guide for Text Clustering
Yuwei Zhang Zihan Wang Jingbo Shang"
University of California, San Diego [yuz163, ziw224, jshang)@ucsd. edu
Abstract
Traditional Text Clustering
Embedder
We introduce CLUSTERLLM, novel text clus- tering framework that leverages feedback from an instruction-tune large language model, such as ChatGPT. Compared with traditional unsupervised methods that builds upon "small" embedders, CLUSTERLLMM exhibits in- triguing advantages: (1) it enjoys the emergent capability LLM even embeddings are inaccessible; and (2) it understands the user's preference on clustering through textual in- struction and/or few annotated data. First, prompt ChatGPT for insights clustering perspective by constructing hard triplet ques- tions <ddnes A better correspond to B than C>, where A, B and C are similar data points that belong to different clusters according to small embedder. We empirically show that this strat- egy is both effective for fine-tuning small em- bedder and cost-efficient to query ChatGPT. Second, prompt ChatGPT for helps clustering granularity by carefully designed pairwise questions <do and B belong to the same cutegory>>, and tune the granular- ity from cluster hierarchies that is the most consistent with the ChatGPT answers. Ex- tensive experiments on 14 datasets show that CLUSTERLLLM consistently improves cluster- ing quality, average cost of ~S0.6' per dataset. The code will be available at https: //gi thub. com/ /ww.steetttttt.LLL.MMMM.
Texts
(Instructor,
E5,GTR
ChatGPT (API-based)
Not Applicable
LlusterL..M
Embodder (Instructor, ES,GTR ....)
IIC
ChaiGPT (API-based)
A should be closer to C than HI
Figure I: LLMs like ChatGPT are not applicable for text clustering directly because of the inaccessible embed- dings. CL.USTERLLLM resolves the dilemma by leverag- ing LLM as a guide on text clustering.
2021a) on top of pre-trained embedders (Muen- nighoff et al., 2022; Wang et al., 2022; Su et al., 2022) which could achieve higher performance with better pre-training quality. State-of-the-ar large language models (LLMs) such as recent GPT series (Brown et al., 2020; Ouyang et al., 2022; OpenAI, 2023) have demonstrated extraordinary language capabilities for various NLP applications however, these GPT models can only be utilized through the APIs without accessible embedding vectors for clustering. Hence, L.LMs cannot be directly applied on text clustering tasks.
1 Introduction
this paper. we provide insights on the ques- tion: Can we leverage API-based L.Mss to guide text clustering efficiently? We attack this challeng- ing question by drawing inspiration from an obser- vation that humans represent an instance through comparing with others (Nosofsky, 2011). For in- stance, people often classify new piece of mu- sic into a specific genre by relating familiar ones. In fact, pairwise relationships have been uti- lized in spectral clustering (Donath and Hoffman, 1972; Cheeger, 1970) before. Nonetheless, naively
Text clustering, as fundamental task in natural lan- guage processing (NLP), has wide spectrum of applications, such as identifying public perception from social media (Park et al., 2022), analysing cause of accidents (Xu et al., 2022), and detecting emerging research topics (Martínez et al., 2022). A common practice for text clustering is to apply clus- tering algorithms (MacQueen, 1967; Zhang et al.,
Corresponding author. 'The cost is calculated with gpt-3.5 3.5 3.5tbbbb..

---

(Smally) Langrage Medals
Entropy-based Triplet Sampling
Tripict
Fine-runed
Pairwise Peammpel || Pxample 2|
Tish
Select based om shupe. closer to * Choice I:
Kmbedder
Kinbedder (Instructor,Es. ...)
A and i the cluster?
(Instructor,bs,.. ...)
Choice 2:
Please answer with Yes or No?
APl-based Lange Language Models +
APl-based Large
Language Models (Embeedings
Stage Deternine Granularity
Embodding Space castopy
(Embeeddings
Text Corpus
Step
should closer I than
Suge Improve Perspective
Step
Skap
and is the serse cluser
should closer than $C
Consistency
Supt
Clasters: (122). (3). (4,5)
Figure 2: An overview of CLUSTERLLMM. It utilizes LLM to guide embedder for text clustering with a low cost.
traversing all the pairs within dataset is obviously intractable and too expensive for querying LLMs.
datasets span a wide range of granularities that have 10 to 150 number of clusters. We show that CLUS- TERLLM is effective overall on improving clus- tering quality, where the clustering performance is improved over both a deep clustering baseline and self-supervis baseline. Moreover, the ablation study shows that our sampling strategy is effective compared to a random sampling baseline. Finally, CLUSTERLLM also outperforms clustering- based methods on determining cluster granularity. In summary, our contributions are three-fold: (i) We propose a framework CLUSTER that uti- lizes sentence relations predicted from API-based LLMs to guide clustering. Furthermore, it allows users to provide textual instructions and/or few- shot annotations specify preferences on cluster- ing. (ii) order to reduce API-queries, we propose a novel entropy-based sampling strategy to find the most informative triplets. Additionally, we utilize pairwise data sampled from hicrarchical cluster- ing to determine cluster granularity. (iii) Extensive experiments show that our proposed method can improve clustering performance at $00.2 for per- spective and $0.4 for granularity with GPT-3.5.
We propose CLUSTER a framework that utilizes LLM to guide a small embedder for finding text clusters with a low cost, as shown in Figure 1. It comprises two stages that are specially designed for two aspects of clustering: (1) perspective, i.e., the grouping criterion such as topic, intent and emo- tion and (2) granularity, i.e. the scope of clusters.
In Stage 1, we prompt LLMs with triplet task that predicts which one of the two candidate choices is closer to anchor instance understand the user-preferred perspectives. We choose this triplet task because (a) it is irrelevant with cluster granularity and (b) the produced triplets can fine- tune small embedder towards the right perspective. In order to improve sample efficiency, we further propose entropy- based triplet sampling to find the most informative triplets. Specifically, we first cal- culate entropy for each instance based on cluster assignment probabilities, and then identify those with highest entropy. Two candidate choices are then sampled from its nearest clusters to guarantee they are close enough to the anchor.
In Stage 2, we first obtain the cluster hierarchy that starts from instance-level clusters and itera- tively merge two closest clusters until the entire dataset. And then we prompt LLMs to determine cluster granularity with few annotated data pairs as demonstrations. We construct the data pairs to prompt by sampling from two clusters that are merged each step of hierarchical clustering, so that they cover wide range of granularities. And the final decision is made by measuring consistency between each level clustering and predictions.
Preliminary
Text clustering takes an unlabeled corpus D [ as input, and outputs a clustering assign- ment y = (v)) that maps the input text to clus- ter indices. To specify user's needs, CLUSTER- LLM integrates additional textual instruction (e.g. "Select the example that better corresponds with the Query in terms of entity type.") understand per- spective and few-shot annotations (e.g. "Sentencel and Sentence2 have the same entity determine cluster granularity.
to
We extensively evaluate CLUSTERLLM on 14 datasets that include diverse tasks such as intent dis- covery, topic mining, type discovery, domain dis- covery, and emotion detection, Furthermore, these
Our CLUSTERLLM
CLUSTERLLM is based on a pre-trained small em- bedder (Wang et al., 2022; Su et al., 2022) (denoted

---

as f) which usually represents sentences individu- ally. In contrast, inspired by human cognitive abil-
Algorithm 1: Fintropy-based Triplet Sampling
Input: embbedding 2 = [2+ [(x)) interval
ity (Nosofsky, 2011), CLUSTERLLM considers a pair or a triplet of sentences through prompting
boundaries "higa and "oow closest clusters fraction = 2%, maximum number of queries
Q.
LLMs that are trained to follow human instruc-
Step I: K clusters t- Clustering(2);
tions (Ouyang et al., 2022; OpenAI, 2023). Specif-
Compute pe for each cluster & by averaging;
ically, CLUSTERLLM is a two-stage framework (See Figure 2). In Section 3.1 we introduce Stage that utilizes triplet task to improve clustering quality with respect to user-specified perspectives,
Compute Kaosa with Eq. 3:
Compute entropy H with Eq. 4:
Ind t argsorn H):::1];
Ind < Indd N Now NI;
Step 2: Initialize triplets (4g] <- O):
along with a sampling strategy that reduces number of API queries. In Section 3.2, we introduce Stage 2 that leverages pairwise task to determine cluster granularity based on predictions from LLMs.
while lení (on((1,)/ Q do
for Ind do
Obtain Kelnes closest clusters: Sample C2, C2 from closest clusters;
C1 C1, C2. t ( if t not (. # a, 02 o then Append to {tg]:
3.1 Triplet Task for Perspective
In this section, we explore how to harness a triplet task refine the cluster structures for a user- specified perspective. A triplet task takes as input a tuple of three sentences / = (a, C1, 02), where a is the anchor and (c1,02) are two choices. We then prompt LLMs to select one of (c1, 02) that better corresponds with using a prompt Pr. Moreover, in order to specify the user's perspective, Py also requires task instruction IT as input. The LLM should make choice
Output: set of triplets (
tent across datasets and only specific to the embed- der model f. Cluster center μ& will thereafter be calculated for cluster & by averaging embeddings assigned to it. Following (Xie et al., 2016; Van der Maaten and Hinton, 2008), we calculate instance- wise soft assignments with Student's t-distribution.
(1+ + 1| | /0) T+1
Pik Ex 1|2 1112s-4x11 12/12/00
(2)
Cj cj= -I,(IIT
(1)
T+
where Cj E {(1, (2) indicates one of the choices that LLM selects as positive and we denote the other (or negative) one as Cgy
where a 1 is the degree of freedom. We then define closest clusters for instance i as Kelosest clus- ters with largest soft assignment Pik- Here, Kelosest is proportional to the total number of clusters K.
3.1.1 Easro Triplet Sampling
While one can randomly sample triplets to query the LLM, we demonstrate it non-efficient in exper- iments. In this section, we pose the question of mining informative triplets to both save the costs from querying LLMs and optimally improve the clustering. To achieve this, we resort to the current clustering results from the extracted embeddings Z = { = (x In summary, our algo- rithm contains two steps: Step 1: We find the most ambiguous instances as anchors based on entropy. Step 2: For each anchor instance, we sample two choices from two of its closest clusters. Refer to Algorithm for entire process.
Kelosest max(eK, 2)
(3)
where we fix be a small value, such as 2%. We then compute entropy based on these closest clusters with renormalized probabilities p/A
Kelosstt
hi
phe log(p())
(4)
where PLA Kekooss Pa
Pik We sort the entire dataset
Ex-=1
in descending order according to the entropies H = [h.) We introduce two hyperparameters Thigh and ?oow that control the proportion interval to filter out from ordered dataset. Our hypothesis is that higher entropy (smaller "high and Now) an- chors form more informative triplets that we verify in Section 4.6. In Step 2, we randomly sample two clusters C, C2 from Keleeest closest clusters,
In Step 1, since the granularity is unknown at current stage, we perform clustering on top of Z, where the clustering hyperparameters are consis-
It can be number of clusters in K-means or maximum distance to merge two clusters in hicrarchical clustering

---

and then sample two sentences C1, C2 from each of them as choices (sce line 11 and line 12). In another word, these choices should be either a pos- itive or a hard negative to the anchor. Finally, we also remove triplets that are either repeated or have identical choice and anchor. We continue to sample triplets until reaching budget Q.
determining granularity into finding the best step in a cluster hierarchy (see Figure right), where each step denotes unique granularity (or equally number of clusters). It is non-trivial, since differ- ent granularities can be applied to the same dataset (such as domains or topics). To tackle this chal- lenge, we query LLM with pairwise task that pre- dicts whether pair of data belong to the same cluster with prompt Pp.
Remarks. (1) Since Q is defined by the user and is independent with the dataset size, our sampling is cost-efficient. For example, in our experiments, using 1, 024 queries can improve performance on
w - PP(Ip, (pa)2-1.P)
(6)
both dataset scales of ~ 3, 000 and ~ 50, 000. (2) From the view of ground truth, the sampled triplets might contain "both are correct" or "none of the above". However, we argue that even these triplets might provide soft aligning information, i.c. the ranking of closeness between choices.
where w {same, different) is the binary decision, Zp is the task instruction and (f4) d are few-shot demonstration pairs used for in-context learning (typically D = 4). We assume these demonstration pairs are annotated by users who have desired cluster granularity in mind. We also combine a brief justification for each demonstration pair (see Table 12 bottom for example).
(3) Our sampling method may also be utilized in active learning to acquire human annotations when no prior knowledge is available on the categorics.
3.2.1 Determine Granularity with Pairwise Hierarchical Sampling
3.1.2 Finetuning Embedder
Now that we have the triplet predictions, is still not clear how to utilize them in clustering. Previous research rely on deep constrained clustering (Zhang et al., 2020; Manduchi et al., 2021) which are of- ten sensitive to noisy labels (Basu et al., 2008). In this paper, we instead focus finetuning the base embedder f towards producing embedding space that better explains the user's perspective. We exploit both hard and in-batch negatives. Fol- lowing (Su et al., 2022; Ni et al., 2022b), for a triplet & = (a, Cj, 9i)) with positive ey and hard negative Cyj we optimize the following objective,
We then introduce how to sample pairs from cluster hierarchy to query LLMs and determine granularity. We assume a maximum and a minimum number of clusters (denoted as kimxx and kmin) following Pel- leg et al. (2000) which depend on the user's expec- tation on the granularity. We then randomly sample À (1 or 3 in our experiments) pairs of data from the two clusters to be merged each step form didate pairs {p:) where N₂, = A(kimax - kmin)- These pairs cover the entire range of granularity between kmax and kmins and will be used to query LLMs. After that, each level of granularity can be examined against LLM predictions to choose the one with the highest consistency measure M.
exp (sp(,
1 EeeBeep exp (s(a,c))/7)
(5)
k argmax M(WP, Wk)
(7)
where B combines Cj. Oú and other in-batch nega- tives. 7 is temperature parameter. Following the original implementation, we also compute the loss with and сj swapped. Finally fine-tuned embed- ders can be applied to find even more informative triplets with our sampling method which will fur- ther improve performance in an iterative manner. We acquire clustering assignments by running clus- tering algorithms on top of extracted embeddings.
where WP (w(" J- denotes the predictions ob- tained from Eq. 6 and W represents set of binary values indicating whether each pair of data is the same cluster at gramularity kt Empirically, we found the following performs better in our frame- work: use F-beta score, weighted harmonic mean of precision and recall, as measurement M and set wr/w' as labels/predictions. Finally, for large- scale datasets, we address the high time complexity of hierarchical clustering by applying it on top of mini-batch K-means. See details in Appendix A. Remarks. Similar to Section 3.1.1, pairwise hicrar- chical sampling can also be used to acquire human
3.2 Pairwise Task for Granularity
In this section, we build upon the refined embed- ding space in Section 3.1 to determine cluster gran- ularity. In this paper, we convert the problem of

---

annotations. Nonetheless, the reliability of the al- gorithm still depends on the quality of clusters. In an extreme case where the clusters are completely random, is unable to find granularity even though all the pairwise predictions are correct.
Task
Name
#clusters
#data(small)
#data(large)
Bank77
77
3,080
10,003
CLINC())
150
4,500
15,000
Intent
MTОP()
102
4,386
15,638
Massive())
59
2,974
11,510
FewRel
64
4,480
40,320
Type
FewNerd
58
3,789
50,000
FewEvent
34
4,742
18,969
4 Experiments
StackEx
121
4,156
50,000
Topic
ArxivS2S
93
3,674
50,000
We first evaluate CLUSTERLLM clustering qual- ity with ground truth number of clusters in Sec- tion 4.4. Then we conduct ablation studies in Section 4.6 to further analyze the effectiveness of CLUSTERLLM. Finally, we show results of deter- mining cluster granularity in Section 4.7.
Reddit
50
3,217
50,000
Emotion
GoEmo
27
5,940
23,485
CLINC(D
10
4,500
15,000
Domain
MTOP(D)
11
4,386
15,667
Mussive(D)
18
2,974
11,514
Table 1: Dataset statistics.
4.1 Datasets
up a max token of 10. We then assign them binary choices by directly checking whether one of the texts "Choice 1" or "Choice 2 is the re- sponse. We also find that a very small amount of responses do not contain any choices and we dis- card them during fine-tuning. We use the Python API tool provided by OpenAI.
We provide a high-level summary of evaluated datasets in this section, and see Appendix for more descriptions. In this paper, we evaluate on a broad range of clustering datasets with various per- spectives and granularities. Furthermore, to better analyze the effect of scale, each dataset has both small-scale and large-scale version. The sions are different number of data while keeping the same number of clusters. A summary of dataset statistics is shown in Table Note that there is no data splits in clustering.
Triplet Sampling. For both small- or large-scale experiments, we budget of Q 1, 024 triplets. We set "nww = 20% and "high = For clustering methods, we fix hyperparameters of these algo- rithms across datasets in Stage 1. We choose ag- glomerative clustering with fixed distance threshold 67 for mall--scaee experiments on Instructor, and 77 on ES (the embeddings are preprocessed by stan- dard scaler). For large-scale datasets, choose mini-batch K-means with fixed number of clusters 100 due to its lower latency. Clustering algorithms are implemented by scikit-learn (Pedregosa et al., 2011). I).
Intent (Domain) Discovery. Intent discov- ery (Zhang et al., 2021b, 2022) discovers unknown intents in unlabeled customer utterances. For CLINC, Massive and MTOP, we also use domains as labels convert them into domain discovery. Type Discovery. Type Discovery (Li et al., 2022) resolves the clored-world set-up of traditional Infor- mation Extraction. In this work, we focus on three tasks: entity, relation and event type discovery. To indicate specific mentions (entities event trig- gers), we directly append them behind sentences with natural language formats, such as "The rela- tion between [ENTITYI] and [ENTITY2]'.
Fine-tune Embedders. In this work, we focus on two state-of-the-ar pre-trained embedders: Instruc- tor (Su et al., 2022) and E5 (Wang al., 2022). We only use the large versions. Refer to Appendix D for details.
Topic Mining. We adapt three topic mining datasets from MTEB (Muemnighoff et al., 2022). Emotion. We adapt GoEmo (Demszky et al., 2020), a fine-grained emotion detection dataset by removing multi-label or neutral instances.
Evaluation. To reduce cost, we run CLUSTER- LLM once for each dataset. We then run (mini- batch) K-means on (large) small-scale datasets for 5 seeds with ground truth K. We show two metrics. The first one is clustering accuracy calculated after Hungarian alignment (Kuhn, 1955) that permutc prediction classes back label classes. Another popular metric for clustering is normalized mutual information (NMT) that calculates mutual informa- tion between two assignments, and normalized by their individual entropies.
4.2 Experiment Details
Query LLMs. The prompt only contains task- specific instruction (see Table 11). We set gen- eration temperature to 0.5. Explanations are sup- pressed by adding a "Pecae respond with 'Choice I' or "Chocee 2' without explanation" and

---

Intent Discovery
Emotion
Method
Bank77
CLINCO)
MTTP())
Massive())
Golmno
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
ES 2.
59.90kmm
77.7 buszs
75.83mm
91.16maa
33.54
70.79ka
52.52nsx
70.67am
22.13ase
20.98
SOCL-Embaasa,
63.60
77.34
77.96um
91.89mm
33.82mum
70.42aa
94.48u.
71.57mm
22.03
20.05
sclf-upaviss--
64.76
80.75
78.9 mm
92.06m2
33.81
71.71
54, v3 vo
21.78
21.35
20.92
CLUSTERLLM-E
69.09kmm
83. 7
79.5 Ium
92. mm
34.661.3
73.19mm4
54, 80
73.97ase
22.
22.07 22.6
CLUSTERLLM-EEiiE
70. 13
84, 34,16Gm 16
80, 80,48mm
92, 92.92mzs
37.22
74.46
56.08mm
74.39
22.22
22.23
Instructor de eat 2012
64,49
81.43mm
19.
92.60
33.35
70.63
54.08
a3.42
25.19h
21 44.54
SCCL-1 CoLL--oaaeeet...m 2t2te
65, 18
81.77
80,85
92,94ma
24.08
73.52
54. a0
73.90k.
34.33
30,54..0
sdffanprviss-]
68. 18
83.31a
80.82
93.88
34.06
55.07 72.50
72. 88
24.11
22.05
CLUSTERLLM-1
70.77 70.77
85.07
82,77
mm
25.84
73.52mm
59. 89
76.96as
27. 27.49huz5
24.78
CLUSTERULMM--iiee
71.20.1..
85.15man
83.80
94.00mzm
35.04pmm
73.83
60.69%am
26.75 77.64km
m)
23 89
Type Discovery
Topic Mining
Method
FewRel
FewNerd
Few
StackEx
Reddit
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
ES (00gggge, 2005
39.6 39.62.r..
55.78mm
25.49mmm
40.62mm
37.30
59, 39,288
37.31
58.59
39.03
48.63
NOCI-Eshapsra.a00
39,93mm
56.49mm
27. 77.86mm
12
35, 35.39mm:
57.23ms
85
59.17ax
44.45
53.29%am
Sdlfaupervis-E
43.01am
58. 78.m8
29.25
37.07mm
62. 14
19
63.06a mo
48 48
57. 62.00
CLUNTRLLLM-E
47.53ku
62.89
28.52mm
44.45mm
42. 17mm0
67.55mm
43.01a
63.81as0
47.72
56.41aa
CLUSTERLL M----tr
52.95
67.64kcm
12.160
16
64.64
74,74
43.93mm
66.666 so
49.05
57.35
Instructor oueeat sody
41.23
57.55mm
30.02
12
41.99pm
62.88
44.8
65.76amo vo
54,98
62.51aa
41.15
57.04mcs
31,09
46.47mm
39. 39.97mmu
57. 36....
45.1 11mo
15.36m
53 66.60
61 1,340mm
seff-ugerviss-]
41.72mm.
57.83
31.39
47,25mm
43.94
64.71
46.
66.49 vo
55.41mm
63.45
CLUNTBBLLM-
47.94
62.43mum
34.75mm.
51.03mm
46. 17am
70. 73
66.78a 47.21mm
56.79
63.87
CLUSTORL M-q-tor
41.22
65.53mm
40.60,mcm
57.35
50.60mmm
73.8%
47.75mmm
67.00%
63.92 57.02mmm
Topic Mining
Domain Discovery
Avg
Method
ArxivS2S
MTOP(D)
Massive(D)
CLINC(D)
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
30.85
54.49m05
91.23
86. 75
63.70mum
66.20mm%
59.64p%
57.23ase
47.72
61.35
32.78mum
55.77km
92.28muu
86.01am
61.22min
65.27aw:
59.88 99.88
56.21asc
48.82
61,70
self-nperviis-::
34.41
57.09mms
91,40p-n
28.20
65.02, 61.79
s7.29
57.07
49.87
63.60
CLUSTERLLM-U
34. 93
57. 77.90pmm
86. 19
60. (3
66. 15aw:
56. 25:30
56.32am
50.77
64.77
CLUSTERLLM-EN E ster
35,73
58.42mmm
89,58
87.25
58.63n.s
65.59mam:
60,842se
58.55
52.40
66.
24.3 lomTs
48. 04.04
90.56
87.30
61.8 Jaw
67.3 1a.0
$2.50ka
56.87
49.90
63,85
SCCL- [alLleeeeeaaao
25, .3
49.07mmm
89,08.mm
84.77am
61..44mm
68. 69.-:
54.22aus
51.08aas
50.74
63,85
self-supervise.!
25. 65.55
49.4 1us
92. 12
88.49
53,97
71.53kmm
58.58p
60.84
$1.39
65.33
CLUSTEBLLM-L
26.61
50.06wza
93.53
89.36mm
61.06mmm
68. 62..
52.39
54.982ac
53.09
66.58
CLUSTERLLM Liter
26.34
50.45mum
13
89.23
60.85,43)
68.67as
51.82am
54,8
53.99
67.53
Table 2: Comparison of clustering accuracy and NMI with known gramularity for evaluation. Average over all 14 datasets shown in the last two columns. Best results are bolded.
4.3 Compared Methods
for training. Notice that our method uses fewer data for training. See Appendix D for details.
E5 and Instructor. We directly apply (mini- batch) K-means on extracted embeddings from instructor-large and e5-large.
4.4 Main Results
self-supervise-I((E) To verify that the performance improvement of CLUSTERLLM does not only come from domain-specific fine-tuning, instead of the more accurate triplet prediction. We propose a self-supervise fine-tuning that uses exactly the same triplcts as CLUSTERLLM but only switch to self superviid triplet predictions that select closest choices in embedding space.
We show main results with small-scale datasets Table 2. We show several variants of our method: CLUSTERLLM-(E) adopt Instructor or E5 as em- bedders. CLUSTERLLM-ICE tire framework twice in an iterative manner by using previous fine-tuned model initialization and the 1, 022 triplets inferred from new embed- dings for fine-tuning. All of these use GPT-3.5 for prediction. We make the following observa- tions: (1) CLUSTERLLM consistently improves upon both embedders. For example, CLUSTER-
applies the en-
SCCL-I(E). We also combine Instructor and E5 with SCCL (Zhang et al., 2021a), an unsupervised deep clustering algorithm that utilizes entire dataset

---

Type
Bank77
CLINC()
(TOP(I)
Massive(1)
GoEmo
FewRel
FowNerd
#GT Triplcts
23
102
61
117
41
156
Instructor
100
100
98.04
88.52
68.38
80.49
71.15
Random
GPT3.5
100
100
85.29
85.25
68.38
85.37
82.05
A
(+0)
(+0)
(-12.75)
(-3,27)
(+0)
(+4,88)
(+10,90)
#GT Triplets
510
462
140
206
266
347
Instructor
64,12
76.19
65.74
63.56
64,08
62.41
59.65
Entropy-hased
GPT3.5
76.67
79.44
67.41
64.56
76.69
(+12.55)
(+3.25)
(+1.67)
(+5.20)
(+0.48)
(+14.28)
(49,23)
79.41
80.74
76.04
74.84
61.65
(+15.29)
(+4.55)
(+10.30)
(+11.28)
(-2.43)
(+24.81)
(+22.48)
Type
Model Triplets
FewEvent 105
StackEx
Reddit
ArxivS2S
MTOP(D)
Massive())
CLINC(D)
40
Instructor
85.71
95.45
96,74
80.41
76.72
Random
GPT3.5
94.29
71.43
70
81.82
85.87
82.43
68.25
(-3.81)
(-14,28)
(-10)
(-13.63)
(-10.87)
(+2.02)
(-18.47)
#GT Triplets
259
271
92
145
144
108
208
Instructor
70.66
68.27
61.98
59.31
70.31
63.82
75.06
GPT3.5
83.78
71,22
63.28
73.79
69.79
72.09
75.78
Enttopy-hased
A
(+13.12)
(+2.95)
(+1.30)
(+14.48)
(-0.52)
(+8.27)
(+0.72)
GPT4
85.71
79.70
67.71
77.93
72.40
70.28
74,58
(+15.05)
(+11,43)
(+5.73)
(+18.62)
(+2.09)
(+6.46)
(-0.48)
Table 3: Analysis on the triplet prediction accuracy (f is used produce results of CLUUTERRLLMMI in Table 2).
Red and green mean decreased or increased performances respectively. "#GT Triplets" means triplets that have ground truth (see Section 4.5 for details).
LLM-1 increases the performance by 6.71% on FewRel. CLUSSERRLLM-E increases the perfor- mance by 9.19 on Bank77. However, we do ob- serve that on Massive(D) and CLINC(D), there are no improvements, (2) CLUSTERLLM outperforms deep clustering and self-supervise basclines. For in- stance, CLUSTERLLM- surpasses self-superrise-I most datasets except for two and it is also better than SCCL-I 11 over 14 datasets. Furthermore, these improvements are consistent across both re- ported metries. (3) Combined with the results in Appendix F, applying CLUSTERLLM iteratively is beneficial, emphasizing the potential of further improvements.
amples, demonstrating our hypothesis. In contrast, with random sampling, the ground truth triplets is significantly fewer and the accuracy gap much smaller or even decreases performance.
4.6 Ablation Study
Clustering Quality. We show ablation studies on CLUSTERLLM based on Instructor in Table 4. Specifically, we present results with 3 kinds of predictions the same set of triplets for fine- tuning: GPT-3.5/4, replace triplet predictions of GPT-3.5 to ground truth on those triplets that have ground truth. We observe that GPT-4 marginally improves upon GPT-3.5 given the much higher cost. When provided with human labels, CI.US- TERLLM-GT&GPT3. achieves the highest perfor- mance, which indicates the possibility for further improvement with more accurate predictions. We make similar observations for large-scale datasets in Table 6.
4.5 Analysis on Triplet Prediction Accuracy
We attribute the improvements on clustering qual- ity more accurate triplet predictions. In Table we show the accuracy on predicted triplets that have ground truth (exactly one positive and one negative choices based on ground truth) with two different sampling methods. Random triplet sam- pling uniformly samples three random instances as query and two choices, and guarantec the two choices are different from the anchor by filtering. Furthermore, we also show selection accuracy with Fuclidean distances between embeddings as a comparison. We observe that, GPT-3.5/4 consis- tently improves upon Instructor on high entropy ex-
Sampling Strategy. In this section, we show abla- tion study on entropy- based sampling. In Figure 3, we observe that clustering accuracy increases when increasing entropies (or equally decreasing mean of interval) except for GoËmo. We make hy- pothesis: (1) LLMs much better than small embedders on harder instances. (2) high-entropy instances are generally more informative. Ta- ble 4, we observe that training with randomly se-

---

Intent Discovery
Emotion
Method
Bank77
CLINCO)
MТОР())
Massive())
Golano
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
ACC.
NMI
CLSSTRLLMMM CuSTEALLLM.OFD.SSaall Srms.o
59,88 2m
79.69
74.40mmm
90.38aa
28.05
61.76asx
51.66240
68.87am
28.62
25.88
CuSTERLLM--PP5
70.77
85.07
82.77n.a
93.88mm
35 m5.84am
73,52ax0
59.89kas
76. 76.96sso
27. 27.49k2
24.78mmm
69.71am
84, 68
81.91
93.76
34.48%
59.10 a3.57
i
76.59
27.41.m.
23.77ma
71.35
85.12
84.00
94.34muM
36.8614
75.36sm
59.276.4
77.37as
30.91
27.71mmm
Type Diseawery
Topic Mining
Method
FowRel
FowNerd
FewEvent
StackEx
Reddit
ACC
NMI
ACC
NMI
ACC
NMI
ACC.
NMI
ACC
NMI
CuSTELLLM Stasbo)
40.6 40.65
56, 54
27.15 15
43. 43.56
44.23
66.75am
40.81eso
62. 10eso
54.60azs
82
47.94
62.43
75
51.03
m6.17
10.73a
47.21mm
66.78
56.79kmm
63.87
CLSTELLLM GPt4
48.96m.
63,58mu
37.54mas
53.94mm
47.98t.
71.32
46.82amo
66. 26.72mm
55.38
45.45
Cusmelll
48.9 pm
34
37.27
m3.57
72.31a
47.55mmm
67.04k.m
58.33
65.34
Topic Mining
Domain Discovery
Avg
Method
ArxivS2S
MTOP(D)
Massive()
CLINC())
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
22.03
45.50mm
87.00
82.09
v6.40
64.39,
60.27
58.T
48.27
61.96
26.61
50.06mar
93.53mu
89.36
61.06am
68.62amm
52.39kse
54.98ase
53.09
66.58
26. 16
50.06
92.04pm:
88. 39,
16.16
67.9 67.98.
$7.45
59.98.
53.22
66.98
CUSVAALLM-CTAGMT
26. 14
50.19
92.26
89.36
61.65am
69.51as
52.87aso
56.4300
53.96
67.64
Table 4: Ablation study clustering quality with Instructor backbone and known granularity for evaluation. See more results with large-scale datasets in Table 6.
Relative Clustenig Accuracy
1.00
198 & 594 pairs in total respectively. To simulate experts for providing demonstrations, we directly sample 16 pairs from small-scale datasets when À = 3 and then choose 2 positive and 2 negative as demonstrations. Notice that we the same demonstrations for large-scale experiments. See more details in Appendix B.
0.99
0.98
0.97
0.96
0.95
0.94
Banc77
We make several observations from Table and Table 7: (1) Our methods have higher ranks. Most baseline methods predict similar number of clusters for domain and intent, while our methods can ef- fectively distinguish between the two. For instance. MTOP(I/(D) in Table 5, BIC predicts num- ber of clusters 69/ /64 while our method (GPT-3.5, À = 3) predicts 92/ 188. (2) Increasing À generally helps (MTOP(D) in Table 5) but might not always make a large difference. (3) GPT-4 significantly improves upon GPT-3.5, probably due to its better understanding of demonstrations.
FowRe!
0.93
StrkEx
Gofma
10%
20%
50%
80% 90%
Mean of Interval (Entropy)
Figure 3: Relative clustering accuracy (divided by max- imum for better aligning across datasets) of CLUSTER- LLM-GPT3.5 with different range of entropy selected. x-axis shows the mean of interval where interval length is set 20%. For example, "mean of interval= 50%" means Thigh 40% and "Now = 60% (see Section 3.1.1). marks the setting for main experiments.
lected triplets even decreases performance, which demonstrates the cruciality of triplet sampling.
Related Works
4.7 Determining Cluster Granularity
Clustering. As a fundamental task in machine learning, clustering has been applied on diverse data types, including texts (Xu et al., 2015; Hadi- far al., 2019; Zhang al., 2021a), images (Yal- ing Tao, 2021: Yang al., 2016; Caron al., 2018; Niu et al., 2020; Xie et al., 2016) and graphs (Huang et al., 2014; Chiang et al., 2019). Recent research has been shifted to deep cluster-
In this section, we show the results for determining cluster granularity. We evaluate on subset of 8 datasets including various cluster granularities with kmax 200 and kimin 2. We compare with different methods that rely on clustering errors. For our methods, we show results with = (1,3) (except for GPT-4 to reduce costs), which involve

---

Method
Bank77
FewRel
Massivell)
Massive(D)
MTOP()) MTORD)
CLINC())
CLINC(D)
Rank
GT fehasters
77
64
59
18
102
11
150
10
Silhouctte (oosoos,, 1860
118 (53.25)
10 (84.38)
38 (35.59)
41 (127.8)
(89.22)
11 (0)
172 (14.7)
163 (1830)
10
Elbow chimalle 1990
53 (31.17)
43 (32.81)
45 (23.73)
46 (155.)
33 (67.65)
34 (209.1)
66 (56.00)
68 (580.0)
9
X-meenn (heligg aral Xoej
69 (10.39)
30 (53.13)
32 (45.76)
34 (88.89)
28 (72.55)
27 (145.5)
130 (13.33)
135 (1250)
8
BIC a 3000
123 (59.74)
58 (9 (9)55)
56 (5.08)
60 (235.3)
69 (32.35)
64 (481.8)
167 (11.33)
176 (1660)
7
Chisterize usung x APrIc
86 (11.69)
71 (19.4)
72 (22.03)
90 (400.))
82 (19.61)
85 (672.7)
105 (30.00)
106 (960.0)
6
Ours (GPT3.5.A = D
64 (16.88)
46 (28.13)
43 (27.12)
90 (400.0)
43 (57.84)
40 (263.6)
151 (0.67)
96 (860.0)
5
Ouns (GPT3,5,A 3)
64 (16.88)
46 (23.13)
52 (11.86)
37 (105.6)
92 (9.80)
18 (63,63)
142 (5.33)
107 (970.0)
2
Ouns (OPTT,A 1)
56 (27.27)
46 (28.13)
41 (30.51)
20 (11.11)
53 (18.04)
8 (27.27)
146 (2.67)
39 (290.0)
3
Ours (GT,A 1)
100 (29,87)
91 (42.19)
42 (28.81)
18 (0)
41 (59.80)
11 (0)
141 (6.00)
39 (290.0)
4
Ours (GT,A 3)
99 (28.57)
94 (16.88)
62 (5.08)
20 (11.11)
37 (63.73)
11 (0)
142 (5.33)
31 (230.0)
1
Table 5: Cluster granularity on small-scale datasets. Maximum & minimum number of clusters are set to 200 & 2. The results are shown in format of [[#clsterr]] (errors)". "Rank" column is computed with 1-level ranking (Colombo et al., 2022) with inverse errors. "GT" is ground truth. See results for large-scale datasets in Table 7.
ing (Zhou et al., 2022) which focuses on how to leverage deep neural network in clustering. Zhou et al. (2022) has categorized deep clustering re- search into four types including multi-stage (Yal- ing Tao, 2021; Huang et al., 2014), iterative (Yang et al., 2016; Caron et al., 2018; Van Gansbeke al., 2020; Niu et al., 2022; Chang et al., 2017; Niu et al.. 2020), generative (Dilokthanakul et al.. 2016) and simultancous (Xie et al., 2016; Zhang et al., 2021a; Hadifar et al., 2019) depended on how representation learning and clustering mod- ules interact with each other. Most recently, con- current work (Wang et al., 2023) studies similar problem by assigning instances to different expla- nations proposed by LLMs. Another recent work IDAS (Raedt et al., 2023) directly encode the con- catenation of sentence and abstractive summariza- tions from LLMs for clustering.
trained text embedding models (Reimers and Gurevych, 2019; Gao et al., 2021; Ni et al.. 2022a,b) are widely applied in text similarity, clas- sification, clustering and information retrieval. Re- cently, two embedding models E5 (Wang et al., 2022) and Instructor (Su et al., 2022) have shown superior performance on a popular bench- mark (Muennighoff et al., 2022). Specifically E5 is pre- pre-trained on web-scraped data pairs with con- trastive objective. Instructor is pre-trained on 330 tasks with instructions. CLUSTERLLM aims at improving these models with LLMs.
Conclusion
In this paper, we study how to leverage API-based LLMs guide small embedders for text cluster- ing in order benefit from high-level language capability of L.LMs and user's instructions on clus- tering. We propose to prompt LLMs with two kinds of sentence relationship tasks: triplet task and pairwise task. Triplet task chooses the sentence that is most similar with anchor combining with a perspective instruction from users. The predicted triplets are used for fine-tuning small embedders. Pairwise task judges whether a pair of sentences belong to the same category hinted by few-shot demonstrations, and then the predictions are used to determine cluster granularity with a consistency measure. Extensive experiments show that our proposed framework CLUSTERLLM can improve clustering quality and propose reasonable cluster granularity at negligible cost. However, CLUS- TERLLM still relies on the embedding model itself, which is inefficient and inapplicable on black-box embedding models. We encourage future works to explore the potential of model-free training such as constrained clustering.
Clustering with Relations. Clustering with rela- tions has been explored in different situations. To start with, spectral clustering (Donath and Hoff- man, 1972; Cheeger. 1970) makes use of similarity matrix where each entry measures the similarity between a pair of data. More recently, several works in deep clustering utilize relational supervi- sion (Yang et al., 2016; Niu et al., 2020; Van Gans- beke et al., 2020; Chang et al., 2017) via pseudo- labelling which could be noisy. Another line of works that is closely related to ours is constrained clustering. It usually incorporates pairwise must- link or cannot-ling constraints (Basu et al., 2004; Wagstaff et al., 2001; Basu et al., 2008; Zhang et al., 2020; Manduchi et al.. 2021). Nonetheless, these constraints are often sampled from labels as a prior which significantly limits its application in our scenario. In this work, we study how to utilize contemporary API-based LLMs infer relations.
Pre-trained Embedding Model. Generic pre-

---

Limitations
ized category discovery with decoupled prototypical network. arXiv preprint arXiv:2211.15155.
We list several limitations our work that we hope to be improved in the future:
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini. Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
Reliance on pre-trained embedder. To find the most informative data, we have to rely on pre- trained embedder that can indicate the largest clus- tering assignment entropy. We hope that self- supervise triplets and LLM-preedictd triplets can be combined to solve such an issue.
Sugato Basu, Arindam Banerjee, and Raymond J Mooney. 2004. Active semi-supervision for pairwise constrained clustering. In Proceedings of the 2004 SIAM international conference on data mining, pages 333-344. SIAM.
Computational cost for fine-tuning. Our initial idea is to utilize constrained clustering which is a light-weight algorithm that do not need to update small embedders. However, the inevitable unsta- ble training will be heavily affected by the errors in LLM predictions. We make comprise by introduc- ing embedder into fine-tuning temporarily solve the issue, but we hope to reduce the computational cost future work.
Sugato Basu, Ian Davidson, and Kiri Wagstaff. 2008. Constrained clustering: Advances in algorithmms the- ory, and applications. CRC Press.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:7877-1901.
Sub-optimal performance on domain discovery. We notice that on domain discovery datasets such as Massive(D) and CLINC(D), the performance is usually sub- sub-optimal compared with original In- structor embedding. We provide discussions on this issuc in Appendix H.
Kaidi Cao, Maria Brbic, and Jure Leskovec. 2022. Open-world semi- eemi-supervised learning. In Interna- tional Conference on Learning Representations.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. 2018. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pages
Ethics Statement
Our work employs LLMs which are accessed through OpenAl APIs. For some applications, up- loading sensicv-ee data is risky and might require efforts to remove sensitive information.
Iñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulie. 2020. Rf- ficient intent detection with dual sentence en- coders. In Proceedings of the 2nd Workshop on NLP for ConvAl ACL 2020. Data avail- able at http://tbbbcoom//lyllLLL.Dlaaa datasets.
Acknowledgements
Our work is sponsored in part by NSF CAREER Award 2239440, NSF Proto-OKN Award 2333790, NIH Bridge2AT Center Program under award IU54HG012510-01, Cisco-UCSD Sponsored Re- search Project, as well as generous gifts from Google, Adobe, and Teradata. Any opinions, findings, and conclusions or recommendations pressed herein are those of the authors and should not be interpreted as necessarily representing the views, either expressed or implied, of the U.S. Gov- ermment. The U.S. Government is authorized to reproduce and distribute reprints for government purposes not withstanding any copyright annota- tion hereon.
Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shim- ing Xiang, and Chunhong Pan. 2017. Deep adaptive image clustering. In Proceedings of the IEEE in- ternational conference on computer vision, pages 5879-5887.
Jeff Cheeger. 1970. A lower bound for the smallest eigenvalue of laplacian, problems in analysis (pa- pers dedicated to salomon bochner, 1969).
Qinyuan Cheng, Xiaogui Yang, Tiamxiang Sun. Linyang Li, and Xipeng Qiu. 2023. Improving contrastive learning of sentence embeddings from ai feedhack.
Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li. Samy Bengio, and Cho-Jui Hsieh. 2019. Cluster-gen cuuterr-gnn An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD international conference on knowl- edge discovery & data mining, pages 257-266
References
Wenbin An, Feng Tian, Qinghua Zheng, Wei Ding, Qian Wang. and Ping Chen. 2022. General-

---

Pierre Colombo, Nathan Noiry, Ekhine Irurozki, and Stephan Clemencon. 2022. What are the best sys- tems? new perspectives on nip benchmarking. arXiv preprint arXiv:2202.03799
on Empirical Methods in Natural Language Process- ing, pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine-grained emo- tions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040 4054, Online. Association for Computational Linguisties.
Gregor Geigle, Nils Reimers, Andreas Rücklé and Iryna Gurevych. 2021. TWEAC: transformer with extend- able QA agent classifiers. CoRR, abs/2104.07081.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers annotation tasks. arXiv preprint arXiv:2303. 15056.
workers for text-
Shumin Deng, Ningyu Zhang, Jiaojian Kang, Yichi Zhang. Wei Zhang, and Huajun Chen, 2020. Meta- learning with dynamic-ment mee-mmmmmooeeeeee prototypical network for few-shot event detection. In Proceed- ings of the 13th International Conference on Web Search and Data Mining, WSDM '20. page 151-159, New York, NY, USA. Association for Computing Machinery.
Cyril Goutte, Lars Kai Hansen, Matthew G Liptrot, and Egill Rostrup. 2001. Feature-space cluster- ing for fmri meta-analysis. Human brain mapping, 13(3):165-133.
Amir Hadifar, Lucas Sterekx, Thomas Demeester, and Chris Develder. 2019. A self-training approach for short text clustering. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepLANLP-2019), pages 194-199, Florence, Italy. Association for Computational Linguistics.
Nat Dilokthanakul, Pedro AM Mediano, Marta Gar- nelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulku- maran, and Murray Shanahan. 2016. Deep unsuper- vised clustering with gaussian mixture variational autoencoders. arxiv preprint (.li::/611.0244
Xingwei He, Zhenghao Lin, Yeyun Gong, A Jin, Hang Zhang. Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. 2023. Annollm: Making large language models to be better crowdsourced annota- tors. arXiv preprint 33iv:2303.16544
Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. 2021. Few-NERD: A few-shot named entity recognition dataset. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Online. Association
William Hogan, Jiacheng Li, and Jingbo Shang. 2023. Open-world semi-supervised generalized relation dis- covery aligned in real-world setting.
Long Papers), pages 3 for Computational Linguistics.
Peihao Huang, Yan Huang, Wei Wang, and Liang Wang. 2014. Deep embedding network for clustering. In 2014 22nd International conference on pattern recog- nition, pages 1532-1537. IEEE.
William E. Donath and Alan Hoffman. 1972. Algo- rithms for partitioning of graphs and computer logic based on eigenvectors of connection matrices. IBM Technical Disclosure Bulletin, 15(3):938-945 -944.
Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quar- terly, 2(1-2):83-97.
Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh. Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natara- jan. 2022. Massive: A Im-example multilin- gual natural language understanding dataset with 51 saalooiiccall--ivvrsss languages.
Stefan Larson, Anish Mahendran, Joseph J. Peper. Christopher Clarke, Andrew Lee. Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and J Jason Mars. 2019. An evaluation dataset for intent classification and out-of- scope prediction. Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)).
Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019. FewRel 2.0: Towards more challenging few-shot relation classi- fication. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNNPP- pages 6251-6256, Hong Kong, China. Association for Com- putational Linguistics.
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021. MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume. pages 2950-2962, Online. Association for Computa- tional Linguistics.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference
Sha Li, Heng Ji, and Jiawei Han. 2022. Open rela- tion and event type discovery with type abstraction.

---

In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 6864-6877, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Chuang Niu, Jun Zhang, Ge Wang. and Jimin Liang. 2020. Gatcluster: Self-supervised gaussian-attention network for image clustering. In European Confer- ence on Computer Vision (ECCV).
Ting-En Lin, Hua Xu, and Hanlei Zhang. 2020. Dis- covering new intents via constrained deep adaptive clustering with cluster refinement. In Thirty-Fourite AAAI Conference on Artificial Intelligence.
Robert M Nosofsky. 2011.
The generalized context model: An exemplar model of classification. Formal approaches in categorization, pages 18 1839.
OpenAl. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
MacQueen. 1967. Classification and analysis of mul-
tivariate observations. In 5th Berkeley Symp. Math. Statist. Probability, pages 281 281-297. University of California Los Angeles LA USA.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carrll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-2774A
Laura Manduchi, Kieran Chin- Shin-Cheong, Holger Michel, Sven Wellmann, and Julia E Vogt. 2021. Deep condi- tional gaussian mixture model for constrained cluster- ing. In Advances in Neural Information Processing Systems.
June Young Park, Evan Mistur, Donghwan Kim, Yun- jeong Mo, and Richard Hoefer. 2022. Toward human- centric urban infrastructure: Text mining for social media data to identify the public perception of covid- 19 policy in transportation hubs, Sustainable Cities and Society, 76: 103524.
José Manuel Guaita Martínez, Patricia Carracedo, Do- lores Gorgues Comas, and Carlos H Siemens. 2022. An analysis of the blockchain and covid-19 19 research landscape using a bibliometric study. Sustainable Technology and Entrepreneurship. 1010:000006.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830
Yutao Mou, Keging He, Yanan Wu, Zhiyuan Zeng. Hong Xu, Huixing Jiang, Wei Wu, and Weiran Xu. 2022. Disentangled knowledge transfer for OOD in- tent discovery with unified contrastive learning. In Proceedings of the 60th Anmual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 46-53, Dublin, Ireland, Associ- ation for Computational Linguistics.
Dan Pelleg, Andrew W Moore, et al. 2000. X- X-means: Extending k-means with efficient estimation of the number clusters. In Ieml, volume 1, pages 727- 734.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2022. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316.
Maarten De Raedt, Fréderic Godin, Thomas Demeester, and Chris Develder. 2023. Idas: Intent discovery with abstractive summarization.
Fionn Murtagh and Pedro Contreras. 2011. Meth- ods of hierarchical clustering. arXiv preprint arXiv:110.00121.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lunguuge Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP- (CCNLP, pages 3982-3992, Hong Kong, China, Association for Com- putational Linguistics.
Jianmo Ni. Gustavo Hernandez Abrego, Noah Con- stant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022a. Sentence-15: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1864-1874, Dublin, Ireland. Association for Computational Linguistics.
Mamshad Nayeem Rizve, Navid Kardan, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. 2022a. Openldn: Learning to discover novel classes for open- world semi-supervised learning. In European Confer- ence on Computer Vision, pages 382-401. Springer.
Jianmo Ni, Chen Qu Jing Lu, Zhuyun Dai, Gustavo Her- nandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wer Wei Chang, and Yinfei Yang. 2022b. Large dual encoders are generalizable retrievers. In Pruceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9844 9855, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Mamshad Nayeem Rizve, Navid Kardan, and Mubarak Shah. 2022b. Towards realistic semi-supervised learning. In European Conference on Computer Vi- sion, pages 437-455. Springer.
Chuang Niu, Hongming Shan, and Ge Wang. 2022. Spice: Semantic pseudo-labeling for image clus- tering. IEEE Transactions on Image Processing, 31:7264-7278.
Peter J Rousseeuw. 1987. Silhouettes: graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics. 20:53-65.

---

Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022. One embedder, any task: Instruction-finctume text em- beddings.
Kouta Nakata Yaling Tao, Kentaro Takagi. 2021. Clustering -eaieriin-friendly representation learning via in- stance discrimination and feature decorrelation. Pro- ceedings of ICLR 2021.
Jianwei Yang, Devi Parikh, and Dhruy Batra. 2016. Joint unsupervised learning of deep representations and image clusters. In Proceedings of the IEEE con- ference on computer vision and pattern recognition. pages 5 5147-5156.
Yiyou Sun and Yixuan Li. 2022. Opencon: Open-world contrastive learning. In Transactions on Machine Learning Research.
Robert Thorndike. 1953. Who belongs in the family? Psychometrika, 18(4):267-276.
Muli Yang, Yuehua Zhu, Jiaping Yu, Aming Wu, and Cheng Deng. 2022. Divide and conquer: Compo- sitional experts for generalized novel class discov- ery. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14268-14277.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
Wouter Van Gansbeke, Simon Vandenhende, Stama- tios Georgoulis, Mare Proesmans, and Luc Van Gool. 2020. Scan: Learning classify images without labels. Computer Vision-ECCT' 2020: 16th Euro- pean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X. pages 268-285. Springer.
Dejiao Zhang. Feng Nan, Xiaokai Wei, Shang- Wen Li, Henghui Zhu, Kathleen McKeown, Ramesh Nalla- pati, Andrew O. Arnold, and Bing Xiang. 202 Supporting clustering with contrastive learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 5419-5430, Online, Association for Computa- tional Linguistics.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. 2022. Generalized category discovery. In IEEE Conference on Computer Vision and Pattern Recognition.
Kiri Wagstaff, Claire Cardie, Seth Rogers, Stefan Schrödl, et al. 2001. Constrained k-means cluster- ing with background knowledge. In Ieml, volume 1, pages 577-584.
Hanlei Zhang, Hua Xu, Ting-En Lin, and Rui Lyu. 202 Discovering new intents with deep aligned clustering. Proceedings of the AAAI Conference Artificial Intelligence, 35(16):14365 14373.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao. Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly- supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.
Hanlei Zhang. Hua Xu, Xin Wang, Fei Long, and Kai Gao. 2023. Usnid: A framework for unsupervised and supervupe new intent discovery. arXiv preprint
Zihan Wang. Jingbo Shang. and Ruiqi Zhong. 2023, Goal-driven explainable clustering via language de- scriptions.
Hongjing Zhang, Sugato Basu, and Ian Davidson. 2020. A framework for deep constrained clustering- algorithms and advances. In Machine Learning and Knowledge Discovery in Databases: European Con- ference, ECML PKDD 2019, Würzburg. Germany, September 16-20. 2019, Proceedings, Part I. pages 57-72. Springer.
Joe H Ward Jr. 1963. Hierarchical grouping to opti- mize an objective function. Journal of the American statistical association, 58(301):236 (:3601))336244,
Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016. Unsupervised deep embedding for clustering analy- sis. In International conference on machine learning, pages 478-487, PMLR.
Yuwei Zhang, Haode Zhang, Li-Ming Zhan, Xiao-Ming Wu, and Albert Lam. 2022. New intent discovery with pre- pre-training and contrastive learning. In Pro- ceedings of the 60th Annual Meeting the Associa- tion for Computational Linguistics (Volume Long Papers), pages 256-269, Dublin, Ireland. Association for Computational Linguistics.
Hui Xu, Yi Liu, Chi-Min Shu, Mingqi Bai, Mailidan Motalifu, Zhongxu He, Shuncheng Wu, Penggang Zhou, and Bing Li. 2022. Cause analysis of hot work accidents based on text mining and deep learning. Journal of Loss Prevention in the Process Industries, 76: 76:107
Sheng Zhou, Hongjia Xu, Zhuonan Zheng, Jiawei Chen, Jiajun Bu, Jia Wu, Xin Wang, Wenwu Zhu, Martin Ester, et al. 2022. comprehensive survey on deep clustering: Taxonomy, challenges, and future dirce- tions. arXiv preprint w2Xiv:206.007777
Jiaming Xu, Peng Wang, Guanhua Tian, Bo Xu, Jun Zhao, Fangyuan Wang, and Hongwei Hao. 2015. Short text clustering via convolutional neural net- works. In Proceedings of the 1st Workshop on Vec- tor Space Modeling for Natural Language Process- ing. pages 62-69, Denver, Colorado. Association for Computational Linguistics.
Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq. Pan Hui, and Gareth Tyson. 2023. Can chatgpt reproduce human- generated labels? a study of social computing tasks, arXiv preprint 230iv:2304. 10145.

---

A Details of Scaling up Hierarchical Clustering
CLENCII CLINCID
consistency score
A major drawback of hierarchical clustering is its ((N3) time complexity which makes the al- gorithm hard to be deployed on larger datasets. However, since are only interested in spe-
0.6
cific range of granularity in our scenario, the hier-
0.2
archical clustering can start from an intermediate step. We address this issue by first running mini- batch K-means with kmax and then run hierarchi- cal clustering with current assignments as inputs. Specifically, we use agglomerative clustering with ward's s method (Ward Jr, 1963). We first calculate distances between each pair of clusters according Murtagh and Contreras, 2011 and then provide them as inputs to nearest neighbor chain algorithm. Finally the returned hierarchy combined with the K-means assignments to infer clusters.
25
50
75
100
125
150
175
200
#clusters
Figure 4: Consistency score v.8. various number of clusters with GPT-4 and À ==
latency, results for Silhouette and X-means not shown on large-scale datasets. After sampling 16 data pairs, we tend to choose positives with finer granularity or negatives with coarser granularity. We also consider the sentence length to minimize the cost. We use label names as justifications and we always put 2 positive before negative (See Table 12 bottom).
More Details about Determining Cluster Granularity
Previous methods often employ clustering errors as a metric and they ignore user's 's need on the granu- larity. Silhouette coefficient (Rousseeuw, 1987) in- dicates the clustering quality without ground truths, which exploits the inter-cluster distance with near- est clusters and the intra-cluster distance. We find the granularity by choosing the one with the best silhouctte coefficient. Elbow method (Thorndike, 1953) is heuristic method that plots the clustering error with respect to different levels of granular- ity in the hierarchy. And then the best granular- ity is determined with the largest elbow length. X-means (Pelleg et al., 2000) is a variation of K-means that starts with the lowest number of clusters, and then repeatedly attempt to split the clusters by running 2-means on them and evaluate with Bayesian Information Criterion (BIC) (Goutte al., 2001). BIC (Goutte al., 2001) calcu- lates BIC for each of the granularity. Cluster- Size (Zhang et al., 2021b) uses confidence thresh- old to filter small clusters starting from the maxi- mum number of cluster. For all methods, we use the same fine-tuned embeddings LLM in Table 2). The same cluster hierarchy is used (ex- cept for X-means that relies K-means), which is either acquired from hierarchical clustering for small-scale or our proposed two-step method in Section 3.2 for large-scale. For out methods, the weight F-beta score is set to 0.92 through em- pirical selection on B7nk77. Because of the high
C Analysis for Determining Granularity
Prompt Design. We show the analysis results of prompt design for determining granularity Ta- ble We experiment with two settings: (1) remove justification for all demonstrations and only keep the "Yes" or "No" answer. (2) remove all demon- strations and any granularity-related words (such as d We observe that demonstrations necessary and adding justifications have positive impact.
Visualization of Consisteney Score. We visualize consistency score with respect to the number clusters. The consistency scores exhibit continuous variations and peak at the best number of clusters.
D Details of Embedders and Fine-tuning
For all the experiments (including those with or without fine-tuning), we use large version of both Instructor and ES (i.e. intfloat/es-larg).
wkunlp/instructor-larg For Instructor, we use the same or similar prompt original paper. See Table 10.
&
adopt the same hyper- parameters in (Su al., 2022), but modify the
For fine-tuning,
AAtter removing, the prompt will be in the format "De- termine whether the two sentences helow helong to the same category."

---

learning rate 2e 6, the maximum gradient steps 3, 840 for Instructor (~ 15 epochs) and 1, 280 for E5, and batch size to We choose this gra- dient in the begining of our experiments by ob-
1.00
Relative Clustering Accuracy
0.95
0.90
serving no performance increase after that on sev- eral datasets. Training is conducted with single
0.85
Bank
Bank?
NVIDIA Quadro RTX 8000 GPU.
Fewdel
For SCCL-I(E), we change the maximum token length 128 due to the limited compute resource'. We use the same learning rate 2c as before for Instructor and 2e 7 for E5 since we found that the performance is unstable with large learn- ing rate. Batch size is set to 16 and we evaluate representations with K-means after 200 iterations. Also notice that we do not interrupt prompts in Instructor during data augmentation.
0.80
StackEx
GoEmo
0.75
1
Iteration
Figure 5: Relative clustering accuracy (divided by max- imum for better aligning across datasets) of CLUSTER- LLM-GPT3.5 over iterations.
entropy of class distribution Table 9.
E Description of Datasets
F
Results of More Iterations
Bank77 (Casanueva et al., 2020) is a popular dataset in intent discovery that focuses on creating fine- uieeggrinedd intent categories for single-domain, "banking". CLINC(I) (Larson et al., 2019) is orig- inally created for detecting utterances that falls outside of supported intents. The dataset also con- tains multiple domains, such as "travel", "utilitt" and "work". In this experiment, we discard all the out-of-scope utterances and only focus on in- domain ones. Moreover, we create a domain dis- covery dataset CLINC(D) that uses domains as labels. Mas) (FitzGerald et al., 2022) and MTOP()/() (Li et al., 2021) are both from MTEB (Muennighoff et al., 2022). Here "I" de- notes intent and "D" for domain (or scenario). These datasets are originally used for classifica- tion but are adapted here for clustering. We also remove those intents with only a few instances and keep English-only data. For all datasets, we use the train & test sets as large- & small- scale datasets respectively. For FewRel (Gao et al., 2019) and FewEvent (Deng et al., 2020), first randomly split datasets into train & test sets, and then sample from train set as large-scale and test set as small- scale. For FewNerd (Ding et al., 2021), we use the original train test splits. For StackEx, Red- dit (Geigle et al., 2021) and ArxivS2S, we combine all the splits into a single dataset and remove topics that only have few instances. Finally, the datasets are randomly splitted into large- & small- scale versions. To show the dataset balancy, we show the
We show the results over 1 iterations of CLUS- TERLLM in Figure 5. During iteration, sam- ple triplets from previously fine-tuned embedding space and continue fine-tune the model with pre- vious checkpoint as initialization. We also show the self superviee results using the same checkpoint fine-tuned with GPT-3.5 predictions as initializa- tion at each iteration. We observe that using GPT- 3.5 predictions is almost always beneficial. The performance generally increases and saturate at the fourth iteration with the exception of GoEmo.
G More Related Works
Generalized Category Discovery (GCD). GCD (Vaze et al., 2022; Lin et al., 2020; Zhang et al., 2021b, 2022; Mou et al., 2022; An et al., 2022) assume partial known classes with annotations which can also be used to infer user's requirement on clustering. As an infant research area, most previous works employ pseudo-labelling, via optimal transport (Rizve et al., 2022b; Yang al., 2022), similarity learning (Rizve al., 2022a: Cao et al., 2022) or prototype-based learning (Sun and Li, 2022). Furthermore, new intent discovery (Zhang et al., 2022, 2021b, 2023; An et al., 2022; Lin et al., 2020) is proposed to study a similar research problem in the domain of intent detection. Most recently, Hogan al., 2023 adapts the setting into relation type discovery. However, GCD relies on sufficient annotated and unlabeled data for training. In contrast, CLUSTERLLM seeks for minimal supervision and studies a setting with controlled
"Wo also tried maximum token length 512 with smaller batch size, but the performance is not better.

---

Intent Discovery
Emotion
Method
Bank77
CLINC()
(TOP()
Massive(()
Golimo
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
Intre
20.30
78.37
79.52
92.63mxz:
35.53
70.78ma
54.72zm
72.29kaan
24.02
20.15
1el1 sgeride
21.48
u9
81.87
93.55ma
35.27
71.88km
38.30
73.73
24.34
21.17aw
65.47am
81 600mm
82, 29
93.91am
36.80
73. 16
57.70kmm
74.24
25.23. m
22. 49
CLUSTTLLLM OTAGPT3
67, 12
82.46mm
80.90
93.80
74,51am 37.75
58.92n.
75.07
27.96am
25. 25.90ten
Type Discovery
Topic Mining
Method
FowRel
FowNerd
FowEvent
StackEx
Reddit
ACC
ACC
NMI
ACC
ACC
NMI
ACC
NMI
Istrox
41.38mm
53,97
29. 19.62mm
42.83mmm
41,42mm
61. 13.13
46. 86
61.20km
35.00cm
59.5 59.55ic
sull opervi
099m
54.02
30.57
43.64
45.54pm
46.24
60.78
56.45, v
60.1
(CussatLLM-GFT GPT35
47.2
59.87
33,86
46.91
47.55mmm
70.21am
47.4 47.42
61. 30
45,47
58,73
CCuennt LM GTACEPTS
61.20
44.85
48.59
46.90
71.51mm
12
62.04.
57.95. s
61.39km
Topic Mining
Domain Discovery
Avg
Mothod
ArxivS2S
MTOP(D)
Massive(D)
CLINC(D)
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
24.55mm
39.86ymm
g5
83,96
56. Tism
61.86
51.74px0
55.28
48.98
60,99
24.49mm
40.70
89.54.u5
86.69
64.49
50.93
53.01
50.30
61,98
25.60as
41. 41.50
84.08
84.57am
58. 14.m0
65.50
50.12as
53.46
51.21
63.37
25.29
41.81
84.99.am
86.59
58.55
65.66
$7.08
58.35
52.55
64.92
Table 6: Ablation study on clustering quality for large-scale datasets.
Method
Bank77
FewRel
Massive(I)
Massive(D)
MTOP()
MTOP(D)
CLINC()
CLINC(D)
Rank
GTT Relisters
77
64
59
18
102
11
150
10
Elbow menees
50 (35.06)
38 (10:63)
47 (20.34)
45 (150.0
35 (65.69)
35 (218.2)
64 (57.3)
71 (610.0)
BIC
183 (137.7)
183 (185.9)
148 (150.8)
141 (683.3)
169 (65.69)
170 (1445)
179 (19.33)
178 (1680)
Cluster Size Steg
90 (16.88)
95 (23.38)
79 (33,90)
81 (350.0
84 (17.65)
(654.5)
105 (30.00)
109 (990.0)
(,
(2.60)
32 (50.00)
122 (106.8)
55 (205.6)
(50.00)
(18.18)
143 (4.67)
118 (1080)
Ours 5,
79 (2.00)
52
108 (83.05)
54 (200.)
61 (40.20)
20 (81.81)
145 (3.33)
118 (1080)
Ours (GPT44A
64 (16.88)
40 (37.50)
58
16
(76.47)
8 (27.27)
143 (4.67)
32 (2200)
(GTTA
59
79
61
17 (5.56)
27 (73.53)
11
148 (1.33)
32 (2200)
Ours (GI,) 33
77
78 (21.88)
54 (8.47)
37 (105.6)
19 (81.37)
11
148 (1.33)
32 (2200)
Table 7: Inferred granularity on large-scale datasets. The setting is the same as in Table 5.
small-scale
large-scale
Method
Massive())
Massive(D)
CLINC()
CLINC(D)
Massive(D
Massive(D)
CLINC(I) CLINC(D)
GT
18
150
10
59
18
1.00
10
Ours
20
146
39
58
16
143
w/o Justification
50
137
41
80
36
129
w/o Demonstration
64
141
108
77
120
105
Table 8: Prompt designs in determining granularity. We use the Instructor embedding with prompts, and we report results of GPT-4 with À= I.
(a) MasciverD)- Inttrction
(b) MassivedD) - CUSTTRLLMM-E
(c) CLINOD) Inarackor
(d) CLINCD) C-issre
Figure 6: Scatter plots for t-SNE of embeddings. We select 10 classes from each datasets, denoted by colors.

---

Task
Name
entropy(small)
carropy(large)
tends be more compact than before and forming small cliques, making it better for clustering fine- grained clusters but not for coarse-grained clus- ters. We showcase scatterplots on two datasets with both Instructor and CLUSTERRLLM-IIiter It can be observed that the clusters in embedding space are tighter and more separated especially on CLINC(D).
Bank77
1.00
CLINC()
1.00
Intent
МТОР()
0.74
Massive(D)
0.91
FewRel
1.00
1.00
Type
PewNerd
0.82
0.82
FewEvent
0.85
0.85
StackEx
0.98
0.98
Topic
Arxiv22S
1.00
1.00
Reddit
1.00
1.00
Emotion
GoEmo
0.91
0.91
Dataset Leakage
CLINC(D)
1.00
1.00
Domain
MTOP(D)
0.98
0.98
Massive(()
0.94
0.94
Since LLMs like ChatGPT are trained on web- scraped texts from internet, it is likely they already have access to our evaluation datasets during train- ing. For example, the topic mining datasets use StackExchange, Reddit and Arxiv tags labels which is freely available online. However, ob- served Table 3, the performance of triplet pre- diction on these datasets are often far from perfect. Furthermore, the other datasets like Bank77 are synthesized datasets which not accessible dur- ing training. FewRel is collected from Wikipedia corpus but their labels are not easily accessible. Similarly, GoEmm is collected from Reddit but the emotion labels not accessible during training. Thus. dataset leakage not primary concern of this paper.
Table 9; Entropy of class distribution.
Dataset
Prompt
Bank?7
Kepresent the bank purpose for retrieval:
CLINCO)
Represent sentence for retrieving parpose:
FewRel
Represent the relation between two enities for retrieval:
Nend
Represent the entity type for retrieval:
Fewn
Represent event type rerrieval:
StackEs
Represent the question for retrieval:
AnxisS2S
Represent the science staternent for retrieval:
GoFmo
Represent
sentence
Massive()
Represent the
for retrieving the purpose:
MTOP()
Represent the sentence for retrieval:
Reddit
represent reddit community title:
MassivetD)
Represent the scene for retrieval:
MTOP(D)
Represent a sentence:
CLINC())
Represent a sentence:
Table 10: Prompts for Instructor.
computation- & data- cost.
LLMs Annotators. Recent instruction-tunee LLMs, such as ChatGPT, have been shown to have the ability reproduce or improve human- generated labels (Gilardi et al., 2023; He et al., 2023; Zhu et al., 2023). Furthermore, several works dedicate to mune models with feedbacks from LLMs (Cheng et al., 2023; Bai et al.. 2022). This paper instead focuses on clustering tasks.
H Sub-optimal Performance on Domain Discovery
We noticed that the performance of domain dis- covery (MTOP(D), Massive(D) and CLINC(D)) is barely improved or even decreased with CLUS- TERLLM from original embedders (see Table 2). Furthermore, the ablation studies reveal that even with CLUSTERLLMM GT&GPT3.5, clustering per- formance is not good self-supervis or CL.US- TTERLL--rande (sec CLINC(D) in Table 4 and MTOP(() in Table 6). We also observe that, CLUS- TERLLM-I-ite will further decrease the perfor- mance (see Massive(D) in Table 2). While we do not have rigorous explanations, one hypothe- size is that the embedding space after fine-tuning

---

Dataset
Prompl
Bank77
Select the banking customer utferance that hetter corresponds with the Query in terms of intent.
CLINC()
Select the customer utterance that better corresponds with the Query in terms of intent.
FewRel
Select the example that better correpoonss with the Query terms relation type.
FowNerd
Select the example that better corresponds with the Query in terms of entity type.
FewEvent
Select the example that better corresponds with the Query in terms of event type.
StackEx
Select StackFxchange question that better corresponds with the Query in of topic. Select the Arxiv paper title that better corresponds with the Query in terms of domain.
ArxivS2S
GoEmo
Select the sentence that better corresponds with the Query in terms of emotion expressed.
Massive())
Select the user utterance that betler corresponds with the Query in
of intent.
МТОР()
Select user utterance that better corresponds with the Query terms of intent
Reddit
Select the Reddit question that better corresponds with the Query in terms of topic.
Massive(D)
Select the utterance that betler corresponds with the Query terms of scenario.
MTOP(D)
Select user utterance that betler corresponds with the Query terms of domain.
CLINC())
Select the customer utterance that better corresponds with the Query in terms of domain.
Table II: Prefix prompts for triplet task. Notice while we use different prompts for domain and intent (such as CLINC(D and CLINC(D) in experiments, they might be used interchangeably.
Dataset
Prompt
Select the banking customer that better corresponds with the Query terms of intent.
Query: Should reinstall the payment app?
Triplet Task
Choice I: I've received my card now need to know how syne to the app.
Choice 2: Can use the app switched phones?
Please respond with 'Choice I' or 'Choice 2 without explanation. [Examplel]
Sentence I: would like to see the source of my money.
Sentence 2: My source of funds need verified.
Yes. Because hoth intents verify source funds.
[Example2]
Sentence Is there fee for tupping up
Sentence What are the top up charges for US cards?
Yes. Because both intents are top up by card charge.
[Example3]
Sentence 1: Can reactivate my lost card that found this morning in my jacket pocket?
Sentence 2: how to activate card?
Pairwise Task
No. Because Sentence has intent card linking and Sentence has intent activate my card.
[Fxample4]
Sentence I: What will I be charged for physical card?
Sentence My card about to expire need know how much it costs and how long
No. Because Sentence has intent order physical card and Sentence 2 has intent card
Determine whether the intents of banking customer utterances below belong to the same intent category using above examples.
Sentence I: $1 extra has been charged my statement, why is that? Sentence it automatically top-up if there isn't much money left?
Please respond with 'Yes" or 'No' without explanation.
Table 12: One example from Bank77 on both triplet task and pairwise task.


Reviews:
[{"review_id": "Uit0vtYxtQ", "reviewer": "Reviewer_s3g2", "summary": "", "questions": "1) The evaluation of the second step (cluster granularity) is rather short. It would have been interesting to see, how well the frameworks perform in the setting when k is unknown. Did you evaluate the clustering accuracy for the models presented in Table 5?\n\n2) Did you perform a qualitative analysis of the merging step and when it stops? Maybe a simple top words per cluster approach would be helpful to visualize, how and why clusters are merged together (or when the merging stops).", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments and feedback. We are glad to see that you notice the novelty of this work is to provide an efficient way to query LLMs and train models. We address your concerns here.\n\n> The authors often discuss results presented in the Appendix in the main paper (standalone document). Section 4.4 gives an overview of the results, but it would be more beneficial to avoid going into detailed discussions of datasets that are solely listed in the appendix, such as CLINIC(D) and Massive(D). Keeping all crucial information within the main body improves the paper's readability and accessibility. It also seems like the related work section 'Clustering' should be a part of the main paper.\n\nAnswer: As also pointed out by Reviewer vYLX and Reviewer odh6, the paper presentation needs to be significantly improved in order to increase readability. The authors want to apologize for the inclarity. And given one more page, we would like to improve especially on the following aspects: (1) in order to present holistic results across datasets, the splitted tables will either appear on the same page or merged together. We will also avoid discussing Appendix in the main paper; (2) both clustering accuracy and NMI will be presented for all datasets and models in the main results; (3) the paper will be rearranged in a better fashion to include the most important sections into the main paper; (4) the authors will meticulously go over the paper and correct grammatical errors.\n\n> Evaluating more than one clustering metric would be vital for analysing the results. The authors only present the NMI results for a subset of models on the last page of the appendix, however, the tables in the main section should also contain NMI in addition to the clustering accuracy.\n\nAnswer: As stated above, the authors will include both metrics for all datasets and models in the main results.\n\n> The experiments concerning the second step of the framework (cluster granularity) are relatively brief in length.\n\nAnswer: As also pointed out by Reviewer vYLX, the proposed method for cluster granularity is under-explored. Reviewer s3g2 suggested providing more analysis. Reviewer vYLX suggested moving these results to appendix or a future work. The authors generally agree that the main focus of this paper is inclined to the first step (perspective of clustering). But we would also emphasize the importance of determining cluster granularity especially for those applications that desire both fine-grained and coarse-grained clustering. Overall, the observation of this paper is that, when presented with in-context examples, LLMs seem to capture the granularity information according to the results on Massive(I/D), MTOP(I/D), CLINC(I/D) in Table.5. However, the authors also agree that more analysis should be done in order to better understand the potential of the method. In particular, we will include visualizations as suggested by Reviewer s3g2 and more ablation studies on the prompts (similar as presented in Appendix.D).\n\n> The evaluation of the second step (cluster granularity) is rather short. It would have been interesting to see, how well the frameworks perform in the setting when k is unknown. Did you evaluate the clustering accuracy for the models presented in Table 5?\n\nAnswer: Clustering accuracy is not only dependent on a correct estimation of granularity, but also the clustering quality. An interesting phenomena the authors observed is that, when provided with ground truth number of clusters, the performance may not be maximized for both clustering accuracy and NMI. To give an example, given ground truth number $59$ for Massive(I), the clustering accuracy is $59.35$. However, the clustering accuracy is increased to $63.69$ when provided with the number of clusters $41$ which is smaller than ground truth. The possible reasons could be (1) clustering metrics are sensitive to an imbalanced dataset; (2) clustering quality is not optimal. Hence, the authors prefer to separate the evaluations of clustering quality and clustering granularity estimation in the main results. But we will include these metrics in analysis.\n\n> Did you perform a qualitative analysis of the merging step and when it stops? Maybe a simple top words per cluster approach would be helpful to visualize, how and why clusters are merged together (or when the merging stops).\n\nAnswer: This is a good point. These visualizations can potentially help readers understand how LLM predictions look like and will be included in the next version as a case study. Furthermore, we have plotted the consistency score against the number of clusters which shows a continuous curve that peaks at a certain number. We can also observe the range of high scores on this plot and correspond to the keyword result analysis.\n\n> As mentioned earlier, the paper should be self-contained. One should not discuss results presented solely in the appendix in the main section. Tables 5 and 11 seem to be mixed up. Figure 3 is never discussed in Section 4. Maybe also just a question of style, but it seems the 'one paragraph' Section 2 should be merged into Section 3.\n\nAnswer: As stated above, the authors will try our best to make the paper self-contained in the next version. We will also add references to Figure 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper exhibits a well-structured and compelling organization, effectively presenting its motivation and research content.\n\n2) The authors introduce a novel entropy-based sampling technique to decrease the number of LLM queries needed, making their framework more feasible and efficient.\n\n3) The authors conduct comprehensive experiments demonstrating how their framework outperforms other text clustering methods. The qualitative analysis provides insights into the operational behaviour of the framework.\n\n4) Figures are meaningful and well-designed", "reasons_to_reject": "Some (minor) points I want to list here:\n1) The authors often discuss results presented in the Appendix in the main paper (standalone document).\n Section 4.4 gives an overview of the results, but it would be more beneficial to avoid going into detailed discussions of datasets that are solely listed in the appendix, such as CLINIC(D) and Massive(D). Keeping all crucial information within the main body improves the paper's readability and accessibility. It also seems like the related work section 'Clustering' should be a part of the main paper.\n\n2) Evaluating more than one clustering metric would be vital for analysing the results. The authors only present the NMI results for a subset of models on the last page of the appendix, however, the tables in the main section should also contain NMI in addition to the clustering accuracy.\n\n3) The experiments concerning the second step of the framework (cluster granularity) are relatively brief in length.", "questions_to_authors": "1) The evaluation of the second step (cluster granularity) is rather short. It would have been interesting to see, how well the frameworks perform in the setting when k is unknown. Did you evaluate the clustering accuracy for the models presented in Table 5?\n\n2) Did you perform a qualitative analysis of the merging step and when it stops? Maybe a simple top words per cluster approach would be helpful to visualize, how and why clusters are merged together (or when the merging stops).", "paper_topic_and_main_contributions": "The authors present a two step fine-tuning framework that enhances the clustering performance of an embedding model. They optimize smaller models, such as Instructor and E5, using feedback from larger language models like ChatGPT. In the first step, the clusters are refined through the use of triplet queries, asking ChatGPT if A is closer to B than to C. In the second step, a cluster hierarchy is established, by querying whether A and B belong to the same category.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "As mentioned earlier, the paper should be self-contained. One should not discuss results presented solely in the appendix in the main section. Tables 5 and 11 seem to be mixed up. Figure 3 is never discussed in Section 4.\nMaybe also just a question of style, but it seems the 'one paragraph' Section 2 should be merged into Section 3."}}, {"review_id": "PgNoOdpkB1", "reviewer": "Reviewer_vYLX", "summary": "", "questions": "1. Did you try any other metric for sampling triplets apart from random or entropy-based? The observation the “random sampling inputs for LLM annotations is not optimal/might even hurt performance” seems to be common across multiple works.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments and feedback. We are glad to see that you notice the advancements we wish to provide on clustering literature. We address your concerns here.\n\n> The cluster granularity aspect of the algorithm seems under-explored/motivated and I couldn’t understand it well enough. Probably the paper would be better written as solely a clustering paper inline with prior work and exploration on granularity could come in appendix or as a potential future work.\n\nAnswer: The motivation is that predicted pairwise relations from the LLM can be used to represent granularity of clustering. A set of pairs are first sampled from hierarchical clustering, and then predicted by LLM with binary “Yes/No” answers indicating whether a pair of sentences belong to the same cluster. We then compare these pairwise predictions against each level of granularity in hierarchical clustering to acquire the granularity with the highest consistency (F-beta score in the paper). To be more specific, each granularity is itself a clustering of the dataset. To calculate consistency, we count the number of predicted pairs that have the same assignments as current clustering of the granularity. The authors hope to convey a simple message that at least there exists a way to automatically estimate granularity with a proper prompt. As stated in the response to Reviewer s3g2, we would like to include more analysis in this work and we also encourage more future works that can bring more insights.\n\n> Grammatically the paper can be improved a lot. Hope the authors will read thoroughly and rectify these mistakes.\n\nAnswer: As also pointed out by Reviewer s3g2 and Reviewer odh6, the paper presentation needs to be significantly improved in order to increase readability. The authors want to apologize for the inclarity. And given one more page, we would like to improve especially on the following aspects: (1) in order to present holistic results across datasets, the splitted tables will either appear on the same page or merged together. We will also avoid discussing Appendix in the main paper; (2) both clustering accuracy and NMI will be presented for all datasets and models in the main results; (3) the paper will be rearranged in a better fashion to include the most important sections into the main paper; (4) the authors will meticulously go over the paper and correct grammatical errors.\n\n> Did you try any other metric for sampling triplets apart from random or entropy-based? The observation the “random sampling inputs for LLM annotations is not optimal/might even hurt performance” seems to be common across multiple works.\n\nAnswer: Triplet sampling is crucial in the literature of deep metric learning (DML) [Wang et al, 2014; Schroff et al, 2015; Wu et al, 2017; Xuan et al, 2019]. However, most of their observations are based on the fact that categories (or positive pairs) are already provided. One insightful finding is that using diverse anchors is better than randomly selected anchors [Sumbul et al, 2021]. Thus the authors tried to sample $20$% anchors from each cluster and keep the total number of anchors the same for ClusterLLM, but the results on 3 datasets (Bank77/FewRel/StackEx) did not show significant improvement. Given one more page, we would like to include more discussions on this issue. Additionally, it would be very helpful if you are willing to share several references for the claim “random sampling inputs for LLM annotations is not optimal/might even hurt performance seems to be common across multiple works.”\n\n[Wang et al, 2014] Learning Fine-grained Image Similarity with Deep Ranking\n\n[Schroff et al, 2015] FaceNet: A Uniﬁed Embedding for Face Recognition and Clustering\n\n[Wu et al, 2017] Sampling Matters in Deep Embedding Learning\n\n[Xuan et al, 2019] Improved Embeddings with Easy Positive Triplet Mining\n\n[Sumbul et al, 2021] Informative and Representative Triplet Selection for Multilabel Remote Sensing Image Retrieval\n\n> Spliting main result tables into multiple tables creates a bit of confusion for the reader. I found myself lost in table numbers and flipping pages. Also would be good to mark Supp tables as \"Supp. Table 5\" to make it explicit.\n> \n> In general i feel the authors can do a better job of presenting the results in a more streamlined fashion with clear agenda contained within the page limits of the paper which wouldn't require the reader to necessarily refer the appendix.\n\nAnswer: As stated above, we will present results in a self-contained fashion in order to increase readability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper advances  the current clustering literature by leveraging the recent developments in instruction-tuned LLMs.\n\n2. Interestingly, they show how naive random sampling for triplet queries might not lead to better clustering performance and how entropy based triplet selection can help alleviate this issue. \n\n3. Empirically the paper shows gains over current SOTA across multiple datasets and includes a thorough analysis.", "reasons_to_reject": "1. The cluster granularity aspect of the algorithm seems under-explored/motivated and I couldn’t understand it well enough. Probably the paper would be better written as solely a clustering paper inline with prior work and exploration on granularity could come in appendix or as a potential future work. \n\n2. Grammatically the paper can be improved a lot. Hope the authors will read thoroughly and rectify these mistakes.", "questions_to_authors": "1. Did you try any other metric for sampling triplets apart from random or entropy-based? The observation the “random sampling inputs for LLM annotations is not optimal/might even hurt performance” seems to be common across multiple works.", "paper_topic_and_main_contributions": "The paper presents a novel algorithm ClusterLLM which leverages LLMs for improving text clustering. Given a text embedder, ClusterLLM prompts LLM for correct assignment of points to clusters and fine tunes the embedder based on this information. It considers 2 broad tasks for it : triplet querying which aims to capture the clustering semantics/perspective and pairwise query which deals with the granularity within a cluster.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Spliting main result tables into multiple tables creates a bit of confusion for the reader. I found myself lost in table numbers and flipping pages. Also would be good to mark Supp tables as \"Supp. Table 5\" to make it explicit. \n\nIn general i feel the authors can do a better job of presenting the results in a more streamlined fashion with clear agenda contained within the page limits of the paper which wouldn't require the reader to necessarily refer the appendix."}}, {"review_id": "QcZShzBGRu", "reviewer": "Reviewer_odh6", "summary": "", "questions": "-In your results tables you state that some results of the compared models are done with BERT instead of the pre-trained embedding models used for the presented technique. If that is the case, isn't it quite obvious that any comparison between K-Means/ClusterLLM using E5/instructor and the GCD methods  using BERT are unfair? How do the GCD methods perform when using the same embedding model as ClusterLLM?\n\nI am unsure about the advantages and differences between the presented approach and any few-shot model trained with labeled documents from ChatGPT. Consequently, the benefits of using the triplet technique are not entirely clear to me. Could you please clarify or do you maybe have already performed experiments on that regard?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments and feedback. We address your concerns here.\n\n> Poorly written, with many grammatical errors and illogical sentences.\n\nAnswer: As also pointed out by Reviewer s3g2 and Reviewer vYLX, the paper presentation needs to be significantly improved in order to increase readability. The authors want to apologize for the inclarity. And given one more page, we would like to improve especially on the following aspects: (1) in order to present holistic results across datasets, the splitted tables will either appear on the same page or merged together. We will also avoid discussing Appendix in the main paper; (2) both clustering accuracy and NMI will be presented for all datasets and models in the main results; (3) the paper will be rearranged in a better fashion to include the most important sections into the main paper; (4) the authors will meticulously go over the paper and correct grammatical errors.\n\n> Not sure how a few-shot method (e.g. SetFit) would compare with documents \"labeled\" from GPT without the triplet technique. Benefit of the method is not quite clear to me.\n\nAnswer: The results on few-shot methods do not serve as a direct comparison with ClusterLLM since our method does not have access to annotated data. As stated in Line 389-393, we wish to show that a better pre-trained encoder is worth hundreds of annotated data which may not be widely known in the area of especially intent discovery. However, it is a good point that perhaps such a comparison is not the focus of this paper, and therefore should better be moved to Appendix. Additionally, we will include the results of SetFit and different embedding models in the next version.\n\n> The fairness of the experimental comparison and chosen models is unclear.\n\nAnswer: The main baselines of this paper are the self-supervised method and SCCL (Line 398-410). We would like to show that with carefully sampled triplets, we can improve clustering performance with a limited amount of LLM predictions. The self-supervised method uses the same triplets as ours but annotated by the embedding itself. SCCL is a well-known clustering method relying on contrastive learning. It is interesting to see that with only 1,024 triplets predicted by LLMs, it can already significantly improve clustering performance while also outperforming these baselines on most datasets (see Table.2 and discussions in Section 4.4).\n\n> -In your results tables you state that some results of the compared models are done with BERT instead of the pre-trained embedding models used for the presented technique. If that is the case, isn't it quite obvious that any comparison between K-Means/ClusterLLM using E5/instructor and the GCD methods using BERT are unfair? How do the GCD methods perform when using the same embedding model as ClusterLLM?\n> \n> I am unsure about the advantages and differences between the presented approach and any few-shot model trained with labeled documents from ChatGPT. Consequently, the benefits of using the triplet technique are not entirely clear to me. Could you please clarify or do you maybe have already performed experiments on that regard?\n\nAnswer: As stated above, our aim is not to compare with few-shot models, but rather to show the importance of pre-training. However, we are willing to show this result in the Appendix and leave space for more important results.\n\n> Line 205 -> hypothesis instead of hypothesize Line 210 sample two data ...?\n> \n> ... a lot more grammatically missing words in sentences, e.g. lines 73, 388, 412 and many many more…\n\nAnswer: As stated above, we will meticulously go over the paper and correct these errors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Overal sound idea\n- Good experimental results", "reasons_to_reject": "- Poorly written, with many grammatical errors and illogical sentences.\n- Not sure how a few-shot method (e.g. SetFit) would compare with documents \"labeled\" from GPT without the triplet technique. Benefit of the method is not quite clear to me.\n- The fairness of the experimental comparison and chosen models is unclear.", "questions_to_authors": "-In your results tables you state that some results of the compared models are done with BERT instead of the pre-trained embedding models used for the presented technique. If that is the case, isn't it quite obvious that any comparison between K-Means/ClusterLLM using E5/instructor and the GCD methods  using BERT are unfair? How do the GCD methods perform when using the same embedding model as ClusterLLM?\n\nI am unsure about the advantages and differences between the presented approach and any few-shot model trained with labeled documents from ChatGPT. Consequently, the benefits of using the triplet technique are not entirely clear to me. Could you please clarify or do you maybe have already performed experiments on that regard?", "paper_topic_and_main_contributions": "The paper introduces CLUSTERLLM, a text clustering framework that utilizes feedback from an instruction-tuned LLM. ClusterLLM  leverages the emergent capabilities of ChatGPT even when its embeddings are inaccessible. The authors prompt ChatGPT to gain insights into clustering perspectives using hard triplet questions and to help with clustering granularity using pairwise questions.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 205 -> hypothesis instead of hypothesize\nLine 210 sample two data ...? \n\n... a lot more grammatically missing words in sentences, e.g. lines 73, 388, 412 and many many more..."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are generally agreed on both their feelings of soundness and excitement, giving high ratings for both categories. The authors are addressing a practical challenge, using methods whose performance will be interesting to many other researchers.

Concerns about clarity recur across the reviews, and this will be important to address for a final version of the paper. Some sections of the paper were confusing to the reviewers and needed additional clarification from the authors during the rebuttal. These confusions were all cleared up, but that additional explanation should be incorporated into the paper.

The reviewers agreed that the experiments were sound and compelling. One reviewer had additional questions about the choice of comparison models, but this was clarified by the rebuttal (and as stated above, these misunderstandings can be easily addressed by clarifying a final version of the paper).
Decision: Accept-Main