Goal-Driven Explainable Clustering via Language Descriptions
Authors: Zihan Wang
Jingbo Shang
Ruiqi Zhong
Venue: EMNLP 2023 Main (2023)
Keywords: clustering
explainability
large language models

Abstract: Unsupervised clustering is widely used to explore large corpora, but existing formulations neither consider the users' goals nor explain clusters' meanings. We propose a new task formulation, "Goal-Driven Clustering with Explanations (GoalEx), which represents both the goal and the explanations as free-form language descriptions. For example, to categorize the errors made by a summarization system, the input to GoalEx is a corpus of annotator-written comments for system-generated summaries and a goal description "cluster the comments based on why the annotators think the summary is imperfect."; the outputs are text clusters each with an explanation ("this cluster mentions that the summary misses important context information."), which relates to the goal and accurately explains which comments should (not) belong to a cluster. To tackle GoalEx, we prompt a language model with "[corpus subset] + [goal] + Brainstorm a list of explanations each representing a cluster."; then we classify whether each sample belongs to a cluster based on its explanation; finally, we use integer linear programming to select a subset of candidate clusters to cover most samples while minimizing overlaps. Under both automatic and human evaluation on corpora with or without labels, our method produces more accurate and goal-related explanations than prior methods.

Goal-Driven Explainable Clustering via Language Descriptions
Zihan Wang ziw224@ucsd.cm edu
Jingbo Shang j jshang@ucsd.ed edu
Ruiqi Zhong
wiigi-zhon@@berkkley.coo edu
Abstract
V User's Goal "I want to cluster based on sentiment"
Unsupervised clustering is widely used to ex- plore large corpora, but existing formulations neither consider the users' goals nor explain clusters' meanings. We propose a new task formulation, "Goal-Driven Clustering with Explanations" (GoALEx), which represents both the goal and the explanations as free-form language descriptions. For example, to catego- rize the errors made by summarization system, the input to GoALEX a corpus annotator- written comments for system-generated maries and goal "cluster the comments based on why annotators think the summary is imperfect." "; the outputs are text clusters each with an explanation ("this cluster mentions that the summary misses important context informa- tion.") "), which relates to the goal and accurately explains which comments should (not) belong cluster. To tackle GOALEX, prompt language model with "[corpus subset] [goal] Brainstorm a list of explanations each rep- resenting cluster." then classify whether each sample helongs to a cluster hased on its explanation; finally, we use integer linear pro- gramming to select subset of candidate clus- ters to cover most samples while minimizing overlaps. Under both automatic and human evaluation corpora with or without labels, our method produces more accurate and goal- related explanations than prior methods.
Legend
sports topic topic
Cluster
*
negative sentiment positive sentiment
Cluster 2
Explanation "Cluster has a negative sentiment Cluster has positive sentiment"
Figure An illustration of our task formulation GOALEX (Goal-Driven Clustering with Explanations), where the input is a set of texts (corpus) and goal, and output constitutes a set of corpus subsets (clusters) each with explanation. Given the goal, successful GoALEX algorithm should cluster based on sentiment instead of topic and for each cluster explain which sam- ples should (not) belong to it.
To address these weaknesses, we propose a new task formulation, GoALEX, "Goal-Driven Cluster- ing with Explanations" (Section 2). As illustrated in Figure the input to the task is a text corpus with multiple attributes (e.g.. sports and arts related texts with different sentiments) and goal descrip- tion in natural language ("clustering based on sen- timent")) " The output of the task constitutes a set of corpus subsets (clusters), each with a natural lan- guage explanation of which text samples should or should not belong to the cluster (e.g. "contains pos- itive sentiment"). The output should satisfy three desiderata: 1) the explanations are goal-related, 2) each cluster is accurately described by its explana- tion, and 3) the clusters should overlap minimally while their union should cover most the corpus.
1 Introduction
Text clustering widely used explore large cor- pora (Aggarwal and Zhai, 2012). However, exist- ing formulations cannot adapt to different users' goals, which might be clustering based on senti- ment, genre, or other properties (Aharoni and Gold- berg, 2020a); result, the desired output is under- specified. Furthermore, since the output clusters are not immediately interpretable, users must man- ually examine the clusters to gain insights. This can be consumnnu,,ng, especially when some clusters are semantically incoherent (Chang et al., 2009).
To tackle GoALEx, we develop three-stage algorithm Poopose-AAsign-Selleee (PAS, Figure 2, Section 3), each designed to address one of the desiderata. At the proposal stage we address the 1st desideratum that the explanation should goal- related; we do this by prompting a language model (LM) to generate a list of goal-related explanations

---

Propose; given a goal and set of texts, propose a list of explanations each representing candidate cluster.
Assign: for each sample, assign it to the supporting candidate explanations.
Select: select clusters such that each sample is "approxxmmtell supported once" via integer linear programming
6. nothing interesting happened
Explenation (): conveys hoppiness
b. erjoyyngg peace of mind ufter yoga Heart braken can't foll asieep
Sample enjoying peoce of mind ofter yogo
Corpus
d. ranry day makes me mildy gloamy
the explonation support sample?
e. !!! My of agreed to mony mel!
Assigrient Matrix (thed variable)
Selection Vector (optimized varianie)
How often each sample is assigned
Assigner
wont to cluster the texts based on Goal their emotions. Generste a list of explonations for condidate clusters.
Yes
(3)
a
Proposer
(0
00
DO
(+)
b
e
(1 conveys happiness
Assignment Mabri
d
d
(1) conveys boredom
Candidate
Explanations
(ii)
conveys sodness
candidate clusters
Selected clusters
(V]
conveys o strong emotion
M
Figure 2: Our AisponseAssinn-Seelee (PAS) algorithm tackle the GOALEX task. Left, Propose: We prompt language model ("proposer") with the goal and a subset of the corpus, obtaining list of explanations for candidate clusters. Middle, Assign: we use a language model ("assigner") to determine whether each explanation supports each sample. Right, Select: we integer linear programming select subset of explanations, ensuring each sample has roughly one explanation, and obtain the selected set of clusters and explanations as our final output.
for candidate clusters based on the goal and a subset of the corpus. At the assignment stage address the 2nd desideratum that the explanations should accurately explain the clusters; we do this by as- signing text samples only to the explanations that support them. At the selection stage we address the 3rd desideratum on maximizing coverage while minimizing overlap; we do this by using integer linear programming to search for subset of can- didate explanations so that each sample is roughly supported once. At last, output the selected ex- planations and their supported samples as clusters.
GOALEX problems from various NLP papers. We compared PAS to previous clustering methods such as LDA and found that PAS's explanations are more accurate and goal-related under human eval- uation. Finally, we applied PAS hierarchically to create progressively finer-grained clusters on OPENGoALEX, inducing taxonomies over dehate arguments, model errors, and customer reviews. Our contributions summarized follows. We introduce GoALEX, novel setting for text clustering that takes into account of user's objec- tives and provides explanations for each cluster. We developed the Propose-Assoe ----in--siinn--- (PAS) algorithm and showed its effectiveness on estab- lished benchmarks.
We benchmarked PAS in two ways: 1) automati- cally evaluating its ability to recover known clusters
from corpora (Section 4), and 2) manually evalu- ating its clusters and explanations on open-ended corpora (Section 5). For automatic evaluation, we first compared PAS to prior methods on recover- ing topic clusters underlying news and DBPedia articles and found that PAS is competitive while additionally providing accurate explanations. To test whether PAS is goal-driven, used an LM synthesize a corpus, SYNGOALEX, where each text has three known attributes: topic, language, and style; PAS effectively adapts to different goals such as "clustering by topic language / style", while prior methods fail catastrophically.
We tested GOALEX to categorize debate points, customer feedback, and model inaccuracies in hierarchical manner to show its potential to help users navigate extensive datasets effectively.
Defining GOALEX
We formalize the input-output space of GOALEx and introduce the desiderata for an output.
2.1 Input-Output Space
The input of GOAL.Xx constitutes
a set of texts X (the corpus);
For open-ended evaluation, we constructed OPENGOAL.Ex, collection of 12 open-ended
'The implementation and data are made public in https: /ggtttu...t

---

string 9 (the goal description); an integer K (the desired number of clusters).
Generate a list of J' explanations for candidate clusters based on the samples.
The output of GoALEX constitutes
where we typically set the maximum T such that the prompt length does not exceed the 75% of the proposer's 's context window size and J' = 8 < T. The proposer would respond with a structured list of J' candidate explanations:
K strings ck: k с [K]. where oj is an explana- tion of cluster; additionally eş needs to be natural language predicate that can be evaluated against an individual text sample;
K subsets of X: Ck C X E [K]; each repre- senting a cluster. Note that goals and explanations can be arbi- trary natural language strings and predicates much more complicated than the ones in Figure 1. See examples in Section 5.
Explanation 1. 61; Explanation J'. cjr-
Figure left shows more illustrative prompt- response pair. Since the proposer's context window is usually not long enough to contain the entire cor- pus, we construct multiple prompts with different subsets from X to allow the proposer "see" many different samples as possible during the pro- posal stage. We sample from the proposer based on different prompts until obtaining J explanations in total. The full prompt is included in Appendix A.
2.2 Desiderata
We list three desiderata for a GoALEx output, which inform our algorithm design Section Goal-Related. The explanations should be goal- related. For example, if user's goal is to cluster based on sentiments, then "has a positive sentiment" is goal-related, while "is about sports" is not.
3.2 Assign Samples to the Correct Clusters
The assignment stage aims determine whether each sample x E X is supported by each explana- tion Cj. We determine this automatically by prompt- ing an LM, which we refer to as the "assigner":
Accurate Explanation. Since each explanation is a predicate, it should have a True evaluation on all samples from its corresponding cluster and False on others. This automatically enforces the clusters to be semantically coherent, where the coherent interpretation is the explanation.
"Predicate: Ej. Text: x. Is the Predicate true on the Text? Yes or No. When uncertain, output No."
We therefore obtain an assignment matrix A E {0, 1} x|xJ, where Anj indicates whether x is sup- ported by the jth explanation." Denote candidate cluster C²² Xx the 5th candidate cluster is thus
Minimal Overlap and Maximal Coverage. The clusters should overlap minimally while their union should cover most of the corpus samples. Ideally, every sample belongs to exactly one cluster.
Cj = {x/ X,A = =1}
(1)
3 The Propose-Assign-Select Algorithm
3.3 Select an Optimal Subset of Clusters
Each section between 3.1 and 3.3 describes one stage of the PAS algorithm (outlined in Figure 2).
The selection stage aims to choose a subset of clusters from J candidate clusters Cý so that each sample belongs to roughly one selected cluster.
3.1 Propose Explanations for Clusters
The proposal stage aims to generate a list of J (around 30-50) candidate explanations ej. We ob- tain them by prompting language model (LM), which we refer to the "proposer" in the remain- ing text, to perform "in-context clustering" based on a random subset of the corpus; concretely, the prompt concatenates T samples from X, the goal g. and a request to generate J' explanations for candidate clusters:
Define the selection vector E 1) to be row vector, where 8j indicates whether CY is selected. Since we require selected clusters, we add the constraint:
11 K
(2)
We introduce a row vector variable m
T N
(3)
Sample I. T1; .aaipll...Smppl T. xT;
Goal: g:
where my counts how many selected clusters in- clude v. An ideal s should result in m, 1 for all
We use to denote candidate explanations, while using ek to denote the final selected explanations
For convenience we also use 2. as an index.

---

2, since my >> 1 implies that at least two selected clusters overlap on æ while my < implies x is "missed" by all clusters. Therefore, we design the following loss function fa to track how much an entry from m diverges from I:
can recover the references. We evaluated PAS un- der two settings: traditional topic clustering and goal goal-driven non-topic clustering. In both settings, we compared 1) the similarity between the refer- ence and the output clusters automatically and 2) the similarity between the explanations manually. We found that PAS is comparable to previous meth- ods for topic clustering and recovers most of the reference explanations; additionally, since PAS is goal -riven, performs significantly better when there are multiple ways to cluster a corpus.
m2)
if my 1, "miss":
fx(ma):
0
if my 1, "ideal";
A(mx - 1) if my 1, "overlap"; (4)
where À is hyper-parameter determining how much overlaps are penalized. To conclude, we
The following sections will present the datasets (Section 4.1), the clustering methods we evaluated (Section 4.2), the evaluation protocol (Section 4.3), and the performance of each method (Section 4.4). In addition, we evaluated the quality of each stage of PAS Appendix F.
will minimize the following loss C for
(()) == fA(m)
(5)
subject to the constraint of Equation 2 and 3. However, is hard to directly minimize this loss as written because it requires searching over dis- crete variables under a piecewise-linear loss. There- fore, we reduce it to an integer linear programming (TLP) problem, which can be effectively solved by existing libraries. To perform the reduction, in- troduce an auxiliary row vector variable a e x| and add the following two constraints
4.1 Datasets
We evaluated on both corpora from prior works for topic clustering and other corpora for non- topic clustering. We considered four datasets: (AG)'s News, (DB))edia, (NYT) News, and (SYN)GOALEX. We use (-) to denote dataset abbreviation in this section.
a x 1 - m, a > X(m - -1),
(6)
(AG)'s News (Zhang et al., 2015) is a news topic classification dataset with four topic clusters: poli- tics, sports, business, and technology.
where > denotes element- wise greater or equal to. To conclude, we will minimize the loss L
(DB)pedia (Zhang et al., 2015) is a corpus of arti- cles classified into ontologies, such as athlete and book, with 14 topic clusters in total.
C-a-1
(7)
subject to the constraints in Equation 2, 3, and 6, which are all linear. We explain our implementa- tion in python code with comments in Appendix B. We refer to one sequential application of propose, assign, and select as one iteration of PAS.
(NYT) News (Meng et al., 2020) is a corpus of New York Times news articles, each with topic label and location label. There are in total topics, e.g., politics, arts, and 10 locations, e.g., France, Italy. We subsampled this corpus so that the topic and location labels are balanced.
In addition to the three stages above, PAS in- volves other auxillary procedures such as 1) run- ning PAS for 5 iterations to cover the entire corpus, and 2) committing each sample to one single cluster when needed. Due to space constraints, we outline other auxiliary steps of PAS in Appendix D.
(SYN)GOALEx To test PAS's ability to clus- ter based on different goals, we synthesized (SYN)GOALEX, which can be clustered based on three different dimensions: Topics, Writing Style. or Language. To synthesize SYNGOALEX, we first designed four values for each dimension, e.g. "Deutshh"""""""
4 Automatic Evaluation
"1 for the Language dimension. Then we took the Cartesian product across three dimensions, obtaining 48=64 value combinations; for example, one combination could be "Language: French, Style: Poem, Topic: Sports". Finally, for each of the 64 value combinations, we prompted Claude-v1.3 to generate 16 text samples condi-
Following the evaluation protocol from prior works, we evaluated PAS by applying it to corpora that are mixtures of known clusters, treating the known clusters and their explanations as the reference so- lutions, and checking how well the outputs of PAS
'efficiently find a reasonable solution empirically even though it is thcoretically NP-Complete

---

tionally on the values, resulting in 1024 samples for SYNGOALEx in total. Therefore, the reference clusters are different if we cluster based on different dimensions, hence penalizing methods that ignore the goals. Appendix C includes more details about the values for each dimension and the prompt we used for conditional generation.
Macro F1 (%)
(AG)
(DB)
(NYT)
(SYN)
Random
27
11
27
LDA
53
51
51
28
ES
86
72
67
96
Instructor
84
82
69
77
PAS
87
71
70
98
Table 1: We compare different methods and PAS for recovering topic clusters and report the macro F1 score for each method, along with a random baseline which assigns each sample to cluster randomly.
The first three datasets might have appeared in the pre-training corpus of apt-3.5-turlle thus raising potential concerns about memorization. We believe our task of proposing explanations on the three datasets did not occur in the pre- pre-training cor- pus, thus justifying the validity of our evaluations. A more detailed justification is in Appendix E.
label
output
F1
Company"
business"
75
" Building"
" architecture"
82
" "nimal"
lakes"
" Plant"
hiology
73
4.2 Methods and Baselines
We compared fours methods: PAS. LDA, E5, and Instructor. For all methods, we set the number of clusters to be that of the reference solution. We use denote method in this section.
Table 2:
We ran PAS on (DB)pedia to cluster based on topics and present its cluster explanations. We abbre- viate each explanation by removing the prefix "has a topic of (e.g.. Company" corresponds to a full expla- nation has a topic Company:") "). The table of all explanations is in Appendix Table 15.
PAS described in Section By default, we used gpt-3. turboo as the proposer and flan-t5-x1 (Chung et al., 2022) as the assigner. We set = 30 and X = 0.5, except for the (DB)pedia dataset and (NYT) News dataset where we set À = 0.3 since they have many target clusters. We additionally require each appear in exactly one cluster using the commitment method deseribed in Section D. LDA (Blei et al., 2003), or Latent Dirichlet Allo- cation, is standard generative probabilistic model that identifies hidden topic clusters in a corpus by assuming that each text a mixture of topics and each topic is a distribution of words.
first match each of the output cluster to a known reference cluster and then compute the similarity of each pair of matched clusters via F1 score. De- noting the 1.th output cluster as CR and the hth reference as Og, we formulate the matching prob- lem as a bipartite matching problem and solve it with Hungarian algorithm, where the edge weight between each pair of reference and output cluster is the size of their overlap. |Öy (Cgl After match- ing finishes, for each pair of matched reference and output clusters C* and Ć we compute the F1 score of predicting whether æ C* based on whether x Ć, and then average across all & to compute the final macro F1 score for evaluation.
Instructor (Su et al., 2022) is contrastively trained on large collection of datasets with annotated instructions; result, it can create specialized text embeddings according the instructions. To perform goal-driven clustering, we rephrased our goal as the embedding instruction. We computed the text embeddings with instructor-xl and then ran K-means clustering.
4.4 Results
We evaluated PAS on topic clustering and goal- driven clustering based other dimensions. All results shown below are the average of 3 trials with different random seeds.
E5 (Wang et al., 2022) is a contrastively trained text embedder on crawled data, e.g. post-comment pairs and annotated data, e.g. NLI. We computed the text embeddings with e5-large and then ran K-means to obtain the clusters.
Recovering Topic Clusters. We first evaluated the clustering methods on recovering topic clus- ters and report the results in Table 1. PAS consis- tently outperforms LDA, PAS slightly outperforms Instructor on (AG), (NYT), and (SYN); on (DB), PAS is underperforming Instructor by 11%.
Appendix includes more implementation de- tails, e.g. what library we used.
4.3 Metries
We follow the standard protocol from Lange et al. (2004) to evaluate a clustering method, where we
To understand why our method does not deliver the best performance on (DB), we manually exam-

---

(NYT)
(SYN)
G and found that running PAS for five iterations improves over one iteration and using an ILP algo- rithm with a positive À improves the performance.
Macro Fi (%)
Location
Language
Style
Random
13
27
LDA
40
83
ES
55
27
27
Instructor
54
25
25
5 Open-Ended Advanced Applications
PAS
76
97
31
PASt
45
While PAS achieves high performance on bench- marks with cluster labels, it does not necessary imply high performance under real applications. Therefore, we constructed OPENGOALEX, col- lections of 12 open-ended realistic GOALEx prob- lems to evaluate PAS. Since these problems do not have cluster labels, we evaluated PAS with the three metrics introduced in Section 2: (1) explana- tion accuracy, (2) goal-relevance, and (3) coverage and overlap. As (1) and (2) require human annota- tors and are hence expensive to conduct repeatedly, used them to test the limit of PAS to inform future research: we applied PAS with the highest quality models under our budgetary constraints and challenged it to generate taxonomics by produc- ing trees of progressively finer-grained clusters on OPENGOALEX. We evaluated PAS quantitatively with human annotators for the first layer of the taxonomy and qualitatively analyzed the rest.
Table 3: We ran PAS to cluster based on non-topic goals and report the macro F1 score. We use gpt-4 as the proposer and 3--tubb as the assigner.
ined the explanation (for one of the three trials) for each output cluster and present in Table along with its matching reference. Overall, the out- puts are similar to the references; the performance drop is mainly because PAS completely missed the "Animal" cluster, since it "merged" it with the "Plant" reference cluster into a "Biology" cluster. Additional evidence on this merging effect can be found in Appendix F. We consider such a mistake benign and hence conclude that PAS is on par with previous state of the art on topic clustering.
Next, we will show that Instructor fails catas- trophically on non-topic based clustering, implying that it has an "inductive bias" to cluster on topics. Recovering Non-Topic Clusters. We now evalu- ale PAS on other goals and report the performance in Table 3 - specifically, clustering based on lo- cations on (NYT) and writing styles or languages on (SYN). Since PAS is goal-driven, it performs significantly better than previous methods.
5.1 OPENGOAL.Ex
To evaluate under real applications, we constructed OPENGOALEEx, a collection of 12 open-ended GoAL.X problems. Each corpus comes from NLP paper or Kaggle website and we annotated with a goal related to the paper. For example: comments for model-generated summaries (Scheurer et al., 2023), with the goal of "cate- gorizing model errors debates why spanking is bad, with the goal of "categorizing the rypes of arguments" (Habernal and Gurevych, 2016)
However, PAS with our default configuration is poor at writing style clustering on (SYN). For- tunately, the performance can improve signifi- cantly by 14% after using more capable mod- els gpt-4 (OpenAI, 2023) as the proposer and 5-tturo as the assigner. We present the output explanations in Table 4 and expect PAS to improve with future better LMs.
Appendix includes all 12 problem descriptions and citations. To reduce reporting bias, we col- lected OPENGOALEX before our experiments.
Sensitivity Study. We conducted a prompt sensi- tivity study in Appendix M and concluded that our method is not sensitive to the prompts we chose for the proposer and assigner. We also conducted dataset sensitivity study on (DB)pedia study how imbalance of classes in the dataset or noisy, out-of- distribution data points would affect our algorithm, and concluded that our algorithm is not especially vulnerable to these noises than Instructor.
5.2 Advanced Application of PAS
To generate a taxonomy for each corpus, we first apply PAS for the entire corpus; then for every output cluster with > 20 samples, we apply PAS again to create finer-grained clusters and output trees of explanations taxonomies; when creating child clusters for a parent cluster, we include the explanation for the parent into the original goal and request the new candidates to be cub-categories the parent's explanation. Here is an example goal
Ablation Studies for PAS. Finally, conducted two ablations for PAS to study the contribution of !) proposing multiple iterations and 2) our selec- tion algorithm. We present the results in Appendix

---

reference
output
F1
"has a writing style of twitter"
"has writing style of instructional or informat.."
45
"has a writing style of screen play"
"has writing style of narrative storytelling... 51
"has writing style of rap"
"has writing style of using rhymes and rhythm"
49
"has writing style of poem"
"has writing style of incorporating foreign lan..." " 34
Table 4: We ran PAS with proposer=gpt-4 -4 and assigner-gpt-3 3.5--33..00 5-turbo to cluster (SYN)GOALEX based on Style. We present the four output explanations and compare them to the references. Although our method is still far from perfect, the PAS is able generate similar explanations for the 2nd and 3rd row.
Sample The bosketball competition is intense
distractor explanation og such that x & Cg-- For each instance, we present three human turk- ers and consider it correct the majority of them choose EK over egz, We include more details for this HIT task and how to generate word-based ex- planations for LDA and Instructor in Appendix J. We ran study with Turkers and found that they can choose the corresponding explanation 80% of the time for PAS, outperforming 56% for LDA (p 10 and 71% for Instructor (p < 10 10-3).5 Relevance. We evaluated how well PAS's S expla- nations relate to the goal and compared them to explanations for LDA and Instructor clusters. For each problem in OPENGOALEX, we randomly sam- pled problem, explanation from PAS's output, and one from baseline approach; we then asked the evaluators choose which explanation is more relevant to the goal, or abstain if they are similar. To ensure reliability and fairness of our evalua- tion, the authors performed evaluations on their own rather than relying on Turkers, since the goals in OPEN are technical and motivated by NLP research; the evaluators whether the baselines or PAS generated each ex- planation: to make the baseline explanations stylis- tically similar to PAS's outputs, we used the D5 system by Zhong et al. (2023) to describe the dif- ferences between each cluster and the rest of the corpus in natural language.
future, al, energy. technology, paper players. body, game, heart, care
is related to technology is related to sports
Figure Explainability evaluation instances. Given the sample z, the evaluator needs to choose which one the explanations (blue, model generated explanation for 2; orange, random different model generated explanation) is more related. Left: top-word based explanations by LDA and Instructor. Right: natural language explanations by PAS.
where the parent's explanation is in bold:
"My goal is to cluster comments for model- generated summaries falling under the following category: whether this comment advises adding omitted details; specifically,... .... For example.... and want to create theenegaiieed cluster descrip- tions that fall under the above category."
We set K 8, À 0.5, proposer gpt-4, and assigner Claude-v1.3 (Bai al., 2022). We allow a sample to appear in multiple clusters so that PAS can see as many samples as possible when creating subcategories. We designed a new prompt template to propose more detailed explanations; see Appendix Figure 9 for more details.
also unaware of
5.3 Quantitative Evaluation
We quantitatively evaluated the first layer of tax- onomy (i.e. the output of the standard GoALEx formulation) based on the three metrics introduced in Section 2. To help the readers interpret our re- sults, we compared PAS to LDA and Instructor.
Table 5 reports the results for direct pair wisc comparisons between PAS and LDA/Instructor: PAS's explanations are more often related the goal compared LDA (p-value 10 3) and Instructor (p-value < 0.05), which are not goal- driven. As a robustness check, two authors inde- pendently reproduced the exact same conclusion. Coverage and Overlap. On average, 66% of the text samples are covered by at least one cluster, and
Explanation Accuracy. If an explanation eg is accurate for cluster Ck. then given sample E C in its cluster, a human should be able to tell whether Ck eg, the explanation for another cluster Cy is more related. We call the tuple (I, ek, e an explainability evaluation instance and show an ex- ample in Figure 3. To sample an instance, we ran- domly sampled a problem from OPENGOAL.EX, sample an output cluster Ck. and then sampled a text sample x E Ck: we then randomly sample a
*Our evaluation weighted each cluster explanation uni- formly, so these results imply that "PAS produces on average more accurate explanations", but not "cach sample is more accurately explained".

---

Baseline
Win (%)
Lose (%)
p-value
clustering, and cannot flexibly adapt to users' goal. Explaining Text Clusters. To explain topic clus- ters, Carmel et al. (2009) proposes to explain each cluster by extracting candidate labels either from text or from Wikipedia; Treeratpituk and Callan (2006) proposes to explain each cluster by selecting candidate labels using information from the clus- ter, the parent cluster, and corpus statistics; Zhang et al. (2018) proposes summarize cluster with group of semantically coherent phrases. How- ever, these solutions are limited, since phrase-level explanations are not flexible enough to describe a complex cluster. Zhong et al. (2022) proposes to explain a text cluster by describing its differences with the rest of the corpus in natural language; how- ever, its explanation usually does not fully cover the entire cluster, while our clusters are explainable by construction during the assignment stage.
Instructor
30
12
1003
LDA
51
13
10-8
Table 5: How often PAS's explanations are more relevant compared to the baselines.
60% of the samples are covered by one and only one cluster; this is one of the key limitations of the current PAS system, as traditional clustering meth- ods such as LDA or Instructor would cover 100% samples exactly once. On the flip side, however, these low numbers might reflect the inherent diffi- culty of producing semantically coherent clusters with K = 8 in goal-driven way; when a sample cannot be supported by any explanation, it might actually be better to explicitly consider it "not cov- ered" our approach does, rather than forcing it a semantically incoherent cluster and creating a delusion of 100% coverage.
Controlling the Clustering Process. We need ad- ditional supervision signals from the users that they can have more control over the clustering pro- cess. Hu et al. (2014) allows the users to shape the clusters by specifying words that should co-occur in a cluster. In the image domain, Open World Classification (OWC) (Shu et al., 2018; Cao et al.. 2021), also known Generalized Category Dis- covery (Vaze et al., 2022), gives the users more control by asking for a few example labels and their example datapoints; for example, given five la- bels and some corresponding images in a CIFARIO dataset (Krizhevsky et al., 2009) (c.g., "automo- bile", "bird", etc), discover the remaining five la- bels on the unlabeld dataset (e.g., "frog", "ship") and classify the entire dataset into 10 labels (Zheng et al., 2022; Zhao and Mac Aodha, 2023; Xic et al., 2023); closest to our work, Wang al. (2023) oper- ates OWC in the text domain. Our work proposes complementary direction and allows the user to control the clustering process with goal descrip- tion, which is more expressive and lightweight.
5.4 Qualitative Analysis
We show one subpart of an example taxonomy of "why spanking is bad" in Figure 4 to obtain a quali- tative understanding of PAS. Most explanations goal-related and they could help the users quickly explore the corpus without inspecting each cluster manually; however, some do not form a coherent taxonomy. For example, the explanation "employs rhetorical questions" is irrelevant to the goal of identifying argument types; additionally, the ex- planation "discusses the cycle of violence" appears both in the first and second levels of the taxonomy and hence should be merged. We present example taxonomies over customer complaints and model errors in Appendix Figure 12 and 13.
6 Related Work
Text Clustering. Most existing text clustering methods first encode each text sample into some vector and then run a clustering algorithm; e.g. one hot bag-of-words encodings and tf-idf (Blei et al., 2003; Aggarwal and Zhai, 2012), or neural word/connexx embeddings (Aharoni and Goldberg (2020b); Wang et al. (2022); Su et al. (2022), inter alia). Using text clustering methods as backbones, many prior works such Luu et al. (2014); Shang al. (2020); Downey al. (2015) apply them hier- archically corpus to produce taxonomies over topics. Nevertheless, previous text clustering al- gorithms do not necessarily produce interpretable clusters (Chang et al., 2009), mostly studies topic
Explaining Patterns via Language. Natural lan- guage can be used to help users explain patterns in text data (Zhong et al., 2022; Singh et al., 2022). With the increasing capability of language mod- els (OpenAI, 2023), they are used to explain more complicated patterns, such as the inner workings of neural networks (Singh et al., 2023; Bills et al., 2023). Our system is closest to D5 developed by Zhong et al. (2023), which describes difference between text distributions in a goal-driven way.
Patterns in other modalities can also be described

---

"orgues that spanking
mendons the chid's perception of love:..
is ineffectie;
[Repeated] "olccussss the cycle vioience:
Goal: categorize reasons why spanking is bad
Corpus: arguments against spanking
Goal-Related] vempleys
ctetonical questions
"comppaes the duration of Impoct between sportiin and non-sponking punisiments: seok dfy the tot aisson Now ast-ppasing cuntorowrs con tow longe Sictin stect saan suasiing for ouaggh oon' tee that sponting jos owain
[Repeated] targues that sponkinn perpetuctes cycle of violence:
spc Eaayy the sagests that cheaaon why spontect bo sisy
own hees Mili dre
coumess dood i speete they right gycn
reprosayy spanting woukd kod coel les up in or werts ouyiee siees"
onyy
focvvses
school context,
*explorres societol implications...
Figure 4: Example taxonomy for arguments against spanking (Habernal and Gurevych, 2016) produced by PAS.
via language. For example, Zhu et al. (2022) de- scribes distribution shifts for image classification tasks and Eyuboglu et al. (2022) describes errors made by vision models. With future advances in multi-modal foundation models, we hope that GoALEx can be extended to cluster images, and potentially sound (Aghajanyan al., 2023) or phys- ical senses (Thomason et al., 2016).
prior work and human evaluation on open-ended GoaLEx problems that we collected, we can more robustly, though not perfectly, establish the con- clusions from our paper. We also did not evaluate our methods under situations where the number of clusters K is large, e.g., K > 50.
Finally, reaching the best performance requires using gpt-4 and claude-v1. 3 as the proposer and the assigner, which might induce a large cost via LM-APIs if one needs to run PAS on a large cor- pus; hope such problem would alleviate in the future if we could use a lighter weight model to approximate the assigner, the cost of compu- tation significantly decreases, or there is a more computationally efficient variant of PAS.
7 Conclusion
We proposed GoALEX, a new formulation for text clustering that adapts to user's goal and outputs explanations for each cluster. To tackle GOALEx, we developed the Propose-Assign-Sellet (PAS) al- gorithm; under automatic evaluation with known references and open-ended applications, PAS can generate accurate and goal-related explanations for the clusters. Finally, we applied GOALEx hier- archically to produce taxonomies over debate ar- guments, customer complaints, and model errors, thus assisting users to explore large corpora. Future works can improve on discovering minority clus- ters, following the goal better, and resolving global inconsistency when applying PAS recursively.
Ethics Statement
The human evaluation is approved by the Institu- tional Review Board.
Acknowledgement
Our work is sponsored in part by NSF CAREER Award 2239440, NSF Proto-OKN Award 2333790, NIH Bridge2A1 Center Program under award (U54HG012510-0) Cisco-UCSD Sponsored Re- search Project, as well as generous gifts from Google, Adobe, and Teradata. Any opinions, findings, and conclusions or recommendations ex- pressed herein are those of the authors and should not be interpreted as necessarily representing the views, either expressed or implied, of the U.S. Gov- ernment. The U.S. Government is authorized to re- produce and distribute reprints for government pur- poses not withstanding any copyright annotation hereon. Ruiqi Zhong is funded by NSF-Simons Theorinet Grant (NSF Award #2031985). We thank members from Jacob Steinhardt's group, Jingbo Shang's group, and Berkeley NLP group for paper feedback.
Limitation
As indicated in Section 5.4, PAS cannot yet con- struct coherent taxonomies. As indicated in Sec- tion 5.3, PAS is far from being able cover all the samples and the clusters have significant overlap. Given these weaknesses, a practitioner should still properly interpret the results of PAS.
Our evaluation is also not universal in scope. Our benchmarks are predominantly in English, and hence our results do no necessarily generalize to other languages. Our dataset OPENGOALEX also implicitly encodes the author's biases for what clus- tering problems are more important than the other, though this a universal problem for any newly proposed benchmark. We hope that with a com- bination of automatic evaluation on datasets from

---

References
Hyung Won Chung, Le Hou, Shayne Longpre. Bar- ret Zoph. Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. 2022. Scaling ceatruction-fincttune language models. arXiv preprint arXiv:2210. 11416.
Charu C Aggarwal and ChengXiang Zhai. 2012. A survey of text clustering algorithms. Mining text data, pages 77-128.
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei- Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zey. 2023. Scaling laws for genera- tive mixed-modal language models. arXiv preprint arXiv:2301.03728.
Doug Downey, Chandra Bhagavatula, and Yi Yang. 2015. Efficient methods for inferring large sparse topic hierarchies. In Proceedings of the 53rd Annual Meeting of the Association for Computational Lin- guisties and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 774-784, Beijing, China. Association for Computational Linguistics.
Roee Aharoni and Yoav Goldberg. 2020a. Unsuper- vised domain clusters in pretrained language mod- els. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7747-7763, Online. Association for Computational Linguisties.
Sabri Eyuboglu, Maya Varma, Khaled Saab, Jean- Benoit Delbrouck, Christopher Lee-Messer, Jared Dunnmon, James Zou, and Christopher Ré. 2022. Domino: Discovering systematic errors with cross-modal embeddings. arXiv preprint arXiv:2203. 14960.
Roee Aharoni and Yoav Goldberg. 2020b. Unsuper- vised domain clusters in pretrained language models. arXiv preprint arXiv:2004.02/05.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers annotation tasks. arXiv preprint arXiv:2303. 15056.
workers for text-
Akari Asai, Sara Evensen, Behzad Golsham, Alon Halevy, Vivian Li. Andrei Lopatenko, Daniela Stepanov, Yoshihiko Suhara, Wang-Chiew Tan, and Yinzhan Xu. 2018. Happydb: A corpus 100,000 crowdsourced happy moments. arXiv preprint arXiv:1801.07766.
Ivan Habernal and Iryna Gurevych. 2016. What makes convincing argument? empirical analysis and de- tecting attributes of convincingness in web argumen- tation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Process- ing, pages 1214-1223, Austin, Texas. Association for Computational Linguistics.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
Jibo He. 2020. Big Data Set from RateMyyrroeessorcom for Professors' Teaching Evaluation.
Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference world wide web, pages 507-517.
Akshay Bhalotia. 2022.
Ye company scraper. https://githuu com/ cww/ htm/akkhhabalotiayy... company_scraper.
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Nya Sutskever, Jan Leike, Jeff Wu, and William Saunders. 2023. [link].
Yuening Hu, Jordan Alison Smith. 2014. Interactive topic modeling. Ma- chine learning, 95:423-469.
Brianna Satinoff,, and
David M Blei, Andrew Y Ng, and Michael Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan)9933100
Alex Krizhevsky, Geoffrey Hinton, al. 2009. Learn- ing multiple layers features from tiny images.
Rohit Kulkarni. 2018. A Million News Headlines.
Kaidi Cao, Maria Brbic, and Jure Leskovec. 2021. Open-world semi-supervised learning. arXiv preprint arXiv: 102.::2002.0526..
Rohit Kulkarni. 2020. The Examiner Spam Clickbait
Catalog.
Tilman Lange, Volker Roth, Mikio I. Braun, and Joachim M Buhmann. 2004. Stability babillty-based based vali- dation of clustering solutions. Neural computation. 16(6): 1299-1333.
David Carmel, Haggai Roitman, and Naama Zwerdling. 2009. Enhancing cluster labeling using wikipedia. Proceedings of the 32nd international ACM SIGIR conference on Research and development in informa- tion retrieval, pages 3961
Anh Tuan Luu, Jung-jae Kim, and See Kiong Ng. 2014. Taxonomy construction using syntactic contextual evidence. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 810-819, Doha, Qatar. Association for Computational Linguistics.
Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-Graber, and David Blei. 2009. Reading tea leaves: How humans interpret topic models. Ad- vances in neural information processing systems, 22.

---

Dheeraj Mekala and Jingbo Shang, 2020. Contextu- alized weak supervision for text classification. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 323- 333.
Chandan Singh, John X Morris, Jyoti Aneja, Alexan- der M Rush, and Jianfeng Gao. 2022. Explaining pat- terns in data with language models via interpretable autoprompting. arXiv preprint arXiv:2210. 0188.
Hongjin Su, Jungo Kasai, Yizhong Wang. Yushi Hu, Mari Ostendorf, Wen-tau Yih. Noah A Smith, Luke Zettlemoyer, Tao Yu, et al. 2022. One embedder. any task: cnstructioo-fintunnee text embeddings. arXiv preprint urXiv:22/2.09711.
Yu Meng, Jiaxin Huang, Guangyuan Wang, Zihan Wang. Chao Zhang, Yu Zhang, and Jiawei Han. 2020. Dis- criminative topic mining via category-nang guided text embedding. In Proceedings of The Web Confer- ence 2020. pages 2121-2132.
Jesse Thomason, Jivko Sinapov, Maxwell Svetlik, Pe- ter Stone, and Raymond J Mooney. 2016. Learning multi- grounded linguistic semantics by play- ing" spy". In IJCAI, pages 3477-3483.
Stuart Mitchell, Michael O' Suullvvn,, and lain Dun- ning. 2011. Pulp: : A linear programming toolkit for python.
Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Jus- tifying recommendations using distantly-labeled re- Proceedings of
Pucktada Treeratpituk and Jamie Callan. 2006. Auto- matically labeling hierarchical clusters. In Proceed- ings of the 2006 international conference on Digital government research, pages 167-176.
views and fine- greegraided aspects. the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP- IJCNLP), pages pages 188-197.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zis- serman. 2022. Generalized category discovery. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 7492- 7501.
OpenAI. 2023. Gpt-4 technical report.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel. M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and F. Duch- esnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research,
Liang Wang, Nan Yang. Xiaolong Huang, Binxing Jiao. Linjun Yang, Daxin Jiang. Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly- supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.
Tianle Wang. Zihan Wang. Weitang Liu, and Jingbo Shang. 2023. Wot-class: Weakly supervised open-world text classification. arXiv preprint arXiv:2305. 12401.
Colin Raffel, Noam Shazeer. Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Leurning Research, 21(1):5485-551).
Thomas Wolf, Lysandre Dehut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz. et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 con- ference empirical methods in natural language processing:
Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2023. Training language mod- els with language feedback at scale. arXiv preprint arXiv:2303. 16755.
system demonstrations, pages 38-45.
Jinheng Xie, Zhaochuan Luo, Yuexiang Li, Haozhe Liu, Linlin Shen, and Mike Zheng Shou. 2023. Open- world weakly-supervised object localization. arXiv preprint
Jingbo Shang, Jialu Liu, Meng Jiang, Xiang Ren, Clare R Voss, and Jiawei Han. 2018. Automated phrase mining from massive text corpora. IEFFF
Transactions on Knowledge and Data Engineering, 18250):1
Chao Zhang, Fangbo Tao, Xiusi Chen, Jiaming Shen, Meng Jiang, Brian Sadler, Michelle Vanni, and Jiawei Han. 2018 Taxogen: Unsupervised topic taxonomy construction by adaptive term embedding and clus- tering. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2701-2709.
Jingbo Shang, Xinyang Zhang, Liyuan Liu, Sha Li, and Jiawei Han. 2020. Nettaxo: Automated topic taxon- omy construction from text-rich network. In Proceed- ings of The Web Conference 2020, pages 1908-1919,
Lei Shu, Hu Xu, and Bing Liu. 2018. Unseen class dis- covery in arXiv:1801. 00060999...
classification. arXiv preprint
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances in neural information processing systems, 28.
Chandan Singh, Aliyah R Hsu, Richard Antonello, Shailee Jain, A Alexander G Huth, Bin Yu, and Jian- feng Gao. 2023. Explaining black box text modules natural language with language models. arXiy preprint 23055:2355.095.
Bingchen Zhao and Oisin Mac Aodha. 2023. Incremen- generalized category discovery. arXiv preprint 2xXi::3304.13310

---

Jiyang Zheng, Weihao Li, Jie Hong, Lars Petersson, and Nick Barnes. 2022. Towards open-set set object detec- tion and discovery. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recog- nition, pages 3961-3970.
has been announced", 3. "has a topic of a tech com- pany releases a new groundbreaking paper". and 4. "has a topic of how to improve your productivity" Writing Style 1. "has a writing style of twitter",
2. "has writing style of screen play". 3. "has a writing style of and 4) "has writing style of poem".
Ruiqi Zhong, Charlie Snell, Dan Klein, and Jacob Steinhardt. 2022. Describing differences between text distributions with natural language. Inter- national Conference Machine Learning, pages 27099-27116. PMLR.
Language 1. "has natural language of English", 2. "has a natural language of French", 3. "has natural language of Deutsch", and 4) "has a natural language of Spanish".
Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. 2023. Goal driven dis- covery of distributional differences via language de- scriptions. arXiv preprint XiXi:23302.12333.
To generate the text samples conditioned on three dimensions, we first generated "content sam- ples" based on the topic. For each topic, we first asked GPT-4 OpenAl (2023) to generate 40 En- glish news summary for each topic using diverse vocabulary as possible. Here is the prompt tem- plate we used, where we substituted (topic) with the topic we want to condition on.
Zhiying Zhu, Weixin Liang, and James Zou. 2022. Gsclip: A framework for explaining distribu- tion shifts in natural language. arXiv preprint arXiv:2206. 15007.
A Prompt Templates
Figure 5 shows the prompt we used on SYNGOALEX to propose simple explana- tions for the clusters. Figure shows the perturbed prompt for conducting prompt sensitivity analysis. Figure 9 shows the formatting instruction used OPENGOALEX propose more sophisticated explanations for the clusters.
Proposal Stage.
"Write 40 news 2-sentence paragraphs about topic (topic], using diverse vocabulary as possible. We prefer being concrete; e.g. we prefer 'James Lebron joins Lake to strengthen the team; how likely is the team going to win next time?', rather than genric statements "it's common to recruil new team members". Additionally, you cannot words that are directly used in the topic.
Assignment Stage. Figure 6 shows the prompt we used to check whether an explanation supports a sample. Figure 8 shows the perturbed prompt for conducting prompt sensitivity analysis. Prompt to Commit to Single Explanation.
Predicate 0: C1
The list continues below."
Predicate 1: 62
For each of the 64 value combinations, we first sampled 16 text samples from the content sam- ples based its topic, and then for then prompted Claude-1. 3 (Bai et al., 2022) to rewrite it with different style and language. The template we used is as follows, where we substituted forigi- nal bal_text] with the text to be rewritten, and (style) and [language] to be conditioned on:
Predicate K: cK
Text: I.
Choose the Predicate the matches the Text the most.
Selection Stage Implementation
To help the reader understand our algorithm for selecting the descriptions, we include our python implementation in Figure 10 and 11.
"(original text]
Rewrite the above paragraph in the style of [style] in (language]."
Synthesizing SYNGOALEX
We first describe the four values for each dimension and then discuss the prompts we used to generate the samples in SYNGOALEX.
D Other Techniques Used by PAS
Multiple Iterations of PAS. As mentioned Sec- tion 3.1, since the proposer's context window might not be long enough to contain the entire corpus, the
Topic: I. "ass a topic of what sports to do to im-
prove your health". "has a topic of a new anime

---

In this task will need to cone up with categories group the text together on a goal, where each category can be described with natural language predictate. For example
1, How are you doing?
2. How do I apply for visa?
Isn't
ridiculous?
4. How to get there by bus?
5. How is it going?
6. Aren't technological advancement good for the hunanity?
Here is goal: I want to cluster the questions based their functionalities, Your responses are:
"is rhetorical quesiton"
meant for greetings"
"is asking for practically useful information"
(nate that all them need to be natural language predicates that can validated on individual samples)
Here are some texts:
2, T
{samp les_ prompt
Here is goal: {goat> Ple suggest most
reun_descriptioos_peee per_ooos_p___pppttttttt escriptions, one line, starting with ".* and surrounded by quotes"". In terms of formatting, each of them needs to be
predicate about text, example:
"uses double negation"
"has conservative stance"
Do not output anything else. (Note that the examples might not be goal related, and your response should be both formatted correct above and related to the goal.)
Again, here's the goal: {goal}. Your responses are:
Figure 5: The template we used to propose candidate explanations, where we will substitute the corresponding variables to construct the prompt.
Check whether the TEXT satisfies a PROPERTY. Respond with Yes or No. When uncertain, respond with Do say anything else,
Now complete lete the following example
input: PROPERTY: {predicate} E TEXT: (text)
output:
Figure 6: The template used for the assigner which decides whether candidate explanation supports sample.

---

In the given task, you are required to devise categories classify the texts according to specific objective. Each category should expressible
language prediiatee. example:
0. How are you doing?
1. How
visa?
2. Ian't it ridiculous?
get there by
4. How going?
5. Aran't technological advancements good for humanity?
The objective here looking categorize these questions based their Your could be: "ss rhetorical question"
intended pleasantries"
"Is asking for practical Information"
(remember responses be writtan natural language predicates which verifid provided
texts]
samples:
(samples_ _ac_jjn_p___ prompt)
The objective is: (goal)
you
descriptors, beginning
formetting purposes, every responae
must be predicate about text, for instance:
"employs double negation"
"exhibits traditional standjaiin"
Do not provide any other outputs. (Note that these examples might not correspond to the actual objective, yet your response should both adhere the above mentionnd farmatting and pertinent the stated goal.) eeample__dssciptton Lanninni___perre p__proppt]
Once the objective is: [goal]], Your could be:
Figure 7: The perturbed proposal template.
Determine if the TEXT adheres to certain PROPERTY. Give Yes or No response. When in doubt, lean towards No.
Now, proceed with the following example-
input: PROPERTY: (candidate_explanatiite explanation) TEXT: (text)
output:
Figure 8: The perturbed assigner template.
Here goal: {goal} suggest dessrripiionsper_prooee descriptions, one in line, starting with and surrounded by quotes"". In terms of formatting, each of them needs predicate about a text, followed by an explanation and an example, for example: "uses double negation; specifically, there is sentence in the text that uses negation twice. For example, "the pilot did not never had "ccident " "has conservative stance; specifically, the overall text exhibits conservative political stance (e.g. pro-life, deregulation, etc). For example, 'Fetus is sentient so we should not abort.""
Figure 9: We used the same template to propose hypotheses in Section 5, except that we changed the formatting instruction to propose more sophisticated predicates. The changed part shown above key changes are underlined in red.

---

import numpy as nр
from typing import List
import pulp
def select_ descrrption_indexii
sesignmentt_mariix np.ndarray, overlap_penality: float, num sun_clusters: int List[int]:
Select the indexes of the descriptions s.t. they the entire dataset approximately once.
Parameters
assignnent_matris np.ndarray
The assignment matrix of the descriptions. The shape is datappatt,,n,s nun_descriptions). if the description supports the datapoint, 0 otherwise.
vveraappenaality float
The penalty for the overlap of the descriptions. Vlamöda paper. num clsstlsstrrss... int
The number of descriptions select. the paper.
Returns
List [intl
The indexes of the selected descriptions.
Figure 10: The function signature of the selection stage.
The function body be in Figure 10
proposer might never "see" some samples or their similar variants; consequently, some samples might not be supported by any of the proposed explana- tions. Therefore, after one iteration of PAS, we col- lect all samples not supported by any explanation and them to propose candidate explanations again, hoping that some of the new candidates will support them. To ensure broad supports over the entire corpus, we ran PAS for iterations.
our prompt template in Appendix A.
E Potential Effects of Memorization
The leakage of test data might affect both the proposer 3 5-turbo and the assigner flan-t5-x1 based on T5 (Raffel al., 2020).
For the proposer, since the prompt represents a novel usage of large language model, we do not expect the proposer prompt specifically based on (AG), (DB), and (NYT) to have appeared in the pre-training corpus. That said, it is plausible that there similar texts which produce categories based on other corpus, and they might have im- proved LM's capability to perform "in-context clus- tering": on the other hand, however, such con- sideration is irrelevant since we are not claiming that 3ppt33.5-trrde can perform "in-context clus- tering" zero-shot without any similar training data. Finally, PAS recovers two of the uncommon topic labels ("anime" and "productivvit")) for (SYN), pro- duces reasonable explanations not identical to the reference on (DB), and generates novel explana- tions for the clusters in Section 5; these empirical evidence suggests that the capability of the pro- poser largely not due to memorizing the training
Commit to Single Cluster. At the end of PAS, some samples might be supported by multiple se- bench-
lected explanations. However, the mark might require each sample to commit one single most appropriate cluster. Consider the fol- lowing sample in a news corpus: z="...afte years of disputes over security and national pride, Ger- many and the United States signed agreed to build a new American embassy on the empty lot...". is supported by both explanations G==is related real estate development" and ez="is related to international politics". While both 61 and e2 liter- ally support x, 62 is more appropriate. To commit each sample to one single cluster, we prompted an LM with all selected explanations and asked it to choose one of them as our final commitment, See

---

Create the pulp optimization problem
problem http.p/pprrommmmmeeeetceeennnnnp num , J = irsiggment_matrix.stie
pulp..Lphnnimize)
the selection vector we will optimize, the description is selected, otherwise
[ cat="Binary") for in range(J)] mn =
how often each datapoint is covered by the selected descriptions
pulp_LLvaariablee("__''' 0, None, cat="Continuous")
for x in range(num_datapoinnt))
the penalty for each datapoint, auxiliary variable
==
wwtp..paariable(faaa... (x)", 0, None, cat="Continuous")
for x in sange(nun_datappoints)
Define objective function problem += pulp. (a)
Constraint: the number of selected descriptions is nun_clusters problem += pulp. (pSum(s) = num_clusters
for in (ange(num_daaa datapoints):
# adding the constraint that n[x] the number of descriptions that
cover datapoint problem [x] pulp. LpSum( Lussuummsssinnneennaariiiii jl s(j] for j in r3))
adding the constraint that z[x] >= overlap_pemalityy (mix) 1)
problem += a [x] > overlap_pemalityy * (m[x] 1)
adding the constraint that z[x] not_cover__pealae m(x))
problem += [x] (
Solve problem holpo-p$Soiverffffattt..ss sroblem.solve()
Extract the solution based the selection vector calortad
ranna/1) auln value/e
Figure 11: The function body of the selection stage. The function signature can seen in Figure 10

---

data.
reference explanation
recall
specificity
As for the assigner, it has a similar functional- ity as classification or entailment. Since both of these tasks are already relatively straightforward for state-of-the-art language models (Gilardi et al., 2023), we do not consider the potential effect of memorization (NYT), (DB), and (AG) to play significant role in our evaluation.
Company"
44
100
Educational Institution"
63
100
"Attsst"
26
100
"Athlete"
99
100
Office Holder"
92
100
Mean Of Transportation"
64
100
Building"
35
100
Natural Place"
41
97
Village"
87
100
Animal
100
F Per-stage Evaluations
Plant"
91
100
"Album"
100
Film"
64
100
In addition to an automatic end-to-end evaluation discussed in Section 4, we also present an auto- matic per-stage evaluation to better understand the quality of each stage of the PAS algorithm.
Written Work"
28
100
Assigner Score
80
Table 6: Assign-stage evaluation on (DB)aadia of flan-t5. We abbreviate each explanation by remov- ing the prefix "has a topic of (c.g.. " "Cmmanyy" corre- sponds a full explanation has a topic of Company.") ").
Assign Stage. We first evaluate the second Assign stage, where we try to understand how well the assigner can recover the clusters when given the reference explanations of each cluster. This evalua- tion will measure the unavoidable discrepancy that might be brought by an imperfect assigner, such that later evaluations for other stages can marginal- ize out this discrepancy. We formalize the evalua- tion as following: Recall that we denote the corpus X. Let Cg be the set of texts that belong to class k and Cj the set of texts that are supported by the reference explanations of the class. We define the recall as
Dataset
Assign Score
(DB)
80
(NYT)
73
Table 7: Assign-stage evaluation on (DB)pedia and (NYT) Topics of flan-t5.
formance. Table 6 contains per-explanation ex- ample for (DB)aedia.
CC Cil [h
(8)
Propose Stage. In the Propose Stage, we would like to understand how well the proposed explana- tions capture the reference explanations. Therefore, for each reference explanation, proposed description that has highest recall to the reference explanation, as our method can operate in a top-down hierarchical manner to identify more specific clusters. We average the highest recall for each reference explanation as the score for the proposer.
and the specificity as
X C - C+| X X-C
look into the
(9)
Then, we take the average of recall and specificity as the score for this class and the average score over all classes as the score for the assigner on the dataset. The reason for analyzing recall and specificity is that recall is invariant if we found superset of a class (e.g., a parent class in a hier- archy) and specificity is invariant when we find a subset of class (e.g.. child class in a hierarchy). For the assigner, we expect the exact clusters to be found, hence we consider both recall and speci- ficity. In latter two evaluations, since our method can operate in a top-down hierarchical manner, we focus on the recall, but not the specificity.
To remedy variances and get better understand- ing of the propose stage, we ask the proposer to make a large number of proposes (we used 128 pro- poses for both datasets) and repeat the experiment three times with different random seeds; the ran- dom seeds vary the text that is given to the proposer, therefore, could yield different proposed explana- tions from the proposer. The results are presented in Table 9, We additionally show the number of proposed explanations that are actually matched (i.e., that are the highest recall for some reference explanations). Table 8 contains a per-explanation example for (DB))ediaa Overall, the proposer has
In Table 7, we illustrate the results for the as- signer (and the later evaluations) on the two harder datasets (NYT) Topics and (DB pediaa) where our method achieves a reasonable, but not perfect per-

---

reference explanation
proposed explanation
recull
reference explanation
proposed explanation
recull
Compuny"
history"
46
Company"
technology"
31
Educational Institution"
history"
Educational Institution"
language"
Artist"
entertainment"
Artisf"
music"
38
Athlete"
sports"
99
Athlete"
sports"
99
Office Holder"
politics and
87
Office Holder"
politics"
79
Meon Of Transportation"
history"
74
Mean Of Transportation"
technology"
63
" Buildngg
history"
69
Building"
architechare"
43
" Natural Place"
geagrapky"
Natural Place"
botany"
36
Village"
geography"
98
Village"
language"
67
Animat"
zoology"
80
Animal
zoology"
80
Plani"
botany"
96
Plant"
botony"
96
Album"
entertaiumend"
60
Album"
music"
" Film"
entertainment"
Film"
film"
" Written Work"
entertainment"
76
Written Work"
literature"
45
Proposer Score
Selector Score
Table 8:
Propose-stage evaluation on (DB)pedia of gpt-3.5-turbe. the used reference assigner is f1an-t5.
Table 10: Select-stage evaluation on (DB)pedia selection ILP algorithm, the used proposer is
gpt-3. 5- turbo, and the reference assigner is flan-t5.
Dataset Propose Score # matched proposes
objective
(DB)
80
(NYT)
86
9
c=a-1+ 10 (s 1),
(10)
Table Propose-stage evaluation on 14- class (DB)pedia and 9-class (NYT) Topies of 8pt-3.5-turbo. 5-turb. The score and the of matched pro- poses are averaged over three runs.
and show the results in Table 12. Notably, by not specifying the cluster number to select, our algo-
rithm is able to pick more compact set of clusters with almost similar coverage.
G Further Ablations
high coverage over the reference explanations. For (Dd)a, the number of matched proposes is one half of the true number of classes, likely due to general explanations matched (c.g., "Albam" and " Film" both are matched by" entertaiiment").
We conducted two ablations for PAS study the contribution of 1) proposing multiple iterations and 2) our selection algorithm. For 1) we compared to only running PAS for iteration, where the proposer "sces" much fewer samples; for 2) we compared to the algorithm of greedily selecting the clusters to maximize coverage and a variant of ILP that sets À = 0 (not penalizing the overlaps).
Select Stage. In the Select Stage, we are also inter- ested in how well the proposed explanations cover the desired reference explanations. We again use the average highest recall the score. In Table 11 we show the score for the select method in PAS. We note that there is a drop in coverage (i.c., a drop from propose score to select score) during the select phase, even though it is potentially pos- sible to pick exactly all the matched proposes in the proposer. This indicates potential room of improvement for the select algorithm.
We report the performance in Table 13. Overall, running PAS for five iterations improves one iteration and using an ILP algorithm with positive improves the performance.
H PAS-Generated Descriptions
We compare the PAS-generated explanations to the reference explanations; for each pair of generated and reference explanations, we also compute the Fj score between the two generated and the refer- ence cluster (100 if they are identical and 0 if they are disjoint). Generally we found that the gener- ated explanations are semantically relevant even equivalent to the references (Table 14, 15, 17, 16, 18, and 21); the only exception is when we used gpt-3. 3et-3.5-turbe 5-turbo as the proposer and Flan-T5 as
Clustering w/o Cluster Number Constraints. We would like to point out that our select algorithm does not have to enforce a number of c We could have remove the constraint of number of clusters and add an penalty proportional to the number of selected clusters in the objective. We conduct an initial experiment by changing the ILP
'The reason that we do this is for fair comparison with prior clustering methods.

---

Dataset
Propose Score
Select Score
# matched selected proposes
(DB)
80
62
10
(NYT)
86
69
7
Table II: Select-stage evaluation on the 14-class (DB)pedia and 9-class (NYT) Topics of our selection IIP algorithm. The score and the # of proposes are averaged over three runs. The number selected proposes the same as the class number, is enforced in the algorithm.
Dataset
Method
Select Score
selected proposes
# matched selected proposes
(DB)
ILP w/ # Clusters Constraint
62
14
10
(DB)
ILP w/ # Clusters Penalty
59
8
8
(NYT) ILP w/ # Clusters Constraint
69
9
7
(NYT) ILP w/ Clusters Penalty
7
Table 12: Select-stage evaluation (eB)aa and (NYT) Topics comparing the ILP method in our paper where we enforce the number of selected clusters with a constraint and a variation where we inject a small cost to favor small number of clusters. The score and the of proposes are averaged over three runs.
the assigner to cluster based on writing styles on SYNGOALEX (Table 19), but the problem allevi- ates when we use better models (gpt-4 as proposer, gpt-3.5-turbo as the assigner, Table 20).
rate-my-prof. We collect reviews of lecturers from RateMyyrofesssorccom from He (2020). We download sample of RateMyProfessor.com re- views from online repo. The goal is to cluster based on what aspects did the students comment on the teacher. This dataset can be downloaded from https://data mendeley.com datasets/fvtffjywWdd under CC BY 4.0
I OPENGOALEx Datasets
Most of our problems are adapted from the OPEND5 dataset from Zhong et al. (2023). To save budget, for each corpus we randomly sampled 400 text samples.
debate-argument arguments for position (Habernal and Gurevych, 2016), with the goal of "categorizing the types of arguments". We took the subset arguments for the position "why spank- ing is bad", "why william farquhar ought not to be honoured as the rightful founder of singapore". and ""tv is better than books".
Auman-written--feedbae human-written feed- back for model-ggnerated summaries (Scheurer al., 2023), with the goal of "categorizing model errors.
abe-headlines. We collect headlines published by ABC news, an American news company from Kulkarni (2018). ABC headlines are directly down- loaded from Harvard Dataverse. The year is ex- tracted from the publication date field. Samples are constructed from the headline text. The goal is to cluster based on the topic of the news. The data is downloadable from https: /doi. org/10. 7910/ DVN/SYBGZL with license CCO 1.0.
Blickbattheedlline We collect headlines across time from the Examiner, a clickbait news site from Kulkarni (2020). The Examiner headlines are di- rectly downloaded from Kaggle. Samples are con- structed from the headline text. The goal is to clus- ter based on their topics. The dataset can be down- loaded at https: /ttp:://wwwwwww/ //ttkk//www.kkggee kaggle. com/ dat/aattsssss with license
therohk/ teeemmiie-tteeeeeeee CCO: public domain.
amazon-reviews. We collect Amazon reviews col- lected from various product categories from Ni et al. (2019). Amazon reviews are downloaded from 2018 crawl of the website. The goal is to cluster based what aspects did the customer complained about the product. The dataset can be downloaded https: //ijj/nniiinooo/o gi thub. io/ amazon/index. html. We considered three cat- egories: Beauty product, electronics, and office products.
happy-moments. We collect self-reported happy moments and demographic characteristics from Asai et al. (2018). The HappyDB dataset is down- loaded from the official GitHub repository. De- mographic data is cleaned and merged into happy moments. Happy moment descriptions are treated as samples. The goal is to cluster based on
whom did the person spend the happy moments with. This dataset can be downloaded at https:

---

(AG)
(DB)
(NYT)
(SYN) Language
Macro (it
Topic
Topic
Topic
Location
Topic
Style
PAS
87
71
70
76
98
97
31
Only Iteration
88
60
48
77
95
97
29
Selection w/ À 0
74
71
57
72
98
99
29
Selection w/ Greedy
70
65
55
68
92
98
28
Table 13: PAS with ablations without the iterative proposing technique and with different selection algorithms described in Section 4.4. Overall, running multiple iterations and using ILP with À >> 0 are helpful.
reference explanation
output explanation
F
"has a topic of Politics"
"has a topic of politics and social issues"
"has a topic of Sports"
"has a topic of sports"
97
"has topic of Business"
"has a topic of finance"
81
"has topic of "Tchnology"
"has a topic of technology"
85
Table 14: AG's News, clustering based on Topics, proposer=gpt-3.5 t-tubo, assigner=flan-t5
//github. www./mmggaggmlllbb//aaaaaaa under un- known license.
and Instructor generated cluster, we choose 5 terms to represent and explain the cluster.
ye- startuus... We collect descriptions of companies that were part of the Combina- tor startup incubator from Bhalotia (2022). YCombinator company descriptions are down- loaded from a 2022 scrape on GitHub. Only companies with long descriptions are pre- served. The goal is to cluster based the type of startups. The dataset can be downloaded from https://www.ket kaggle. com/ /atasets/ oathhammrryy-cmmmmiiatrrcooooooe
Instructor is only able to produce clusters text based on representations. We couple it with representative term mining method that is com- monly used in text mining and taxonomy construc- tion (Zhang et al., 2018; Shang al., 2020; Mekala and Shang, 2020). The method involves first iden- tifying vocabulary that the representative terms might fall in. This step is usually done by applying AutoPhrase (Shang et al., 2018) on the entire text corpus and thresholding the unigrams and multi- grams found. Then, for each cluster, we assign each term in the vocabulary score based on statis- tical signals specific to that cluster that correspond to popularity, discriminativeness and informative- ness?. This score can also be seen as generalized version of tf-idf. Finally, for each cluster, we take the top terms as its explanation.
J Explainability Evaluation Instance
Explanation by PAS. To reduce the workload of the crowdworkers, we only showed them a con- densed summary of the explanation. For example, for the explanation:
"whether this feedback advises adding omitted details; specifically, the feedback points out that certain key details or aspects are missing from the text, which is necessary for a complete understand- ing. For example, 'The summary should include specific details about why things in their day went wrong."
LDA is able to word-based explanations for topic clusters by itself. However, we found the word clus- ters LDA generated lack in quality, despite stop word pruning and tf-idf reweighting. We there- fore first apply LDA to obtain the topic clusters, and then use the same representative term mining method above to find the top 5 terms.
we only showed the workers
Implementation Details. For Autophrase, use the official implementation at https: thub. com/ wwwsssiinnngiooo 226/AutoPhrase. and do not change the distant supervision or stop words list that was provided. We apply cutoff threshold
"whether this feedback advises adding omitted d
This reduces our average explanation length to be around 5 words.
Keyword-basee Explanations for IDA and In- structor. As PAS SAS-generated explanations have around 5 terms on average, for each of the LDA
'While this score has slightly different definitions in differ- ent papers, we follow the one from Shang et al. (2020)

---

reference explanation
output explanation
F1
"has topic of Company"
"has a topic of business"
"has a topic of Educational Institution"
"has a topic of education"
91
"has topic of Artist"
"has a topic of rivers"
0
"has topic of Athlete"
"has a topic of sports and recreation"
93
"has topic of Office Holder"
"has topic of politics and government"
90
"has topic of Mean Of Transportation""
"has topic of military equipment"
82
"has topic of Building"
"has a topic of architecture"
82
"has topic of Natural Place"
"has a topic of mountains"
56
"has topic of Village"
"has a topic of villages and towns"
99
"has topic of Animal"
"has a topic of lakes"
"has topic of Plant"
"has a topic of biology"
73
"has topic of Albumm"
"has a topic of music"
75
"has topic of Film" "has topic of Written Work"
"has topic of cinema"
89
"has topic of literature"
70
Table 15: DBpedia, clustering based on Topics, proposer=got-3.1 3et.:/gtttt...-rtttte 55tturb, assigner=flan-t5
reference explanation
output explanation
F1
"has a location of iraq"
"has a location of Iraq"
63
"has location of russia"
"has location of Russia"
77
"has location of japan'
"has location of japan"
86
"has location of canada"
haas location of canada"
81
"has location of britain"
"has location of Britain" "has location of France" "has location of germany"
87
"has location of france"
82
"has location of germany"
79
"has location of america"
"has location of the United States"
48
"has a location of china"
"has location of china"
84
haa location of italy"
"has location of italy"
94
Table 16: NYT, clustering based on Locations, proposer=gpt-3. 5-turbo, assigner=flan-t5
reference explanation
output explanation
Fi
"has lopic of health"
"has topic of healthcare"
78
"has a topic of estate"
"has topic of housing and living situations"
80
"has a topic of politics"
"has topic of war and weapons"
68
"has a topic of science"
"has a topic of climate change"
28
"has topic of sports"
"has topic of sports and competition"
97
"has topic of business"
"has topic of business and economics"
77
"has topic of arts"
"has topic of art exhibition"
63
"has topic of technology"
"has topic of technology and communication" "has topic of education"
60
"has topic of education"
82
Table 17: NYT, clustering based on Topics, proposer=gpt-3 S-turbo, assigner=flan-t5

---

reference explanation
output explanation
Fi
"has natural language of English"
"has natural language of english"
95
"has natural language of French"
"has natural language of french"
99
"has natural language of Deutsch"
"has natural language of german"
96
"has a natural language of Spanishi"
"has natural language of spanish"
100
Table 18: SYNGOALEX, clustering based Language, proposer=gpt-3. 5- 5-turbo, assigner=flan-t5
reference explanation
output explanation
F
"has writing style of twitter"
"has a writing style of health and wellness advice"
32
"has writing style of screen play"
"has a writing style of news article"
42
"has writing style of rap"
"has a writing style of instructional text"
"has a writing style of poem"
"has a writing style of artistic description"
Table 19: SYNGOALEX, clustering based Style, proposer=gpt-3. 5-turbo, assinnrr-lan-t5
reference explanation
output explanation
"has a writing style of twitter"
"has writing style of instructional or informat..."
45
"has a writing style of screen play"
"has writing style of narrative or storytelling.."
"has a writing style of rap"
"has writing style of using rhymes and rhythm"
49
"has a writing style of poem"
"has a writing style of incorporating foreign lan.."
34
Table SYNGOALEX, clustering based on Style, proposer=gpt-4, assigner=gpt-33
reference explanation
output explanation
"has a topic of what sports to do to improve your.."
"has topic of sports and physical activity"
98
"has a topic of new anime has been announced"
"has topic of anime and animation"
99
"has a topic of a tech company releases new gro..."
"has topic of advanced technology"
99
"has a topic of how to improve your productivity"
"has topic of workplace productivity"
97
Table 21: SYNGOALEX, clustering based Topics, proposer=gpt-3 5-turbo, assigner=flan-t5

---

Corpuz: human feadback madel-generaked summaries
used sklearn (Pedregosa et al., 2011) LDA and K-means on text embeddings.
Goal: categorize model errors
"corrects inaccuracles; .."
"mentoon
M Sensitivity Study
vaqueness: ..."
"pornts out information: specifcally
Prompt Sensitivity Study. We conduct study to understand how sensitive method is to our crafted prompts. We perturb our proposer and signer prompt template by paraphrasing it exten- sively; the perturbed prompts are in Appendix A. The results are shown in Table 22. The perfor- mance does not change much when the prompt changes. Most importantly the two claims, (1) our method is on par with prior clustering methods on topic clustering, and (2) our method is much better than prior methods when the goal is non-topic, still holds robustly.
the feeubock nighlights that the text fuiled
incllee certain detaiss focts. For exampie,
The surmmary does not mention that they
have been communicating for months,"
"averlocks critica
"laves out timeline Information .."
"misses essentiol question or request
background Information.....
Figure 12: By applying PAS hierarchically to cor- pus auman-written comments for menel-generated summaries (Scheurer al., 2023), cally induce taxonomies of error categories for text summarization system. The texts in quotes are copies (sometimes abbreviated) from PAS's output.
automati-
Dataset Sensitivity Study, Real world data is usu- ally not clean and perfectly balanced. To under- stand how the data balance and data noise affects our method, we conduct the following two studies on the (DB)pedia dataset. For the imbalanced sce- nario, we randomly sampled 7 classes and removed half of their data points. Then among the 7 classes, further random sampled 3 classes and again removed half of their remaining data points. Table 23, we report the delta changes of Instructor and our method PAS, and observe that the change similar when the dataset becomes imbalanced. For the noisy experiment, we randomly sampled 4 classes and removed of their data points. Then, we consider the clustering problem on the 10 re- maining classes, but with the additional data of the 4 removed classes extra noise. The evaluation is only done on the data points of the 10 remaining classes, but the model needs to robust noisy data points. From Table 23, we observe that both Instructor and our method have a similar small drop in performance.
0.5 for unigram and 0,8 for multigram which we tested to work well on heldout data. For representative term mining, remilleemenn the representativeenss score from the Shang et al. (2020) by ourself.
For LDA, we use the implementation in sklearn.
Sanity Check on Keyword Based Implemen- tation. To ensure that our implementation of keyword-based explanation is reasonable, ap- plied LDA and Instructor cluster topics the English subset of SYNGOALEX, task that we know that they can achieve better performance. As expected, Instructor achieves 80% accuracy while LDA achieves 76%, which is much higher than the performance on OPENGOALEX.
HIT Task. We paid crowdworkers $0.05 for each binary choice of explanations. The authors erage can perform HITs per minute, which trans- lates to around $1 2/hour of payment. We recruited Turkers with 98% of HIT acceptance rate in the history.
Both results indicate that our method is not es- pecially vulnerable to imbalance or noise in the dataset.
K Additional Example Taxonomy
We provide additional example taxonomy over model errors and customer reviews in Figure 12 and 13.
L Implementation Details
In terms of software libraries, we used pulp (Mitchell al., 201 implement ILP for PAS; we used transformers (Wolf et al., 2020) to run flan-t5-x1, ea-lar and instructor-xl; we

---

Corpus: customer complaints about beauty products Goal: categorize types of complaints
"addresses product quality durability issues;
"refers to misleoding or
inaccurate product representation;
"complains about product size or fit;
meentions shipping or delivery issues; specificayy the text decusses picelens rilted stipping honding, or the receppt of the itom kidde cralien"
The asted damaged
"complains about physical discomfort: specticayy by
negotve jayskui for example. The lezan caused xosh ara denousss on my sen afte spelyng."
ary
ppoviddss details on pinching or binding: specticdty the toxt remonk on the prenuct cousing prysica dicontatt by phooling or pinched
referrs to product-coused allergles: spectiraty. aesmibes eqpeeeiing ollege orsaen the lotoon nad on obonse wartion and bocke our atses"
bliding, noi my zon's hor and coused ihn paln stxing he halroue"
for
usng
excmpin,
Figure 13: Example taxonomy for complaints about beauty products from Amazon (He and McAuley, 2016).
Macro F1 (%)
(AG)
(DB)
(NYT)
(SYN)
(NYT)
(SYN)
(SYN)
Goal
Topic
Location
Language
Style
Instructor
84
77
54
25
25
PAS
70
76
31
PAS (new proposer prompt)
75
28
PAS assigner prompt)
82
29
Table 22: We paraphrase the prompt used for the proposer and assigner and assess their performance.
Delta Macro Fi (%)
sample seed seed
seed Imbalance
seed =
Instructor
-14 -8
PAS
-12
Noise
Instructor
-1
-1
-2
PAS
-2
33
-3
Table 23: We create three imbalanced and noisy versions of the DBpedia dataset where the difference at the random seed during data creation. calculate difference between the performance the clean dataset Instructor and PAS.


Reviews:
[{"review_id": "j2QNjCrhHU", "reviewer": "Reviewer_qK88", "summary": "", "questions": "see reasons to reject.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for appreciating our writing and the new task that we proposed!\n\n> Q: Do LLM-proposed explanations need to be “correct” for our method to work?\n\nEmpirically we found that LLM usually produces explanations closely related to the goal and the data. The biggest failure modes is that some of their explanations can be (a) too specific and cover too few data points, (b) too broad and cover too many data points, or (c) too similar to other explanations. **Our algorithm can address all of these failure modes**, as it attempts to select a subset of explanations to cover each data point exactly once: too specific explanations will be penalized because they reduce coverage, while too broad or similar explanations will be penalized because they induce overlaps.  For example, in the nyt-topic dataset, the proposer generated several fine-grained explanations about politics,\n- has a topic of cyberterrorism concerns in Japan\n- has a topic of international relations and diplomacy\n- has a topic of politics\n- has a topic of non-entertainment\n\nOur selection algorithm can correctly identify a single explanation “has a topic of politics” among these, because all of the rest are either too broad or too specific to make our final result cover each sample once.\n\nAdditionally, we did inspect the quality of the final selected cluster explanation. In Section 5, we measured the correctness of the final selected explanations based on human evaluation; in Appendix H table 13-20, we listed the final selected explanations for the automatic evaluations for human inspection. Both experiments reveal that the final selected explanations are of high quality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The task of obtaining personalized and goal-oriented clusters with explanations describing the content is interesting. \n- Workflow figure (Figure 2) is clear and understandable.\n- The paper is clearly written and easy to understand.\n- Interesting approach and creative applications to generating taxonomies.", "reasons_to_reject": "- to the best of my understanding, in the main paper, it is assumed that the list of explanations given by the LM is correct without any way of evaluating if this assumption holds and how the result changes based on this fact.", "questions_to_authors": "see reasons to reject.", "paper_topic_and_main_contributions": "This paper proposes an approach for personalized and goal-oriented clustering with explanations for each cluster. Given a corpus, string describing the goal of the clustering process and the number of clusters, the approach GOALEX outputs the number of clusters each accompanied by an explanation. To do this, the authors rely on the algorithm PAS which can generate goal-related explanations for\nthe clusters. The method is based on prompting a language model to obtain a list of goal-related explanations, then assigning each explanation to a cluster and finally selecting the relevant cluster by maximizing the coverage while minimizing the overlap. The authors propose two different evaluation approaches: one based on automatic evaluation by comparing their proposed method to sota methods approaches on clustering datasets, and the second method is based on manual evaluation on open-end corpora.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No missing reference.", "Typos_Grammar_Style_and_Presentation_Improvements": "Nothing was found to the best of my understanding."}, "extra_text": {}}, {"review_id": "gEve65kCus", "reviewer": "Reviewer_8dya", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for appreciating our method and task formulation. Please see our response below. \n\n> Q: Is the comparison fair if the baselines use smaller models than LLMs?\n\nWe did not try to claim that our method is superior to previous methods developed for topic clustering. This is not an apple-to-apple comparison in the first place, since our task formulation is different: prior methods could neither condition on specific goals nor provide natural language explanations. We ran our method on existing benchmarks (e.g., Table 1) only to help the readers develop a calibrated and contextualized understanding of what our system can achieve. \n\nTo best help the readers contextualize the results, we tried the best-performing embedding-based clustering systems available to us. We took Instructor (and also another method E5 in the Appendix L) as the major reference point for performance since (1) it is the state of the art clustering method and (2) it was claimed to be able to follow instructions given to the embedder, and (3) it has the same size of our validator (1.5B parameter encoder). However, even though we tried various prompting strategies and reported the best results (Section 4.4), it still fails to condition on the goal.\n\nIndeed, it might have been more informative to include results that cluster based on GPT-3.5 embeddings; however, 1) they are not accessible, and 2) task formulation-wise, pure embedding based methods cannot condition on goals specified in natural language or provide explanations for each cluster. \n\n> Q: The innovation of this paper has not been sufficiently highlighted.\n\nWe summarized our contribution (a new task formulation, a new method, and example applications) in the conclusion section, but we do agree that the contribution could have been emphasized more. In the updated version, we will also summarize the contribution at the end of the introduction.\n\n> Typos and other stylistic comments\n\nThanks for reading our paper carefully and we will fix all of them in our updated version. Regarding the underline and parentheses around method and dataset names – we added them because some other readers complained that there are too many entities in the paper, and adding these notations helped them distinguish the type of a noun (e.g., method or dataset).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. I personally think the proposed formulation has practical value, so the tasks and methods based on it are application-oriented.\n2. The proposed method (PAS) is intuitive, effectively solves GOALEX based on the experimental results.\n3. The paper presents new formulation, dataset, and detailed methodology using LLMs. This provides value and insights to the research community.", "reasons_to_reject": "1. Potential unfair comparisons. The authors use LDA and Instructor as baselines, however, LDA is a classic work from 20 years ago, and Intructor-XL only has 1.5B parameters, which are far from the proposed PAS using gpt-3.5-turbo. \n2. The writing clearly describes the approach but does not sufficiently highlight the contributions. More emphasis on the key innovations would improve readability.\n3. There are some typos, formatting errors, and incomplete paragraphs needing further proofreading (Please refer to typo field).", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduced a novel text clustering formulation, GOALEX, which considers users' goals and output natural language explanations for clusters. The author proposed a multi-stage algorithm, PAS, to address this new problem. The automatic/human-involved evaluation results show that the proposed method produces more accurate and goal-oriented clusters compared to existing methods.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Table 4: Missing top line.\n- Line 648 Incorrect hyphen.\n- Line 647 Incomplete paragraph.\n- Line 1114 Redundant \"=\" symbol.\n\nAdditionally, reducing the underlining in Sections 4 and 5 may improve the readability, as they do not indicate core ideas. Same as the bracketed dataset abbreviations in Section 4."}}, {"review_id": "fNi3jTFGBX", "reviewer": "Reviewer_v3tK", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for appreciating our task, method and writing! Please see our response below.\n\n> Q: Is our method sensitive to different prompt formats?\n\nFollowing your recommendation, we **include a new experiment** (details below) to study the sensitivity of our results to different proposer and assigner prompts, and we find that 1) the change in performance is minor in most datasets, and 2) the conclusions in our paper still robustly hold. Thanks for your insightful comments which make our paper stronger!\n\nDetails for the experiments:  We analyze the sensitivity of the proposer and the assigner to the prompt format; we paraphrase the prompt extensively while not changing its meaning (prompt attached at the end). The table below shows the results\n\n|                           | AGNews | DBpedia | NYT Topic | SYN Topic | NYT Location | SYN Language | SYN Style |\n|---------------------------|--------|---------|------------|-----------|--------------|--------------|-----------|\n| Instructor                | 84     | 82      | 69         | 77        | 54           | 25           | 25        |\n| PAS (default)             | 87     | 71      | 70         | 98        | 76           | 97           | 31        |\n| PAS (new proposer prompt) | 86     | 72      | 68         | 98        | 75           | 98           | 28        |\n| PAS (new assigner prompt) | 87     | 63      | 67         | 98        | 82           | 98           | 29        |\n\nThe performance does not change much when the prompt changes. Most importantly, the following major claims in our paper still hold robustly:\n- Our method is on par with prior clustering methods on topic clustering. Our method is indeed lacking on DBpedia, due to a cluster merging effect we illustrated in the paper (paragraph L356-368)\n- Our method is much better than prior methods when the goal is non-topic. \n\n> Q: Additional experiment of clusters of different sizes, noise in the data, and high-dimensional data.\n\nWe conducted the following two preliminary studies regarding data size and noise on the DBpedia dataset (14 classes). We found that **our method is as insensitive as our baseline method, Instructor, to imbalanced or noisy data.** \n \nFor clustering on imbalanced data, we randomly sampled 7 classes and removed half of their data points. Then among the 7 classes, we further random sampled 3 classes and again removed half of their remaining data points. We repeat this procedure 3 times and create 3 imbalanced datasets by varying the sampled classes. In the table below, we report the delta changes of Instructor and our method PAS, and observe that the change is similar when the dataset becomes imbalanced. This shows that our method is not more vulnerable to imbalanced data.\n\n| DBpedia Imbalance (delta) | sample seed = 0 | sample seed = 1 | sample seed = 2 |\n|---------------------------|-----------------|-----------------|-----------------|\n| Instructor                | -11             | -14             | -10             |\n| PAS                       | -10             | -8              | -12             |\n\nFor the noisy experiment, we randomly sampled 4 classes and removed 7 / 8 of their data points. Then, we consider the clustering problem on the 10 remaining classes, but with the additional data of the 4 removed classes as noise. The evaluation is only done on the data points of the 10 remaining classes, but the model needs to be robust to the data points not belonging to any of them. We observe that both Instructor and our method have a similar small drop in performance, showing that our method is also not especially sensitive to noise in the dataset.\n\n| DBpedia Noise (delta) | sample seed = 0 | sample seed = 1 | sample seed = 2 |\n|-----------------------|-----------------|-----------------|-----------------|\n| Instructor            | -1              | -1              | -2              |\n| PAS                   | -2              | -3              | -3              |\n\nCan you clarify what “high-dimensional data” refers to?\n\n> Here are the original and changed proposer and assigner prompts.\n\nOriginal Proposer Prompt:\n```\nIn this task you will need to come up with categories to group the text together based on a goal, where each category can be described with a natural language predicate. For example\n\n0. How are you doing?\n1. How do I apply for visa?\n2. Isn't it ridiculous?\n3. How to get there by bus?\n4. How is it going?\n5. Aren't technological advancement good for the humanity?\n\nHere is our goal: I want to cluster the questions based on their functionalities. Your responses are:\n- \"is a rhetorical quesiton\"\n- \"is meant for greetings\"\n- \"is asking for practically useful information\"\n\n(note that all of them need to be natural language predicates that can be validated on the given samples)\n\nHere are some texts:\n\n{samples_in_prompt}\n\nHere is our goal: {goal} \nPlease suggest me a few descriptions, one in a line, starting with \"-\" and surrounded by quotes\"\". In terms of formatting, each of them needs to be a predicate about a text, for example:\n- \"uses double negation\"\n- \"has a conservative stance\"\n\nDo not output anything else. (Note that the examples might not be goal related, and your response should be both formatted correct as above and related to the goal.)\n{example_description_in_prompt}\n\nAgain, here's the goal: {goal}. Your responses are:\n- \"\n```\n\nNew Proposer prompt:\n```\nIn the given task, you are required to devise categories to classify the texts according to a specific objective. Each category should be expressible via a natural language predicate. Here's an example:\n\n0. How are you doing?\n1. How do I apply for a visa?\n2. Isn't it ridiculous?\n3. How to get there by bus?\n4. How is it going?\n5. Aren't technological advancements good for humanity?\n\nThe objective here is: I am looking to categorize these questions based on their purpose. Your response could be:\n- \"is a rhetorical question\"\n- \"is intended for pleasantries\"\n- \"is asking for practical information\"\n\n(remember these responses must be written as natural language predicates which can be verified against the provided sample texts)\n\nHere are some text samples:\n\n{samples_in_prompt}\n\nThe objective is: {goal} \nPlease can you provide a number of descriptors, beginning with \"-\" and are within \"\". For formatting purposes, every response must be a predicate about a text, for instance:\n- \"employs double negation\"\n- \"exhibits a traditional standpoint\"\n\nDo not provide any other outputs. (Note that these examples might not correspond to the actual objective, yet your response should both adhere to the above-mentioned formatting and be pertinent to the stated goal.)\n{example_description_in_prompt}\n\nOnce more, the objective is: {goal}. Your responses could be:\n- \"\n```\n\nOriginal Assigner Prompt:\n```\nCheck whether the TEXT satisfies a PROPERTY. Respond with Yes or No. When uncertain, output No. \n\nNow complete the following example -\ninput: PROPERTY: {candidate_explanation}\nTEXT: {text}\noutput:\n```\n\nNew Assigner Prompt:\n\n```\nDetermine if the TEXT adheres to a certain PROPERTY. Give a Yes or No response. When in doubt, lean towards No. \n\nNow, proceed with the following example -\ninput: PROPERTY: {candidate_explanation}\nTEXT: {text}\noutput:\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper introduces a practical method for making text clustering understandable, useful across various real-world applications.\n- Implementation details are provided for replicating the results.\n- The paper is easy to read with intuitive figures.", "reasons_to_reject": "- How much does the output vary with tiny variations of prompts/instructions to the models? Any sort of sensitivity analysis may help measure the robustness. \n- It would also make the experiment more solid if the author could explain how the proposed method handles clusters of different sizes, noise in the data, and high-dimensional data.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper aims to make unsupervised text clustering more user-centric and explainable. It proposes a text clustering task that considers the users' goals and explains the meanings of detected clusters as free-form language descriptions. A three-stage algorithm, Propose-Assign-Select (PAS), is proposed to tackle the task. The algorithm prompts a language model to generate goal-related explanations for candidate clusters, assigns text samples to the explanations that support them, and then uses integer linear programming to select a subset of candidate clusters that cover most samples while minimizing overlaps. The authors also evaluate the PAS algorithm under both automatic and human evaluation on corpora with or without labels", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "v2DxA4SR24", "reviewer": "Reviewer_wCff", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q: Reproducibility of the results.\n\nWhile the exact same performance reported in our paper might not be deterministically replicable due to API changes, the main claims of our paper are robustly reproducible. To illustrate this, we conducted an experiment with gpt-3.5-turbo-0301 and gpt-3.5-turbo-0613, two different gpt 3.5 models and show their results in the table below. \n\n|                          | AGNews | DBpedia | NYT Topic | SYN Topic | NYT Location | SYN Language | SYN Style |\n|--------------------------|--------|---------|-----------|-----------|--------------|--------------|-----------|\n| Instructor               | 84     | 82      | 69        | 77        | 54           | 25           | 25        |\n| PAS (paper)              | 87     | 71      | 70        | 98        | 76           | 97           | 31        |\n| PAS (gpt-3.5-turbo-0301) | 87     | 69      | 67        | 98        | 81           | 99           | 29        |\n| PAS (gpt-3.5-turbo-0613) | 88     | 68      | 64        | 98        | 75           | 98           | 28        |\n\nThe main claims from our paper still robustly hold even under different API endpoints. \n- Our method is on par with prior clustering methods on topic clustering. By looking at the columns with AGNews, DBpedia, NYT Topic and SYN Topic this claim is still supported regardless of the differences in the LLM. As already acknowledged in our submission, our method has lower performance on DBpedia due to the cluster merging effect illustrated in our paper (paragraph L356-368). \n- Our method is much better than prior methods when the goal is non-topic. This is also supported with the numbers in columns NYT Location, SYN Language and SYN Style.\n\n> Q: Method requires high-end LLMs which may be of high costs\n\nWe acknowledged this in our limitations section, and we are working on a follow-up project of this paper to reduce this cost.\n\nWe’d also like to point out that when we set our proposer as gpt3.5, and validator as t5-xl (the majority of our experiments in section 4), the cost is affordable. One would need about **0.1$** in total for querying the openly accessible api for gpt 3.5, and **a single 24G gpu** to run inference with t5-xl.\n\nWe agree that the current best performance relies on the most powerful large models but we hope that the cost would decrease soon due to algorithmic and hardware improvements.\n\n> Q: Referring to discussions/tables in the appendix.\n\nThanks for your recommendation. We believe that the content in the main paper is self-contained and does not rely on the appendix. However, we agree that it would be better to move some discussion to the main paper when an additional page is available. Concretely, we will move L277 (validity of using LLMs), L384 (syn style with gpt 4 as proposer and gpt 3.5 as assigner) to the main text as suggested. Further, we will move Appendix L (experiments on an additional clustering method E5) to the main text as well in the updated version. Please feel free to point out more places where the presentation could be improved.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- proposing a task that has practical application for data analysis and solves the hard problem of determining meaning fully labels for clusters\n- Well written paper\n- thorough analysis and experiments\n", "reasons_to_reject": "- the solution with the best performance depends on closed LLMs (GPT-4/claude-v1.3)\n  - high cost \n  - because training data for these LLMs hasn't been released, there could be test data leakage \n[the authors address the potential for leakage in appendix E and I mostly agree with their reasoning]\n- the authors refer to discussions/tables in the appendix in the main text (e.g., 277, 384). \n- difficult to reproduce because the work relies on closed LLMs that (a) are non-deterministic in their output and (b) can change at any time", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes **a new task** called \"Goal-Driven Clustering with Explanations\" (GOALEX) considering the users' goals and produces an explainations of a  clusters' meanings. The input to GOALEX is a corpus of texts with multiple attributes and a goal description in natural language (e.g., \"I want to cluster based on sentiment\"), and the output is a set of clusters, each with a natural language explanation of which text samples should or should not belong to the cluster. The paper also presents a **three-stage algorithm** called Propose-Assign-Select (PAS) to tackle GOALEX: (a) prompting a language model to generate a list of goal-related explanations for candidate clusters, (b) assign text samples only to the explanations that support them, and (c) uses integer linear programming to search for a subset of candidate explanations so that each sample is roughly supported once. PAS was evaluated on known and open-ended corpora and was found to produce more accurate and goal-related explanations than prior methods (e.g., LDA).", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "648: re- quires -> requires"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new framework for unsupervised clustering of documents, which tries to satisfy the desiderata of goal driven clustering with free-form textual explanations of clusters. Their approach involves prompting a language model to generate candidate explanations, assigning documents to those explanations, and aligning possible matches using an ILP. Using both automatic and human evaluations, the authors find they are able to generate clusters and explanations that conform to the user-driven goals.

Reviewers were intrigued by this task, and found the paper to be clearly written and easy to follow. They also thought this was creative work with practical value in applications.

The reasons to reject given by the reviewers were relatively minor, including pointing out the high computational cost of this method, suggesting additional baselines, and requesting sensitivity analyses. The most serious concern seems to be that the authors to some extent assume that the generated explanations are correct, without adequately verifying this, but the reviewer who pointed this out was still quite favorable about the paper.

Although the reviewers were evenly split on soundness (between Good and Strong), 3 of the 4 reviewers rated their excitement as Strong, suggesting that this would be a reasonable choice for a main conference paper.
Decision: Accept-Main