ZeroDL: Zero-shot Distribution Learning for Text Clustering via Large Language Models
Authors: Hwiyeol Jo, Hyunwoo Lee, Taiwoo Park
Accepted at: acl25

ZeroDL: Zero-shot Distribution Learning for Text Clustering via Large Language Models

Hwiyeol Jo1, Hyunwoo Lee2, Kang Min Yoo1, Taiwoo Park3*
1NAVER Cloud, 2NAVER, 3NAVER Search US
{hwiyeol.jo,hanu.lee,kangmin.yoo,taiwoo.park}@navercorp.com

Abstract
The advancements in large language models (LLMs) have brought significant progress in NLP tasks. However, if a task cannot be fully described in prompts, the models could fail to carry out the task. In this paper, we propose a simple yet effective method to contextualize a task toward a LLM. The method utilizes (1) open-ended zero-shot inference from the entire dataset, (2) aggregate the inference results, and (3) finally incorporate the aggregated meta-information for the actual task. We show the effectiveness in text clustering tasks, empowering LLMs to perform text-to-text-based clustering and leading to improvements on several datasets. Furthermore, we explore the generated class labels for clustering, showing how the LLM understands the task through data.

1 Introduction
Large language models (LLMs) have demonstrated impressive performances on various downstream tasks (Devlin et al., 2019; Radford et al., 2019). These also exhibit the ability to understand the context of input text, known as in-context learning (ICL) (Brown et al., 2020; OpenAI, 2023). ICL allows leveraging LLMs for specific tasks without further extensive training. However, effective use of ICL hinges on well-designed prompts.

While prompts with few-shot examples demonstrably improve performance, they can easily overfit a model to the examples (Perez et al. (2021); Mizrahi et al. (2023); inter alia). This led to a growing interest in zero-shot learning, which reduces the need for intricate few-shot selection. Recent advancements in zero-shot learning involve incorporating more sophisticated use of prompt structures, such as Chain-of-Thought (Wei et al., 2022), zero-shot reasoning (Kojima et al., 2022), and models trained to follow instructions (Ouyang et al., 2022; Chung et al., 2024). However, how to design prompts for target tasks remains challenging.

Figure 1: Illustration of proposed method ZeroDL. While in-context learning (ICL) relies on examples (D) tailored to specific tasks, ZeroDL aggregates all the outputs from these zero-shot inferences (\( \hat{m} \)), resulting in meta-level information (\( \hat{M} \)). This information is then used by the LLM to generate its final predictions.

Motivated by the core principle of ICL—providing task and data contexts within prompts—we propose an approach to construct more effective zero-shot prompts by understanding how LLMs describe datasets across prompting outputs.

As illustrated in Figure 1, Zero-shot Distribution Learning (ZeroDL) aims to learn data distributions through zero-shot inferences. The method comprises two key components: open-ended zero-shot inference and output aggregation. Zero-shot prompts are then constructed with the generated meta information, and used for actual task. This method takes advantage of the self-generated frame of LLMs to successfully carry out a given task.

We exemplify the effectiveness of ZeroDL on text clustering tasks where complete task descriptions cannot be provided due to an absence of ground-truth class labels. In addition, our method works in a text-to-text format, allowing clustering with specific context. For instance, "I love this movie" and "I hate this movie" express opposite sentiment but belong to the same cluster of movie reviews (see Table 9 in the Appendix for potential
risk of absence of the perspectives).

Our contributions in this paper are as follows:
• We propose a novel approach called Zero-shot Distribution Learning that leverages zero-shot inferences to generate meta-level information about the data distribution by aggregating open-ended inference outputs from datasets.
• ZeroDL allows models to perform text-based clustering, empowering them to handle data with specific context, which offers advantages over embedding-based clustering methods.
• ZeroDL is competitive against embedding-based clustering methods on several datasets. Notably, ZeroDL even achieves better performances than models with ground-truth class labels in some cases.

2 Related Works

The majority of existing LLM-based clustering approaches rely on traditional methods like K-Means, employing LLM-generated embeddings as input (Petukhova et al., 2024; BehnamGhader et al., 2024). ClusterLLM (Zhang et al., 2023) introduced LLM-guided refinement of embedding-based clustering models. In contrast, our approach ZeroDL, directly leverages text-level prompting to inject specific viewpoints into LLMs, enabling more targeted and contextualized clustering.

Other works have explored using LLMs as clustering models (Wang et al., 2023; Viswanathan et al., 2023; Pham et al., 2023; Huang and He, 2024), but these approaches typically relied on few-shot settings with access to gold labels. IDAS (De Raedt et al., 2023) addressed this limitation by selecting representative data using an auxiliary embedding model to generate pseudo-labels without demonstrations. These labels are subsequently refined through in-context learning.

ZeroDL distinguishes itself by operating in a fully zero-shot setting, eliminating the need for additional embedding models. Moreover, while IDAS’s class labels are primarily influenced by a limited set of data points and generate the labels independently, ZeroDL considers the entire data distribution through an aggregation step. This enables our method to generate class labels in an auto-regressive manner, taking into account the relationships between different labels.

The importance of appropriate ground-truth labels extends beyond clustering and permeates ICL. While Min et al. (2022) observed cases where the input-label correspondence does not play significant roles, Yoo et al. (2022) argued that the impact heavily depends on target tasks and experiment settings. We believe that this work would serve as a reference that appropriate class labels can be successfully generated by LLMs themselves.

3 Proposed Method: ZeroDL

Stage 1: Open-Ended Inference  We begin by designing a prompt for zero-shot classification. This prompt intentionally avoids any detailed information about the task, minimizing the risk of overfitting. Based on the idea, we opt for the simplest prompt format:

<table>
  <tr>
    <th>Open-Ended Inference Prompt Template</th>
  </tr>
  <tr>
    <td>Text: [text]</td>
  </tr>
  <tr>
    <td>Classify the text to the best [type_of_task] class.</td>
  </tr>
</table>

where [text] is the input data and [type_of_task] provides view of the task. In the experiment, it can be either sentiment or topic. Leveraging this prompt, we perform model inferences on all the input data (\( D \)). This process generates open-ended class predictions, which will be denoted as \( \hat{m} \).

Stage 2: Aggregation  The open-ended predictions lack constraints, leading to potentially inconsistent output formats. For instance, the model might predict "positive" and "non-positive" classes, while the ground-truth is "positive" and "negative". The predictions could even be entire sentences. To address the inconsistencies in the open-ended predictions, we employ aggregation strategy.

Before the aggregation, we count the frequency of each predictions and sort it, denoted as \( \hat{m}_1, \hat{m}_2, \cdots, \hat{m}_U \) where \( U \) denotes the unique number of predictions and the frequency of \( \hat{m}_n \) is equal or larger than \( \hat{m}_{n+1} \). After that, the predictions which frequency is only 1 are dropped in order to remove extraordinary predictions and save computation. Next, we iteratively construct subsets of the predictions by removing the least frequent predictions one by one. This process results in a list of subsets, denoted as \( S \); \( S_U = \{ \hat{m}_1, \hat{m}_2, \cdots, \hat{m}_U \} \), \( S_{U-1} = \{ \hat{m}_1, \hat{m}_2, \cdots, \hat{m}_{U-1} \} \), \( \cdots \), \( S_1 = \{ \hat{m}_1 \} \). We input each subset to the LLM, providing more weights to the frequently occurred predictions (e.g., \( \hat{m}_1 \) is repeatedly included in the subsets while \( \hat{m}_U \) occurs only once). The prompt template for aggregation is as follows:
Aggregation Prompt Template

[type_of_task] List:

<table>
  <tr>
    <th>S_U</th>
    <th>S_{U-1}</th>
    <th>S_{-}</th>
    <th>S_1</th>
  </tr>
  <tr>
    <td>\hat{m}_1</td>
    <td>\hat{m}_1</td>
    <td>\hat{m}_1</td>
    <td>\hat{m}_1</td>
  </tr>
  <tr>
    <td>\hat{m}_2</td>
    <td>\hat{m}_2</td>
    <td>\hat{m}_2</td>
    <td></td>
  </tr>
  <tr>
    <td>...</td>
    <td>...</td>
    <td>...</td>
    <td></td>
  </tr>
  <tr>
    <td>\hat{m}_{U-1}</td>
    <td>\hat{m}_{U-1}</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>\hat{m}_U</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Aggregate the [type_of_task] List into [NUM_CLUSTER_CLASS] classes.

where [NUM_CLUSTER_CLASS] is pre-defined number of classes to cluster. However, the model outputs often still lack coherence, especially in generating exact number of classes. To address this, we select the generation results only when the number of aggregated classes matches the pre-defined number of cluster classes. We then use the most frequent aggregated classes as meta-information \( \hat{M} \) (class labels in the experiments). This information represents the model’s understanding of potential views over the entire data.

Stage 3: Leveraging Meta-Information We incorporate the aggregated meta-information (\( \hat{M} \)) into the original prompt\footnote{The order of Text: and Class description: can be reversed.}:

Final Prediction Prompt Template

Text: [text]

Class description:
- Class 0: \( M_0 \)
- Class ... : \( M_{...} \)
- Class C: \( M_C \)

Based on the class description, classify the text to the best [type_of_task] class.

By incorporating the meta-information (\( \hat{M} \)) into the prompt, we enable the LLM to perform conditioned classification within the clustering context.

4 Experiments

Models Our primary experiments utilize mistral-7b-instruct-v0.2 (Jiang et al., 2023). We further evaluate our approach with several other models, including Qwen-2.5 (Team, 2024), Gemma-2 (Team et al., 2024), Llama-3.1 (Meta, 2024). Lastly, HyperCLOVA X SEED (Yoo et al., 2024) demonstrates the result of a relatively small instruct-following language model. All the results are averaged over 5 runs.

Baselines IDAS (De Raedt et al., 2023) represents a zero-shot clustering approach. We employ 8 demonstrations as described in the paper, using SBERT (Opitz and Frank, 2022) as an additional embedder. Final prediction is performed using the same prompt template as our Stage 3. We also include IIm2vec (BehnamGhader et al., 2024) with K-Means where applicable. This method represents a baseline using (not exactly but) the same backbone computed by embedding-level\footnote{Note that IDAS requires an additional embedding model and IIm2vec involves additional training steps.}. We exclude other few-shot settings from our comparisons as they are not considered fair due to the inherent advantages of leveraging additional information from demonstrations and the subjectivity involved in the few-shot selection.

Setting Our method aims to effectively cluster data distributions, converging them into a suitable number of classes. We recognize that clustering datasets often exhibit a high number of classes relative to the available data points. To address this, we utilize text classification datasets, which typically offer a larger number of data instances per class than clustering datasets. Moreover, the availability of ground-truth labels facilitates direct comparison between generated and actual labels, enabling qualitative analysis and interpretability.

Datasets We thus use 6 text classification datasets for clustering: IMDB (Maas et al., 2011), SST-2, SST-5 (Socher et al., 2013), YelpReviews (Zhang et al., 2015) for sentiment classification, and AGNews, DBpedia (Lehmann et al., 2015), YahooAnswers (Chang et al., 2008) for topic classification. Data information is presented in Table 4 of the Appendix.

Evaluation Model performance is evaluated based on the highest accuracy achieved across all possible mapping combinations. To determine the predicted class, we leverage the Class n anchor tokens within the LLM outputs. However, LLMs might not directly predict the same classes as the ground-truth labels\footnote{For example, Class 0 means positive in the prediction while Negative is labeled as Class 0 in the ground-truth.}, so we test all possible mapping combinations and report the best performed one. We split datasets with more than 7 classes (i.e., DBpedia and YahooAnswers) into 2 subsets Front (F) and Back (B) to avoid out-of-memory\footnote{The computation is a factorial of the number of classes.} after
<table>
  <tr>
    <th>Model</th>
    <th>Method</th>
    <th colspan="2">IMDB</th>
    <th colspan="2">SST-2</th>
    <th colspan="2">SST-5</th>
    <th colspan="2">YRev</th>
    <th colspan="2">AGNew</th>
    <th colspan="2">DBp(F)</th>
    <th colspan="2">DBp(B)</th>
    <th colspan="2">Yah(F)</th>
    <th colspan="2">Yah(B)</th>
    <th>Macro</th>
    <th>Micro</th>
  </tr>
  <tr>
    <td rowspan="4">hcx-seed</td>
    <td>ZeroDL(C-T)</td>
    <td>56.2</td>
    <td>62.5</td>
    <td>33.6</td>
    <td><b>43.6</b></td>
    <td>33.1</td>
    <td>24.1</td>
    <td>16.9</td>
    <td>21.3</td>
    <td>18.3</td>
    <td>34.4</td>
    <td>33.1</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td><b>-0.5b-it</b></td>
    <td>IDAS(C-T)</td>
    <td>88.3</td>
    <td>74.9</td>
    <td>27.7</td>
    <td>35.0</td>
    <td>29.4</td>
    <td>17.0</td>
    <td>26.8</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Gold(C-T)</td>
    <td><b>72.1</b></td>
    <td><b>75.2</b></td>
    <td><b>35.2</b></td>
    <td>27.6</td>
    <td>31.4</td>
    <td>31.2</td>
    <td>24.1</td>
    <td>18.3</td>
    <td><b>18.3</b></td>
    <td><b>37.0</b></td>
    <td><b>33.6</b></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>ZeroDL(T-C)</td>
    <td>86.7</td>
    <td>72.2</td>
    <td><b>36.0</b></td>
    <td><b>43.9</b></td>
    <td><b>59.5</b></td>
    <td>32.3</td>
    <td>44.5</td>
    <td>26.7</td>
    <td>22.8</td>
    <td>47.2</td>
    <td>46.4</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>IDAS(T-C)</td>
    <td><b>89.4</b></td>
    <td>62.9</td>
    <td>28.9</td>
    <td>28.7</td>
    <td>27.3</td>
    <td>23.7</td>
    <td>27.2</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Gold(T-C)</td>
    <td>89.1</td>
    <td><b>73.7</b></td>
    <td>34.4</td>
    <td>39.1</td>
    <td>59.4</td>
    <td><b>61.1</b></td>
    <td><b>57.2</b></td>
    <td><b>40.3</b></td>
    <td><b>56.2</b></td>
    <td><b>56.7</b></td>
    <td><b>56.3</b></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="4">mistral</td>
    <td>1lm2vec+KMeans</td>
    <td>62.1</td>
    <td>55.2</td>
    <td>30.3</td>
    <td><b>56.0</b></td>
    <td><b>84.4</b></td>
    <td><b>96.1</b></td>
    <td>70.8</td>
    <td><b>48.0</b></td>
    <td>46.2</td>
    <td>61.0</td>
    <td>67.5</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td><b>-7b-it</b></td>
    <td>ZeroDL(C-T)</td>
    <td><b>88.8</b></td>
    <td><b>85.7</b></td>
    <td>40.5</td>
    <td>48.7</td>
    <td>75.2</td>
    <td>58.9</td>
    <td>64.1</td>
    <td>47.2</td>
    <td>61.4</td>
    <td>63.4</td>
    <td>61.5</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>IDAS(C-T)</td>
    <td>51.1</td>
    <td>63.1</td>
    <td><b>41.4</b></td>
    <td>37.3</td>
    <td>38.1</td>
    <td>32.1</td>
    <td>21.8</td>
    <td>30.2</td>
    <td>41.8</td>
    <td>39.7</td>
    <td>35.4</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Gold(C-T)</td>
    <td>87.5</td>
    <td>77.0</td>
    <td><b>41.8</b></td>
    <td>50.8</td>
    <td>60.7</td>
    <td>74.2</td>
    <td><b>85.2</b></td>
    <td>40.8</td>
    <td><b>62.5</b></td>
    <td><b>64.5</b></td>
    <td><b>68.1</b></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>ZeroDL(T-C)</td>
    <td>90.2</td>
    <td>84.2</td>
    <td>36.0</td>
    <td>46.8</td>
    <td>79.5</td>
    <td>56.7</td>
    <td>72.2</td>
    <td><b>51.0</b></td>
    <td>66.3</td>
    <td>64.8</td>
    <td>63.0</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>IDAS(T-C)</td>
    <td>50.1</td>
    <td>68.1</td>
    <td><b>39.6</b></td>
    <td>41.0</td>
    <td>44.6</td>
    <td>35.8</td>
    <td>27.6</td>
    <td>33.3</td>
    <td>46.1</td>
    <td>42.9</td>
    <td>38.9</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Gold(T-C)</td>
    <td><b>91.7</b></td>
    <td><b>82.5</b></td>
    <td><b>43.3</b></td>
    <td>51.8</td>
    <td>82.7</td>
    <td>84.1</td>
    <td><b>82.7</b></td>
    <td>50.9</td>
    <td><b>73.8</b></td>
    <td><b>71.5</b></td>
    <td><b>72.7</b></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="4">Qwen-2.5</td>
    <td>ZeroDL(C-T)</td>
    <td>87.6</td>
    <td>74.8</td>
    <td><b>45.5</b></td>
    <td><b>53.8</b></td>
    <td><b>70.9</b></td>
    <td>62.3</td>
    <td>75.9</td>
    <td>45.3</td>
    <td><b>68.8</b></td>
    <td>65.0</td>
    <td>65.8</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td><b>-7b-it</b></td>
    <td>IDAS(C-T)</td>
    <td><b>93.3</b></td>
    <td><b>87.3</b></td>
    <td>42.7</td>
    <td>51.7</td>
    <td>68.8</td>
    <td>47.4</td>
    <td>68.4</td>
    <td>30.3</td>
    <td>41.8</td>
    <td>58.9</td>
    <td>59.0</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Gold(C-T)</td>
    <td>84.3</td>
    <td>87.4</td>
    <td>45.0</td>
    <td>49.9</td>
    <td>62.2</td>
    <td><b>84.5</b></td>
    <td><b>89.0</b></td>
    <td><b>51.9</b></td>
    <td>68.2</td>
    <td><b>69.2</b></td>
    <td><b>71.3</b></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>ZeroDL(T-C)</td>
    <td>93.5</td>
    <td>85.2</td>
    <td><b>50.5</b></td>
    <td>49.3</td>
    <td><b>84.3</b></td>
    <td>66.7</td>
    <td>78.3</td>
    <td>50.8</td>
    <td>74.4</td>
    <td>70.3</td>
    <td>68.2</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>IDAS(T-C)</td>
    <td><b>95.3</b></td>
    <td><b>88.9</b></td>
    <td>45.1</td>
    <td><b>51.6</b></td>
    <td>83.1</td>
    <td>56.0</td>
    <td>72.2</td>
    <td>31.6</td>
    <td>43.3</td>
    <td>63.0</td>
    <td>62.7</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Gold(T-C)</td>
    <td>94.7</td>
    <td>88.5</td>
    <td>46.3</td>
    <td>49.0</td>
    <td>82.9</td>
    <td><b>94.2</b></td>
    <td><b>98.0</b></td>
    <td><b>64.5</b></td>
    <td><b>79.7</b></td>
    <td><b>77.5</b></td>
    <td><b>78.6</b></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="4">gemma-2</td>
    <td>ZeroDL(C-T)</td>
    <td>94.3</td>
    <td>88.0</td>
    <td><b>52.6</b></td>
    <td>51.6</td>
    <td><b>74.5</b></td>
    <td>52.8</td>
    <td>53.5</td>
    <td>50.3</td>
    <td>60.0</td>
    <td>64.2</td>
    <td>60.1</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td><b>-27b-it</b></td>
    <td>IDAS(C-T)</td>
    <td><b>95.3</b></td>
    <td><b>91.1</b></td>
    <td>42.7</td>
    <td><b>58.3</b></td>
    <td>31.7</td>
    <td>56.9</td>
    <td>44.0</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Gold(C-T)</td>
    <td>79.8</td>
    <td>86.3</td>
    <td>48.7</td>
    <td>57.3</td>
    <td>70.5</td>
    <td><b>89.8</b></td>
    <td><b>97.1</b></td>
    <td><b>56.1</b></td>
    <td><b>65.9</b></td>
    <td><b>72.4</b></td>
    <td><b>75.9</b></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>ZeroDL(T-C)</td>
    <td>94.0</td>
    <td>88.3</td>
    <td>50.8</td>
    <td>50.8</td>
    <td>81.8</td>
    <td>57.3</td>
    <td>57.6</td>
    <td>52.7</td>
    <td>73.0</td>
    <td>67.4</td>
    <td>62.7</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>IDAS(T-C)</td>
    <td><b>95.2</b></td>
    <td>89.2</td>
    <td>42.4</td>
    <td><b>55.6</b></td>
    <td>35.0</td>
    <td>60.5</td>
    <td>49.8</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>Gold(T-C)</td>
    <td>95.3</td>
    <td><b>89.6</b></td>
    <td><b>51.8</b></td>
    <td><b>55.7</b></td>
    <td><b>85.4</b></td>
    <td><b>92.6</b></td>
    <td><b>98.7</b></td>
    <td><b>66.9</b></td>
    <td><b>84.4</b></td>
    <td><b>80.0</b></td>
    <td><b>81.0</b></td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Table 1: The performance of ZeroDL for text clustering. C-T denotes the prompt order with class information then input text. T-C is the reversed. Bold means the best accuracy in the same prompt order and underline denotes the outperforming cases than baselines (except for Gold label). IDAS with HyperCLOVA X SEED and Gemma-2 is constrained by the maximum 8,192 token length. More baselines are presented in Table 5 of the Appendix.

removing the 3 smallest size of classes\footnote{CarsAndTransportation, SocialScience, and Sports classes in YahooAnswers dataset}.

\section*{5 Results}

Table 1 presents the performance of our ZeroDL method. Notably, ZeroDL surpasses models provided with ground-truth (Gold) class labels on several datasets. This suggests that ZeroDL may uncover richer or more nuanced class labels within the data compared with pre-defined labels. Our method demonstrates performance comparable to K-Means clustering using LLM embeddings, particularly excelling on datasets with relatively smaller sizes. ZeroDL outperforms IDAS, especially on datasets with more than 2 classes. IDAS often struggles to generate appropriate class labels, particularly when relying on a limited coverage of representative data instances as demonstrations. Furthermore, IDAS is susceptible to the input length limitations of LLMs when the demonstrations are long, as evident in the Gemma results. In contrast, ZeroDL achieves its results through flexible zero-shot prompting without requiring any modifications.

Table 2 shows examples of class labels generated from ZeroDL. These labels often provide richer and more informative explanations\footnote{Note that the generation of class description varies depending on datasets and LLMs.} compared with the original ground-truth labels. Moreover, ZeroDL can potentially uncover novel classes based on the data, such as Ambiguous Sentiment and Mixed Sentiment\footnote{Examples are provided in Table 6 of the Appendix.}. This highlights the limitations of pre-defined labels in capturing the full spectrum of sentiment complexity.

Figure 2 illustrates the confusion matrix of the results. The relatively low performance of SST-5, even with gold labels, underscores the inherent difficulty of sentiment classification. In contrast, AGNews demonstrates successful matching with our method. This observation emphasizes the potential value of ZeroDL in scenarios where ground-truth labels are unavailable.

Table 3 investigates the significance of class la-
<table>
  <tr>
    <th>Data</th>
    <th>Method</th>
    <th>ClassLabels</th>
  </tr>
  <tr>
    <td rowspan="6">SST-5</td>
    <td rowspan="6">ZeroDL</td>
    <td><b>Neutral Sentiment</b>: This class includes all the sentiment labels that express a neutral sentiment towards the movie or documentary. Examples include (…)</td>
  </tr>
  <tr>
    <td><b>Negative Sentiment</b>: This class includes all the sentiment labels that express a negative or sad emotion towards the movie or documentary. Examples include (…)</td>
  </tr>
  <tr>
    <td><b>Ambiguous Sentiment</b>: This class includes all the sentiment labels that do not clearly express a positive or negative emotion towards the movie or documentary. Examples include (…)</td>
  </tr>
  <tr>
    <td><b>Mixed Sentiment</b>: This class includes all the sentiment labels that express a mixed sentiment towards the movie or documentary. Examples include (…)</td>
  </tr>
  <tr>
    <td><b>Positive Sentiment</b>: This class includes all the sentiment labels that express a positive emotion towards the movie or documentary. Examples include (…)</td>
  </tr>
  <tr>
    <td><b>Disappointing, Critical, Negative, Positive, Disappointing or Unsatisfying</b></td>
  </tr>
  <tr>
    <td rowspan="3">IDAS</td>
    <td>Very Negative, Negative, Neutral, Positive, Very Positive</td>
  </tr>
  <tr>
    <td>Gold</td>
    <td>International Relations and Politics: This class includes topics related to international relations, diplomacy, Middle East politics, terrorism, nuclear politics, and elections.</td>
  </tr>
  <tr>
    <td>ZeroDL</td>
    <td>Sports and Entertainment: This class includes topics related to sports, tennis, golf, basketball, baseball, cricket, and entertainment.</td>
  </tr>
  <tr>
    <td rowspan="4">AGNews</td>
    <td rowspan="4">ZeroDL</td>
    <td>Business and Economy: This class includes topics related to the economy, finance, stocks, mergers and acquisitions, retail, real estate, and labor markets.</td>
  </tr>
  <tr>
    <td>Technology and Science: This class includes topics related to technology, computing, internet, cybersecurity, space exploration, and science.</td>
  </tr>
  <tr>
    <td>Iran’s Nuclear Program, Oil Prices, Businesses prioritize hardware upgrades in economic recovery,</td>
  </tr>
  <tr>
    <td>Volcanic activity at Mount St. Helens</td>
  </tr>
  <tr>
    <td>Gold</td>
    <td>World, Sports, Business, Sci/Tech</td>
  </tr>
</table>

Table 2: The example of generated class labels in 5-class sentiment classification (SST–5), and topic classification (AGNews). ZeroDL can generates alternative class labels and its description. Compared with IDAS that has overlap in the generated labels, ZeroDL considers other labels in generation. Furthermore, IDAS’s representative data points fail to generate general topic class labels. Additional examples are in Table 8.

![Confusion matrix of proposed method ZeroDL in SST-5 (left) and AGNews (right). x-axis and y-axis are generated labels and gold labels, respectively.](page_172_1012_563_246.png)

Figure 2: Confusion matrix of proposed method ZeroDL in SST-5 (left) and AGNews (right). x-axis and y-axis are generated labels and gold labels, respectively.

<table>
  <tr>
    <th></th>
    <th>S2(C-T)</th>
    <th>S2(T-C)</th>
    <th>S5(C-T)</th>
    <th>S5(T-C)</th>
  </tr>
  <tr>
    <td>RandToken</td>
    <td>51.6</td>
    <td>59.5</td>
    <td>28.6</td>
    <td>28.6</td>
  </tr>
  <tr>
    <td>AutoL(Best)</td>
    <td><b>86.8</b></td>
    <td><b>84.9</b></td>
    <td><b>43.9</b></td>
    <td><b>40.5</b></td>
  </tr>
  <tr>
    <td>AutoL(Worst)</td>
    <td>82.5</td>
    <td>80.7</td>
    <td>41.1</td>
    <td>39.7</td>
  </tr>
  <tr>
    <td>Gold</td>
    <td>77.0</td>
    <td>82.5</td>
    <td>41.8</td>
    <td><b>43.3</b></td>
  </tr>
  <tr>
    <td>ZeroDL(Mistral)</td>
    <td>80.6</td>
    <td>83.7</td>
    <td>40.3</td>
    <td>40.5</td>
  </tr>
  <tr>
    <td>ZeroDL(gpt-4.1)</td>
    <td>77.0</td>
    <td>82.5</td>
    <td>41.0</td>
    <td>40.3</td>
  </tr>
</table>

Table 3: The mistral-7b performance with various class labels. ZeroDL(Mistral) uses only the generated class titles for fair comparison. The best labels from AutoL in SST-2 are [Wonderful, Bad] and the worst are [Irresistible, Pathetic]. In SST-5, [Terrible, Better, Good, Extraordinary, Unforgettable] are the best while [Awful, Better, Hilarious, Perfect, Wonderful] are the worst. gpt-4.1-mini generates [Positive, Negative] (the same as Gold) and [Positive, Negative, Neutral, Mixed, Other / Unclear], respectively.

bels in text clustering tasks. We explore the performance of mistral-7b-instruct-v0.2 using class labels suggested by AutoL (Gao et al., 2021) designed for prompt-based model fine-tuning. These labels represent a curated selection. We also investigate the class labels generated by gpt-4.1-mini. The results demonstrate that clustering performance is generally higher when using manually selected labels than using the original dataset labels. This suggests that carefully chosen labels can significantly improve clustering outcomes. In the context, ZeroDL performs particularly well on SST-5, implying that the potential of ZeroDL to capitalize on informative class labels automatically.

ZeroDL involves a trade-off in computational cost. We evaluate the impact of varying input data amounts on performance (see Table 7 in the Appendix). While using only 10% of the data yields reasonable performance, it can lead to inconsistencies in model performance as evidenced by increased standard deviation.

6 Conclusion

We introduce ZeroDL, a novel approach to contextualize tasks for a given LLM. ZeroDL employs open-ended zero-shot inference and output aggregation to learn data distributions. We demonstrate its effectiveness, showing competitive performances against embedding-based clustering methods and superior performance than ground-truth labels in some cases. Beyond its clustering capabilities, ZeroDL offers the generation of informative class labels that provide deeper insights into LLMs.
7 Limitations

Prompt Dependency and Heuristics ZeroDL relies on carefully designed prompts to guide LLMs towards effective clustering. While our focus was on using simple and intuitive prompts, prompt selection can potentially influence the model’s behavior and introduce biases. Future work could explore more sophisticated prompt engineering techniques to further enhance ZeroDL’s performance.

Experiments with Diverse LLMs and Prompts
While we acknowledge the computational limitations (and price) of ZeroDL, investigating its behavior with a wider range of LLMs (including commercial models like GPT, Claude, and Gemini) and prompt templates could provide valuable insights into the generalizability and robustness of the approach.

Lower than State-of-the-Art Performance
Achieving state-of-the-art performance is not the sole focus of ZeroDL but it offers a valuable framework in understanding data distributions with zero-shot inference via LLMs. By addressing the limitations mentioned above, ZeroDL has the potential to become a powerful and versatile tool not only for text clustering but data exploration.

Expensive Computational Cost in Inferences.
Although ZeroDL have an alternative approach to reduce computational burden through data sampling, effectively sampling data to generate appropriate class labels remains a challenge. Although we report the trade-off in computational cost and performance in Table 7 of the Appendix, the technique within the framework presents a valuable future direction.

Acknowledgement
The authors would like to thank reviewers in several rounds of discussion. Thanks to Alice Lee for her help with English writing and Minji Hong for her support in writing this work. Lastly, we extend our gratitude to Hyperscale AI Team at NAVER Cloud, for unwavering support in connecting us with key resources throughout this research.

References
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05961.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Ming-Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic representation: Dataless classification. In AAAI, volume 2, pages 830–835.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53.

Maarten De Raedt, Frédéric Godin, Thomas Demeester, and Chris Develder. 2023. Idas: Intent discovery with abstractive summarization. arXiv preprint arXiv:2305.19783.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Yanjun Gao, Ting-Hao Huang, and Rebecca J. Passonneau. 2021. Learning clause representation from dependency-anchor graph for connective prediction. In Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15), pages 54–66, Mexico City, Mexico. Association for Computational Linguistics.

Chen Huang and Guoxiu He. 2024. Text clustering as classification with llms. arXiv preprint arXiv:2410.00927.

AQ Jiang, A Sablayrolles, A Mensch, C Bamford, DS Chaplot, D de las Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. 2023. Mistral 7b (2023). arXiv preprint arXiv:2310.06825.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.

Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. 2015. Dbpedia—a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6(2):167–195.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.

Meta. 2024. Introducing meta llama 3: The most capable openly available llm to date. https://ai.meta.com/blog/meta-llama-3/.

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2023. State of what art? a call for multi-prompt llm evaluation. arXiv preprint arXiv:2401.00595.

OpenAI. 2023. Gpt-4 technical report.

Juri Opitz and Anette Frank. 2022. SBERT studies meaning representations: Decomposing sentence embeddings into explainable semantic features. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 625–638, Online only. Association for Computational Linguistics.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. Advances in neural information processing systems, 34:11054–11070.

Alina Petukhova, Joao P Matos-Carvalho, and Nuno Fachada. 2024. Text clustering with llm embeddings. arXiv preprint arXiv:2403.15112.

Chau Minh Pham, Alexander Hoyle, Simeng Sun, Philip Resnik, and Mohit Iyyer. 2023. Topicpt: A prompt-based topic modeling framework. arXiv preprint arXiv:2311.01449.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussein, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118.

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Vijay Viswanathan, Kiril Gashteovski, Carolin Lawrence, Tongshuang Wu, and Graham Neubig. 2023. Large language models enable few-shot clustering. arXiv preprint arXiv:2307.00524.

Zihan Wang, Jingbo Shang, and Ruiqi Zhong. 2023. Goal-driven explainable clustering via language descriptions. arXiv preprint arXiv:2305.13749.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Kang Min Yoo, Jaegeun Han, Sookyo In, Heewon Jeon, Jisu Jeong, Jaewook Kang, Hyunwook Kim, Kyung-Min Kim, Munhyong Kim, Sungju Kim, et al. 2024. Hyperclova x technical report. arXiv preprint arXiv:2404.01954.

Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, and Taeuk Kim. 2022. Ground-truth labels matter: A deeper look into input-label demonstrations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2422–2437, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649–657.

Yuwei Zhang, Zihan Wang, and Jingbo Shang. 2023. Clusterllm: Large language models as a guide for text clustering. arXiv preprint arXiv:2305.14871.
A Appendix

A.1 Data Information

<table>
  <tr>
    <th>#Train</th>
    <th>#Valid</th>
    <th>#Test</th>
    <th>#Class</th>
    <th>Pre-defined ClassTitle</th>
  </tr>
  <tr>
    <td>IMDB</td>
    <td>25,000</td>
    <td>3,750</td>
    <td>25,000</td>
    <td>2</td>
    <td>Negative, Positive</td>
  </tr>
  <tr>
    <td>SST-2</td>
    <td>9,645</td>
    <td>1,101</td>
    <td>2,210</td>
    <td>2</td>
    <td>Negative, Positive</td>
  </tr>
  <tr>
    <td>SST-5</td>
    <td>9,645</td>
    <td>1,101</td>
    <td>2,210</td>
    <td>5</td>
    <td>Very Negative, Negative, Neutral, Positive, Very Positive</td>
  </tr>
  <tr>
    <td>YelpReviews</td>
    <td>650,000</td>
    <td>97,500</td>
    <td>49,999</td>
    <td>5</td>
    <td>Very Negative, Negative, Neutral, Positive, Very Positive</td>
  </tr>
  <tr>
    <td>AGNews</td>
    <td>120,000</td>
    <td>18,000</td>
    <td>7,600</td>
    <td>4</td>
    <td>World, Sports, Business, Sci/Tech</td>
  </tr>
  <tr>
    <td>DBpedia(F)</td>
    <td>280,000</td>
    <td>41,939</td>
    <td>35,000</td>
    <td>7</td>
    <td>Company, EducationalInstitution, Artist, Athlete, OfficeHolder, MeanOfTransportation, Building</td>
  </tr>
  <tr>
    <td>DBpedia(B)</td>
    <td>280,000</td>
    <td>42,213</td>
    <td>35,000</td>
    <td>7</td>
    <td>NaturalPlace, Village, Animal, Plant, Album, Film, WrittenWork</td>
  </tr>
  <tr>
    <td>Yahoo(F)</td>
    <td>59,518</td>
    <td>8,879</td>
    <td>10,489</td>
    <td>7</td>
    <td>ArtsAndHumanities, BeautyAndStyle, BusinessAndFinance, ComputersAndInternet, ConsumerElectronics, EducationAndReference, EntertainmentAndMusic</td>
  </tr>
  <tr>
    <td>Yahoo(B)</td>
    <td>59,493</td>
    <td>8,973</td>
    <td>10,514</td>
    <td>7</td>
    <td>FoodAndDrink, GamesAndRecreation, Health, HomeAndGarden, Pets, PregnancyAndParenting, SocietyAndCulture</td>
  </tr>
</table>

Table 4: Data information used in the experiments.

A.2 Additional Model Performances

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="2">IMDB</th>
    <th colspan="2">SST-2</th>
    <th colspan="2">SST-5</th>
    <th colspan="2">YRev</th>
    <th colspan="2">AGNew</th>
    <th colspan="2">DBp(F)</th>
    <th colspan="2">DBp(B)</th>
    <th colspan="2">Yah(F)</th>
    <th colspan="2">Yah(B)</th>
    <th>Macro</th>
    <th>Micro</th>
  </tr>
  <tr>
    <th>IMDB</th>
    <th>SST-2</th>
    <th>SST-5</th>
    <th>YRev</th>
    <th>AGNew</th>
    <th>DBp(F)</th>
    <th>DBp(B)</th>
    <th>Yah(F)</th>
    <th>Yah(B)</th>
  </tr>
  <tr>
    <td>TF–IDF</td>
    <td>KMeans</td>
    <td>52.1</td>
    <td>53.0</td>
    <td>26.2</td>
    <td>35.2</td>
    <td>43.8</td>
    <td>55.9</td>
    <td>62.8</td>
    <td>44.7</td>
    <td>46.5</td>
    <td>46.7</td>
    <td>48.8</td>
  </tr>
  <tr>
    <td>SBERT</td>
    <td>KMeans</td>
    <td>65.0</td>
    <td>54.0</td>
    <td>33.9</td>
    <td>30.3</td>
    <td>82.7</td>
    <td>95.4</td>
    <td>93.9</td>
    <td>67.2</td>
    <td>68.3</td>
    <td>65.6</td>
    <td>67.5</td>
  </tr>
  <tr>
    <td rowspan="2">llama-3.1–8b-it</td>
    <td>ZeroDL (C–T)</td>
    <td>76.0</td>
    <td><b>73.8</b></td>
    <td>35.0</td>
    <td><b>51.1</b></td>
    <td>54.5</td>
    <td>37.7</td>
    <td>58.5</td>
    <td>46.4</td>
    <td>47.9</td>
    <td>53.4</td>
    <td>53.2</td>
  </tr>
  <tr>
    <td>IDAS (C–T)</td>
    <td><b>92.2</b></td>
    <td><b>73.8</b></td>
    <td>42.4</td>
    <td>45.4</td>
    <td>55.4</td>
    <td>40.8</td>
    <td>64.1</td>
    <td>29.2</td>
    <td>35.3</td>
    <td>53.2</td>
    <td>53.9</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>Gold (C–T)</td>
    <td>80.4</td>
    <td>73.5</td>
    <td><b>44.6</b></td>
    <td>48.7</td>
    <td><b>64.1</b></td>
    <td><b>74.4</b></td>
    <td><b>80.8</b></td>
    <td><b>49.6</b></td>
    <td><b>69.5</b></td>
    <td><b>65.1</b></td>
    <td><b>66.7</b></td>
  </tr>
  <tr>
    <td>ZeroDL (T–C)</td>
    <td>94.4</td>
    <td>79.4</td>
    <td>39.1</td>
    <td>49.4</td>
    <td>77.6</td>
    <td>46.1</td>
    <td>68.1</td>
    <td>53.7</td>
    <td>60.0</td>
    <td>63.1</td>
    <td>61.1</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>IDAS (T–C)</td>
    <td><b>94.6</b></td>
    <td><b>84.5</b></td>
    <td>38.4</td>
    <td><b>50.1</b></td>
    <td>67.7</td>
    <td>37.6</td>
    <td>66.4</td>
    <td>32.0</td>
    <td>36.2</td>
    <td>56.4</td>
    <td>56.2</td>
  </tr>
  <tr>
    <td>Gold (T–C)</td>
    <td>94.4</td>
    <td>81.5</td>
    <td><b>43.4</b></td>
    <td><b>50.1</b></td>
    <td><b>78.9</b></td>
    <td><b>80.8</b></td>
    <td><b>75.1</b></td>
    <td><b>62.8</b></td>
    <td><b>78.3</b></td>
    <td><b>71.7</b></td>
    <td><b>71.2</b></td>
  </tr>
  <tr>
    <td rowspan="2">Qwen-2.5–72b-it</td>
    <td>ZeroDL (C–T)</td>
    <td>94.5</td>
    <td><b>90.2</b></td>
    <td>44.5</td>
    <td><b>62.6</b></td>
    <td>71.3</td>
    <td>60.8</td>
    <td>64.4</td>
    <td>53.4</td>
    <td>71.3</td>
    <td>68.1</td>
    <td>67.5</td>
  </tr>
  <tr>
    <td>IDAS (C–T)</td>
    <td><b>95.3</b></td>
    <td>88.8</td>
    <td>47.1</td>
    <td>50.5</td>
    <td>68.1</td>
    <td>62.3</td>
    <td>57.2</td>
    <td>35.7</td>
    <td>55.8</td>
    <td>62.3</td>
    <td>61.1</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>Gold (C–T)</td>
    <td>95.1</td>
    <td>89.4</td>
    <td><b>51.0</b></td>
    <td>57.7</td>
    <td><b>81.3</b></td>
    <td><b>94.2</b></td>
    <td><b>97.9</b></td>
    <td><b>69.0</b></td>
    <td><b>84.8</b></td>
    <td><b>80.0</b></td>
    <td><b>81.6</b></td>
  </tr>
  <tr>
    <td>ZeroDL (T–C)</td>
    <td>95.3</td>
    <td><b>90.1</b></td>
    <td>45.4</td>
    <td><b>63.3</b></td>
    <td>76.6</td>
    <td>65.2</td>
    <td>65.4</td>
    <td>55.1</td>
    <td>73.6</td>
    <td>70.0</td>
    <td>69.4</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>IDAS (T–C)</td>
    <td><b>95.9</b></td>
    <td>89.9</td>
    <td>46.3</td>
    <td>51.2</td>
    <td>76.4</td>
    <td>63.2</td>
    <td>58.9</td>
    <td>34.5</td>
    <td>47.7</td>
    <td>62.7</td>
    <td>61.7</td>
  </tr>
  <tr>
    <td>Gold (T–C)</td>
    <td><b>95.8</b></td>
    <td><b>90.0</b></td>
    <td><b>52.7</b></td>
    <td>59.9</td>
    <td><b>83.9</b></td>
    <td><b>96.8</b></td>
    <td><b>99.7</b></td>
    <td><b>73.7</b></td>
    <td><b>87.4</b></td>
    <td><b>82.2</b></td>
    <td><b>83.8</b></td>
  </tr>
</table>

Table 5: The performance of ZeroDL for text clustering compared with more methods. C–T denotes the prompt order with class information then input text. T–C is the reversed. Bold means the best accuracy in the same prompt order and underline the outperforming cases than baseline except for ground-truth class labels.
A.3 Qualitative Examples

<table>
  <tr>
    <th>Predicted</th>
    <th>Gold</th>
    <th>Example</th>
  </tr>
  <tr>
    <td rowspan="4">Mixed Sentiment</td>
    <td>VeryNeg</td>
    <td>It’s hard not to feel you’ve just watched a feature-length video game with some really heavy back story .</td>
  </tr>
  <tr>
    <td>Neg</td>
    <td>But it pays a price for its intricate intellectual gamesmanship .</td>
  </tr>
  <tr>
    <td>Neutral</td>
    <td>The appearance of Treebeard and Gollum’s expanded role will either have you loving what you’re seeing, or rolling your eyes .</td>
  </tr>
  <tr>
    <td>Pos</td>
    <td>An utterly compelling ‘who wrote it’ in which the reputation of the most famous author who ever lived comes into question .</td>
  </tr>
  <tr>
    <td rowspan="5">Ambiguous Sentiment</td>
    <td>VeryPos</td>
    <td>…a roller-coaster ride of a movie</td>
  </tr>
  <tr>
    <td>VeryNeg</td>
    <td>It’s difficult to say whether The Tuxedo is more boring or embarrassing – I’m prepared to call it a draw .</td>
  </tr>
  <tr>
    <td>Neg</td>
    <td>Like most Bond outings in recent years , some of the stunts are so outlandish that they border on being cartoonlike .</td>
  </tr>
  <tr>
    <td>Neutral</td>
    <td>Effective but too-tepid biopic</td>
  </tr>
  <tr>
    <td>Pos</td>
    <td>But he somehow pulls it off .</td>
  </tr>
  <tr>
    <td></td>
    <td>VeryPos</td>
    <td>Emerges as something rare , an issue movie that’s so honest and keenly observed that it does n’t feel like one .</td>
  </tr>
</table>

Table 6: Example of data predicted to newly generated classes in SST-5. Randomly selected.

A.4 Data Sampling Ablation

<table>
  <tr>
    <th>[C-T]</th>
    <th>IMDB</th>
    <th>SST-2</th>
    <th>SST-5</th>
    <th>YRev</th>
    <th>AGNew</th>
    <th>DBp(F)</th>
    <th>DBp(L)</th>
    <th>Yah(F)</th>
    <th>Yah(L)</th>
  </tr>
  <tr>
    <td>All</td>
    <td>88.8</td>
    <td>85.7</td>
    <td>40.5</td>
    <td>48.7</td>
    <td>75.2</td>
    <td>58.9</td>
    <td>64.1</td>
    <td>47.2</td>
    <td>61.4</td>
  </tr>
  <tr>
    <td>(std)</td>
    <td>3.20</td>
    <td>0.18</td>
    <td>1.22</td>
    <td>0.21</td>
    <td>1.22</td>
    <td>7.85</td>
    <td>6.61</td>
    <td>1.02</td>
    <td>1.92</td>
  </tr>
  <tr>
    <td>1%</td>
    <td>81.3</td>
    <td>79.3</td>
    <td>46.9</td>
    <td>51.5</td>
    <td>66.0</td>
    <td>59.4</td>
    <td>58.4</td>
    <td>36.1</td>
    <td>43.7</td>
  </tr>
  <tr>
    <td>(std)</td>
    <td>2.97</td>
    <td>1.5</td>
    <td>0.88</td>
    <td>0.12</td>
    <td>5.71</td>
    <td>1.93</td>
    <td>5.11</td>
    <td>0.72</td>
    <td>3.35</td>
  </tr>
  <tr>
    <td>5%</td>
    <td>74.6</td>
    <td>83.6</td>
    <td>45.8</td>
    <td>49.4</td>
    <td>74.2</td>
    <td>58.8</td>
    <td>66.4</td>
    <td>39.3</td>
    <td>58.5</td>
  </tr>
  <tr>
    <td>(std)</td>
    <td>18.49</td>
    <td>1.37</td>
    <td>2.78</td>
    <td>0.47</td>
    <td>1.94</td>
    <td>5.38</td>
    <td>9.13</td>
    <td>4.82</td>
    <td>2.73</td>
  </tr>
  <tr>
    <td>10%</td>
    <td>87.0</td>
    <td>84.5</td>
    <td>44.8</td>
    <td>49.2</td>
    <td>74.5</td>
    <td>67.2</td>
    <td>63.9</td>
    <td>40.4</td>
    <td>60.4</td>
  </tr>
  <tr>
    <td>(std)</td>
    <td>5.90</td>
    <td>1.50</td>
    <td>5.02</td>
    <td>0.29</td>
    <td>1.44</td>
    <td>6.35</td>
    <td>8.95</td>
    <td>5.11</td>
    <td>2.96</td>
  </tr>
  <tr>
    <th>[T-C]</th>
    <th>IMDB</th>
    <th>SST-2</th>
    <th>SST-5</th>
    <th>YRev</th>
    <th>AGNew</th>
    <th>DBp(F)</th>
    <th>DBp(L)</th>
    <th>Yah(F)</th>
    <th>Yah(L)</th>
  </tr>
  <tr>
    <td>All</td>
    <td>90.2</td>
    <td>84.2</td>
    <td>36.0</td>
    <td>46.8</td>
    <td>79.5</td>
    <td>56.7</td>
    <td>72.2</td>
    <td>51.0</td>
    <td>66.3</td>
  </tr>
  <tr>
    <td>(std)</td>
    <td>3.52</td>
    <td>0.24</td>
    <td>3.84</td>
    <td>0.47</td>
    <td>1.73</td>
    <td>9.73</td>
    <td>6.86</td>
    <td>0.75</td>
    <td>2.80</td>
  </tr>
  <tr>
    <td>1%</td>
    <td>88.4</td>
    <td>58.6</td>
    <td>42.0</td>
    <td>49.8</td>
    <td>74.6</td>
    <td>57.7</td>
    <td>67.5</td>
    <td>37.7</td>
    <td>49.4</td>
  </tr>
  <tr>
    <td>(std)</td>
    <td>0.51</td>
    <td>11.93</td>
    <td>3.99</td>
    <td>2.05</td>
    <td>4.34</td>
    <td>1.31</td>
    <td>8.37</td>
    <td>4.1</td>
    <td>3.15</td>
  </tr>
  <tr>
    <td>5%</td>
    <td>85.5</td>
    <td>80.2</td>
    <td>41.7</td>
    <td>46.2</td>
    <td>80.7</td>
    <td>57.1</td>
    <td>74.1</td>
    <td>37.2</td>
    <td>68.1</td>
  </tr>
  <tr>
    <td>(std)</td>
    <td>7.36</td>
    <td>1.92</td>
    <td>4.09</td>
    <td>1.42</td>
    <td>2.83</td>
    <td>10.58</td>
    <td>9.36</td>
    <td>6.87</td>
    <td>4.58</td>
  </tr>
  <tr>
    <td>10%</td>
    <td>91.2</td>
    <td>82.0</td>
    <td>39.1</td>
    <td>46.6</td>
    <td>79.9</td>
    <td>63.2</td>
    <td>75.3</td>
    <td>45.2</td>
    <td>68.5</td>
  </tr>
  <tr>
    <td>(std)</td>
    <td>2.16</td>
    <td>1.95</td>
    <td>2.26</td>
    <td>0.33</td>
    <td>2.74</td>
    <td>6.53</td>
    <td>9.91</td>
    <td>3.43</td>
    <td>3.42</td>
  </tr>
</table>

Table 7: ZeroDL performances according to the number of input data.

A.5 Additional Examples of Generated Classes
<table>
  <tr>
    <th>Data</th>
    <th>Method</th>
    <th>ClassLabels</th>
  </tr>
  <tr>
    <td rowspan="3">IMDB</td>
    <td>ZeroDL</td>
    <td><b>Negative Sentiment:</b> The list also includes various expressions of negative sentiment towards movies, films, shows, and documentaries. Some examples include (...)
      <br><b>Positive Sentiment:</b> The list includes various expressions of positive sentiment towards movies, films, shows, and documentaries. Some examples include (...)
    </td>
  </tr>
  <tr>
    <td>Gold</td>
    <td><b>Negative, Positive</b></td>
  </tr>
  <tr>
    <td>SST-2</td>
    <td>ZeroDL</td>
    <td><b>Positive Sentiment:</b> All the sentiment labels that express a positive sentiment towards the movie, film, documentary, or subject. For example, (...)
      <br><b>Negative or Neutral Sentiment:</b> All the sentiment labels that do not express a positive sentiment towards the movie, film, documentary, or subject. For example, (...)
    </td>
  </tr>
  <tr>
    <td rowspan="6">Yelp Rev</td>
    <td>ZeroDL</td>
    <td><b>Negative Sentiment:</b> This class includes sentences expressing negative sentiments towards a place, food, or experience. Examples include (...)
      <br><b>Very Positive Sentiment:</b> This class includes sentences expressing highly positive sentiments towards a place, food, or experience. Examples include (...)
      <br><b>Mixed Sentiment:</b> This class includes sentences expressing mixed sentiments towards a place, food, or experience. Examples include (...)
      <br><b>Neutral Sentiment:</b> This class includes sentences expressing neutral sentiments towards a place, food, or experience. Examples include (...)
      <br><b>Positive Sentiment:</b> This class includes sentences expressing positive sentiments towards a place, food, or experience. Examples include (...)
    </td>
  </tr>
  <tr>
    <td>Gold</td>
    <td><b>Very Negative, Negative, Neutral, Positive, Very Positive</b></td>
  </tr>
  <tr>
    <td rowspan="7">DBp (F)</td>
    <td>ZeroDL</td>
    <td><b>Aviation and Transportation:</b> This class includes topics related to aviation, aerospace technology, military history, maritime history, and transportation.
      <br><b>Business and Economy:</b> This class includes topics related to business, finance, industries, companies, and economics.
      <br><b>Sports and Biographies:</b> This class includes topics related to sports, athletes, and their biographies.
      <br><b>Politics and Government:</b> This class includes topics related to politics, government, elections, and specific political parties.
      <br><b>Education:</b> This class includes topics related to education, universities, schools, and specific educational institutions.
      <br><b>History and Architecture:</b> This class includes topics related to history, architecture, historic sites, castles, and landmarks.
      <br><b>Art and Entertainment:</b> This class includes topics related to art, music, entertainment, and specific artists or celebrities.
    </td>
  </tr>
  <tr>
    <td>Gold</td>
    <td><b>Company, EducationalInstitution, Artist, Athlete, OfficeHolder, MeanOfTransportation, Building</b></td>
  </tr>
  <tr>
    <td rowspan="7">DBp (B)</td>
    <td>ZeroDL</td>
    <td><b>Science and Technology:</b> This class includes topics related to paleontology, geology, volcanology, space exploration, and academic journals.
      <br><b>Music and Entertainment:</b> This class includes topics related to music, album releases, jazz music, heavy metal music, hip hop music, and entertainment.
      <br><b>Geography and Hydrology:</b> This class includes topics related to geography, hydrology, rivers, water bodies, and water resources.
      <br><b>Botany and Plant Sciences:</b> This class includes topics related to botany, horticulture, plant taxonomy, plant conservation, and endangered species.
      <br><b>Literature and Books:</b> This class includes topics related to literature, novels, fiction, mystery and crime fiction, and academic publications.
      <br><b>Zoology and Entomology:</b> This class includes topics related to zoology, entomology, moths, butterflies, fish species, and arachnids.
      <br><b>Film and Television:</b> This class includes topics related to film, cinema, movies, movie reviews, Bollywood, and television.
    </td>
  </tr>
  <tr>
    <td>Gold</td>
    <td><b>NaturalPlace, Village, Animal, Plant, Album, Film, WrittenWork</b></td>
  </tr>
  <tr>
    <td rowspan="7">Yah (F)</td>
    <td>ZeroDL</td>
    <td><b>Personal Finance and Economics:</b> Topics related to personal finance, credit scores, debt management, taxes, and economics.
      <br><b>Health and Wellness:</b> Topics related to health, medicine, fitness, nutrition, and wellness.
      <br><b>Pop Culture and Entertainment:</b> Topics related to music, movies, TV shows, books, art, and entertainment.
      <br><b>Technology and Computing:</b> Topics related to computers, technology, software, internet, telecommunications, and mobile phones.
      <br><b>Miscellaneous:</b> Topics that do not fit neatly into any of the above categories, such as philosophy, religion, science, and humor.
      <br><b>Fashion and Beauty:</b> Topics related to fashion, clothing, makeup, cosmetics, hair care, and beauty.
      <br><b>Education and Careers:</b> Topics related to education, academic programs, scholarships, student loans, careers, and employment.
    </td>
  </tr>
  <tr>
    <td>Gold</td>
    <td><b>ArtsAndHumanities, BeautyAndStyle, BusinessAndFinance, ComputersAndInternet, ConsumerElectronics, EducationAndReference, EntertainmentAndMusic</b></td>
  </tr>
  <tr>
    <td rowspan="7">Yah (B)</td>
    <td>ZeroDL</td>
    <td><b>Food and Cooking:</b> This class includes topics related to various cuisines, recipes, food items, and cooking techniques.
      <br><b>Home Improvement and DIY:</b> This class includes topics related to home repair, renovation, decorating, gardening, and DIY projects.
      <br><b>Miscellaneous:</b> This class includes topics that do not fit neatly into any of the above categories, such as politics, education, art, and entertainment.
      <br><b>Technology and Gaming:</b> This class includes topics related to video games, computer hardware, software, technology, and internet culture.
      <br><b>Health and Wellness:</b> This class includes topics related to physical and mental health, nutrition, dieting, weight loss, fitness, exercise, and medical conditions.
      <br><b>Animals and Pets:</b> This class includes topics related to various animals, pet care, animal rights, and wildlife.
      <br><b>Religion and Philosophy:</b> This class includes topics related to various religions, theology, philosophy, and ethics.
    </td>
  </tr>
  <tr>
    <td>Gold</td>
    <td><b>FoodAndDrink, GamesAndRecreation, Health, HomeAndGarden, Pets, PregnancyAndParenting, SocietyAndCulture</b></td>
  </tr>
</table>

Table 8: Additional examples of generated class labels. Class title is marked as bold and its description is colored as gray.
<table>
  <tr>
    <th>Data</th>
    <th>TaskType</th>
    <th>Generated Class Labels (Descriptions are omitted)</th>
  </tr>
  <tr>
    <td rowspan="2">IMDB</td>
    <td>sentiment</td>
    <td>Negative, Very Positive, Highly Negative, Mixed, Neutral-Positive, Positive, Highly Positive, Neutral-Negative, Extremely Negative, Neutral; [Total 10]</td>
  </tr>
  <tr>
    <td>topic</td>
    <td>Film and Television Influence on Business, Film and Television Influence on Politics, Film and Television Production Companies, Film and Television Influence on Art, Film and Television Productions, Film Analysis or Review, Film and Television Influence on Society and Culture, Film and Television Influence on Technology, Film and Television Recommendations, Film and Television Awards, Film and Television, Film and Television Genres, Film and Television Influence on Society, Film and Television Marketing, Movie Reviews or Film Criticism, Film and Television Critics, Film and Television Technologies, Film and Television Festivals, Film or Media Criticism, Film and Television History, Film and Television Technology Trends, Film and Television Education, Film and Television Influence on Entertainment, Film and Television Influence on Education, Personal Opinion or Review, Film and Television Industry, Film and Television Distribution; [Total 27]</td>
  </tr>
  <tr>
    <td rowspan="2">SST</td>
    <td>sentiment</td>
    <td>The text expresses a negative sentiment towards the subject being described. The text expresses a negative or sad sentiment towards the film. The text expresses a negative or sad sentiment. The text has a negative tone. The text expresses a negative or cautionary sentiment. The text expresses a negative or slightly negative sentiment, Negative Sentiment. The text expresses a negative or slightly negative sentiment towards the film. The text expresses a negative sentiment. The text expresses a negative sentiment towards the movie being described. The text has a negative sentiment; [Total 11]</td>
  </tr>
  <tr>
    <td>topic</td>
    <td>Literature and Writing, Media and Entertainment, Mental Health and Psychology, Travel and Adventure, Film and Television, Family and Relationships, Science and Technology, Food and Cooking, Performing Arts, Sports, Education and Learning, Business and Finance, Greetings and Open-Ended Texts, Entertainment; [Total 14]</td>
  </tr>
  <tr>
    <td rowspan="2">Yelp Rev</td>
    <td>sentiment</td>
    <td>Positive Sentiment, Highly Negative Sentiment, Neutral to Positive Sentiment, Negative Sentiment, Mixed Sentiment, Neutral Sentiment, Very Positive Sentiment, Neutral to Negative Sentiment; [Total: 8]</td>
  </tr>
  <tr>
    <td>topic</td>
    <td>Automotive, Environment or Nature, Legal or Law, Customer Service or Business, Nightlife or Entertainment, Education or Training, Sports or Fitness, Discrimination or Racism, Religion or Spirituality, Technology or Gadgets, Health or Medical, Travel or Tourism, Education or Learning, Beauty or Personal Care, Customer Reviews or Testimonials, Politics or Government, Food or Cooking, Food or Beverage, Shopping or Retail, Home Improvement or Construction, Science or Technology, Real Estate or Housing, Food Safety or Food Poisoning, Entertainment or Leisure, Arts or Culture, Personal Care or Beauty, Business or Economy, Personal Experiences, Dining Experience or Food Review; [Total 29]</td>
  </tr>
  <tr>
    <td rowspan="2">AG News</td>
    <td>sentiment</td>
    <td>Negative: 12 expressions that contain negative sentiment, Positive: 25 expressions that contain positive sentiment, Negative: 12, Sentiment not clear: 3, Mixed: 11 expressions that contain a mix of positive and negative sentiment, Positive: 25, Neutral: 33, Neutral: 33 expressions that do not contain any clear positive or negative sentiment, Sentiment not clear: 3 expressions that do not provide enough context to determine a clear sentiment, Mixed: 11; [Total 10]</td>
  </tr>
  <tr>
    <td>topic</td>
    <td>Same Result with Table 2</td>
  </tr>
  <tr>
    <td rowspan="2">DBp (F)</td>
    <td>sentiment</td>
    <td>Neutral, Positive, Negative; [Total 3]</td>
  </tr>
  <tr>
    <td>topic</td>
    <td>Baseball, Aviation, Sports and Biographies, History, Aircraft, Soccer or Football, People and Biographies, Ice Hockey, Education, American Football, General, Aircraft Design, Music or Entertainment, Higher Education or Universities, Healthcare or Hospitals, Football or Soccer; [Total 15]</td>
  </tr>
  <tr>
    <td rowspan="2">DBp (B)</td>
    <td>sentiment</td>
    <td>Ambiguous, Neutral, Positive, Romantic, Negative, Mixed, Objective; [Total 6]</td>
  </tr>
  <tr>
    <td>topic</td>
    <td>Botany or Biology (specifically, Plant Science or Taxonomy), Botany or Endangered Species or Conservation Biology, Botany or Horticulture, Botany or Algae, Botany or Brazilian Flora, Botany or Cacti, Botany or Mexican Flora, Botany or Tillandsia species, Botany or Orchids, Botany or Aquatic Plants, Botany or Plant Science, Botany or Tropical Plants, Botany or Hawaiian Flora, Botany or Plant Taxonomy, Botany or Palm Trees [Total 15]</td>
  </tr>
  <tr>
    <td rowspan="2">Yah (F)</td>
    <td>sentiment</td>
    <td>Positive, Neutral, Mixed, Informational, Negative [Total 5]</td>
  </tr>
  <tr>
    <td>topic</td>
    <td>Miscellaneous (for topics that do not fit neatly into any specific category), Education and Careers, Makeup and Beauty, Gaming and Technology, Fashion and Clothing, Telecommunications, Housing and Real Estate, Music and Entertainment, Pop Culture and Entertainment, Personal Finance and Credit Scores; [Total 16]</td>
  </tr>
  <tr>
    <td rowspan="2">Yah (B)</td>
    <td>sentiment</td>
    <td>Neutral; [Total 1]</td>
  </tr>
  <tr>
    <td>topic</td>
    <td>Health and Medical Concerns, Pop Culture and Entertainment, Mental Health and Psychology, Literature and Writing, Business and Finance, Food and Cooking, Video Games and Technology, Art and Creativity, Philosophy and Ethics, Education and Learning, Humor and Satire, Sports and Fitness, Religion and Theology, Home Improvement and DIY, Science and Technology, Travel and Adventure, Pets and Animals, Politics and Society, Pregnancy and Reproductive Health; [Total 19]</td>
  </tr>
</table>

Table 9: The examples of generated class labels when no constraints are given; we experiment with different task type and unlimited the number of clusters.

Reviewer paper summaries: This paper presents a new prompting engineering approach to improve text classification. The approach involves first prompting the model to do classification and then clean and aggregate the outputs of this first step to generate metadata. The metadata is then used to build a second prompt which is used to generate the final predictions. The proposed approach is tested with respect to the quality of the generated clusters, where the quality is measured by looking if the resulting clusters are aligned with the underlying gold classes.

The method is essentially some form of prompt heuristic which I found not to be sufficiently motivated. The experiments are done on several datasets and illustrate the potential of the proposed approach. However, there is essentially no significant improvement with respect to simpler method of clustering LLM embeddings directly.

This paper proposes a methodology for clustering of text samples through zero-shot prompting of LLMs. The clustering is done in three stages. In the first stage, every text sample is assigned a label by the LLM without any restrictions. In the second stage, the model is prompted to reduce this list of labels into a subset of desired size. Finally, each text sample is assigned a final label from this subset, hence defining the final clusters. The paper also includes some experiments to demonstrate the importance of label selection for the clustering task.

## Problem description
The paper introduces a technique ZeroDL that aggregates the results of many open-ended zero shot inferences into a meta-information. This meta-info can then be incorporated in a new prompt for other tasks. The paper is particularly focused on showing how this technique can be used in text clustering tasks when the cluster types are unknown. 

## Proposed method
The approach consists of three steps. The first step refers to open-ended inference where the LLM is prompted to classify a given text based on a classification type. The exact set of class types are not disclosed. Instead a sentiment or general topic name is disclosed. This step is repeated for every text in the dataset.

Step 2 involves aggregating all the results from step-1 to generate the list of classes. It begins by counting the frequency of every class and then removes any one-off classes with a frequency of 1. This is important because step-1 is run without any constraints on the class count or exact class types. A list of subsets of the remaining classes are generated, by iteratively removing the class with the lowest frequency. The LLM is prompted to aggregate the classes to a required number.

Step-3 involves prompting the LLM to classify the text using the class names from Step-2. Essentially, it repeats Step-1 but constrains it to specific class names instead of being open ended. The authors tried two variants of prompts: (C+T) and (T+C). They differ in the way the prompt is structured. The text is the same in both prompts. (C+T) lists the class names before the input text whereas (T+C) lists the text before the class names.

## Performance
To evaluate performance of ZeroDL, they ran it on some data sets where the class names and the labels for each input text is available. The ground truth class labels for each input can be incoherent from the classes given out by the aggregation step of ZeroDL. The ordering can be different too. So the accuracy is calculated for all k! permutations of the class labels (k being the number of class labels) and highest accuracy count is reported in the tables.

The accuracy is compared against some other approaches. The notable ones are Gold and llm2vec+Kmeans. Gold refers to just using step-3 with the actual ground truth labels. Some other approaches like TF-IDF and SBERT had weaknesses as mentioned in Table-5 and hence were not used.
Summary of strengths: The paper performs experiments on several datasets, that illustrates the potential of the approach.

The proposed approach is a simple and intuitive method for text clustering without requiring generation of embeddings, and has been shown to be competitive with approaches that use embeddings with popular clustering algorithms like K-Means. This method would work without requiring very large context windows since only a single text sample is included in the LLM prompt at a time. The authors also use very short and straightforward prompts. Experiments where this approach sometimes outperforms scenarios where golden labels are provided, show the merit of the proposal.

It is a novel approach for open-ended clustering that does not require any embedding or pre-training. Despite that it shows competitive performance against embedding-based clustering methods (llm2vec) and in some cases it performs better than when the ground truth class names are provided.

Just using the step-1 and 2 of the algorithm can be useful in applications where you just want the different class types for the set of input texts. The resulting outputs are shown in Tables 2 and 9 as an example. In most cases they align well with the ground truth label names. In fact they also provide additional description and context for each class name, which is not usually present in ground truth labels.

The paper is well organized with appropriate sections, diagrams and tables as needed. The descriptions are generally clear and the examples in main text and appendix help a lot in understanding the individual steps and the overall strengths of ZeroDL.
Summary of weaknesses: The method is essentially some form of prompt heuristic which I found not to be sufficiently motivated. There is essentially no significant improvement with respect to simpler method of clustering LLM embeddings directly.

This paper seems to be largely influenced by the paper `Chen Huang and Guoxiu He. 2024. Text clustering as classification with llms` and the novel contributions seem to be limited to not providing any “seed labels” in the initial prompt, and experiments showing the importance of chosen labels. The original paper already provides a framework for the LLM to generate new labels and aggregate similar labels. There also seem to be some gaps in the explanation of the aggregation step. For example, there is no mention of descriptions in the stage 1 / 2 prompts but the examples in Table 2 and prompt 3 show descriptions. While the emphasis is on simple prompts, this means that the output could be largely unstructured with very different formats, requiring human intervention to interpret and parse the output.

The limitations section describes the weaknesses well. The first two limitations are particularly important here. 
1. The prompt to use for Step-3 is not very robust since the results are quite different when class names and input text are ordered in a different way. There is no consensus on what order to use.The authors have called out this weakness and mentioned that this has a potential of further research and is not the main focus of this paper. 

2. The limitation regarding diverse LLMs is important too, since they haven’t tested ZeroDL with some of the other current SOTA LLMs like gpt-4.

3. Other than that, the way of evaluating performance of clustering is not scalable since it requires k! Permutations. They had to break the Yahoo dataset into two to get over this issue. The coherence of the best permutation with the ground truth labels is not discussed.
Comments, suggestions and typos: It would be good to better explain an analyze the differences between the proposed approach and the approach based on clustering LLM embeddings directly.

- Illustrate stage 2 more clearly with an example - why iteratively construct subsets? 
- The motivation for dropping predictions with frequency 1 does not seem very clear. The authors state “save computation”, but it seems unlikely that few additional class labels would significantly impact computation cost? 
- Add a line in the prompt to produce structured outputs that can be machine parsed, hence enabling the automation of the algorithm. 
- This statement does not very clearly indicate the process: “we select the LLM outputs where the number of aggregated classes matches the pre-defined number of cluster classes. We then use the most frequent class pairs” 
- There seem to be several grammatical/formatting errors in the paper and a thorough critique would be helpful. Some examples:
     - The prompts in section 3 sometimes have “\n” characters and sometimes have new lines, reducing the readability.
     - Description of Table 1 has poor readability 
     - “U denotes the number of remained predictions” 
     - “We input all the subset”

1. The description for the aggregation step (ZeroDL step-2 on page 2) is a bit unclear. This is particularly for the part following the construction of possible subsets. It is not clear whether the prompt is run for every subset or just the biggest subset. This could be because of ambiguous notation since the prompt contains U classes, which is the set of all classes. Also the part regarding the selection of the most frequent class pairs is unclear. It would help a lot if there are examples of this step in an appendix table.

2. Expanding on the third weakness point, the performance evaluation criteria of choosing the permutation with the accuracy is not discussed well. The text should contain explanations of whether or not it is generally coherent with the ground truth labels. The appendix only contains the list of class labels generated, but not how coherent the best permutation label ids are with the ground truth label ids.
Overall assessment scores: 2.5

2.5

3.5
Metareview score: 3
Metareview: The paper presents ZeroDL, a three-stage zero-shot prompting framework for text clustering using LLMs. It generates open-ended labels, aggregates them into meta-information, and then refines predictions with constrained prompts to produce interpretable clusters. The goal is to obtain interpretable clusters without relying on embeddings or pre-defined labels.
Summary of reasons to publish: Some reviewers (va7N, 1VSX) praise the novelty of the method and clarity of presentation. Others appreciate that the method yields interpretable labels and competitive results in some cases.