Image Clustering Conditioned on Text Criteria
Authors: Sehyun Kwon
Jaeseung Park
Minkyu Kim
Jaewoong Cho
Ernest K. Ryu
Kangwook Lee
Venue: ICLR 2024 poster (2024)
Keywords: image clustering
vision-language models
large language models
foundation models
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a novel image clustering method that perform clustering based on a user-specified criterion.

Abstract: Classical clustering methods do not provide users with direct control of the clustering results, and the clustering results may not be consistent with the relevant criterion that a user has in mind. In this work, we present a new methodology for performing image clustering based on user-specified criteria in the form of text by leveraging modern Vision-Language Models and Large Language Models. We call our method Image Clustering Conditioned on Text Criteria (IC$|$TC), and it represents a different paradigm of image clustering. IC$|$TC requires a minimal and practical degree of human intervention and grants the user significant control over the clustering results in return. Our experiments show that IC$|$TC can effectively cluster images with various criteria, such as human action, physical location, or the person's mood, significantly outperforming baselines.

Published as a conference paper at ICLR 2024
IMAGE CLUSTERING CONDITIONED ON TEXT CRITERIA
Sehyun Kwon ! Jaeseung Parkí, Minkyu Kim JJewoong Cho Ernest K. Ryul", Kangwook Lee 2 &
SSoul National University, KRAFTON, University of
Co-senior authors
ABSTRACT
Classical clustering methods do not provide users with direct control of the clus- tering results, and the clustering results may not be consistent with the relevant criterion that a user has in mind. In this work, we present new methodology for performing image clustering based on user-specified text criteria by leveraging modern vision-language models and large language models. We call our method Image Clustering Conditioned on Text Criteria (IC)TC), and it represents a differ- ent paradigm of image clustering. IC/TC requires minimal and practical degree of human intervention and grants the user significant control over the clustering results in return. Our experiments show that TC effectively cluster im- ages with various criteria, such as human action, physical location, or the person's s mood, while significantly outperforming basclines."
INTRODUCTION
Image clustering has been studied as a prototypical unsupervised learning task, and it has been used organize large volumes of visual data (Platt et al., 2003), reduce the cost of labeling an unlabeled image dataset (Russell et al., 2008; Schmarje et al., 2022), and to enhance image retrieval systems (Wu et al., 2000; Jégou and Chum, 2012). Modern deep image clustering methods are often evaluated against pre-defined class labels of datasets viewed as the ground truth.
In practice, however, a user may have a criterion in mind for how to cluster or organize a set of images. The user may even want multiple clustering results of the same dataset based on differ- ent criteria. (See Figure 1.) But, classical clustering methods offer no direct mechanism for the user to control the clustering criterion; the clustering criteria for existing methods are likely deter- mined by the inductive biases of the neural networks and the loss function, data augmentations, and feature extractors used within the method. This necessitates new paradigm in image clustering, enabling diverse outcomes from a single dataset based on user-specified criteria and revolutionizing the conventional, implicitly dictated clustering processes.
Recently, foundation models have received significant recent interest due to their ability to under- stand and follow human instructions at unprecedented level. Large language models (LLMs) (Brown et al., 2020; Chowdhery et al., 2022: Touvron et al., 2023a;b; Chiang et al., 2023; OpenAI, 2023; Adams et al., 2023) perform remarkably well on a wide range of natural language tasks such as understanding, summarizing, and reasoning in zero- or few-shot settings. Vision-languag mod- els (VLMs) (Alayrac al., 2022; Liu al., 2023; Awadalla al., 2023; Dai al., 2023; Li al., 2023a; Zhu et al., 2023; Gong et al., 2023) interpret natural language instructions in visual contexts and produce responses that seemingly exhibit in-depth image analyses and complex reasoning.
In this work, we present a new methodology based on foundation models for performing image clustering based on user-specified criteria provided in natural language text. We call our method Image Clustering Conditioned on Text Criteria (IC)TC), and it represents a different paradigm of image clustering: the user directs the method with the relevant clustering criterion, the same dataset can be clustered with multiple different criteria, and if the clustering results not satisfactory, the user can edit the text criterion to iteratively refine the clustering results. IC|TC requires a minimal and practical degree of human intervention and grants the user significant control over the clustering results in return, and we argue that this makes IC|TC more practical and powerful compared to the classical purely unsupervised clustering methods.
Work done KRAFTON Our code is available http:es///gttha....o//eeeeeoooo
com wttp:/swwe/y//e/eee//o ICrC.

---

Published as a conference paper at ICLR 2024
Blowing bubbles
Applauding
Jumping
Shooting an arrow
Criterion
Action
Restaurant
Educational institute
Store
Sports facility
Criterion
Location
Joyful
Adventurous
Focused
Relaxed
Criterion
Mood
(a) Sample images from the clustering results on Stanford 40 Action datasct. Each result obtained using a different text criterion: Act on, Location i and Mood.
Brass instruments
String instruments
Criterion
Instrument
(K=2)
Trumpet
Flute
Cello
Guitar
Criterion
Instrument
(K=7)
(b) Sample images from the clustering results on the PPMT dataset using the text criterion Instrument with different cluster numbers K 2 and 7.
Figure 1: Sample images from clustering results of IC|TC. The method finds clusters consistent with the user-specified text criterion. Furthermore, IC/TC provides cluster names (texts above each image cluster) along with the clusters, enhancing the interpretability of clustering results.
2

---

Published as a conference paper at ICLR 2024
1.1 CONTRIBUTION
Our main contributions are the proposal of the novel task of image clustering conditioned on text criteria and our method IC/TC for solving this task. The task is interesting because the setup where the user is willing and able to provide a textual description of the clustering criterion is practical, arguably more practical than the classical purely unsupervised clustering setup. The method TC|TC is interesting because it leverages modern multi-modal foundation models and solves the task well; our experiments demonstrate that IC can indeed produce satisfactory clustering results consistent with the user-specified criteria.
2 TASK DEFINITION: IMAGE CLUSTERING CONDITIONED ON ITERATIVELY REFINED TEXT CRITERIA
The main task we consider in this work defined as follows: Given a set of images, a number of clusters K, and a user-specified criterion expressed in natural language, partition the set of images into K clusters such that the semantic meanings the clusters are distinguished in a manner that is consistent with the specified user criterion.
Recent image clustering methods (Van Gansbeke et al., 2020; Park et al., 2021; :; Niu and Wang, 2021) find clusters that agree with pre-defined class labels for datasets such as CIFAR-10 (~90% accu- racy). The semantic meanings of the clusters tend to correspond to the category of the foreground object, and the inductive biases of the neural networks and the loss function, data augmentations, and feature extractors used within the method are likely the cause of the clusters being chosen in this manner. In given setup, however, the clusters returned by such classical clustering methods may not be consistent with the relevant criterion that a user has in mind.
Iterative refinement of text criteria. Under our main task, the text criterion is chosen through process of iterative refinement: The user specifies a text criterion, performs clustering, examines the clustering results, and, not satisfied, edits the text criterion to iteratively refine the clustering results. Sometimes, user-defined text criterion immediately leads to a clustering result that is sufficiently consistent with what the user has in mind, but if not, this iterative prompt engineering procedure provides practical means for converging desired results. In practice, hyperparameters of classical clustering algorithms are chosen through an iterative process where the user inspects the clustering output and adjusts the parameters accordingly. In this work, explicitly acknowledge the process of iteratively determining the text criterion and consider it to be part of the main task.
Comparison with classical clustering. Our task differs from classical clustering in that the user provides information characterizing the relevant criterion by which the images should be clustered. In contrast, classical clustering methods are purely unsupervised and use no such information.
Deep clustering methods are often evaluated against pre-defined set of labels of a dataset, and such labels tend to focus on the type of object in the foreground. However, the question of how clustering algorithms could (or cannot) perform clustering with arbitrary criteria has been raised and studied in several prior works (Wolpert and Macready, 1997; Kleinberg, 2002; Caruana et al., 2006; Cui et al., 2007; von Luxburg et al., 2012; Caruana, 2013; McCarthy et al., 2020; Viswanathan et al., 2023). The use of user-defined text criteria makes our task not an instance of (classical) unsupervised clustering, but providing text criterion is necessary and practical intervention from the user if the goal is to perform clustering with arbitrary criteria.
Comparison with zero-shot classification. Our task differs from zero-shot classification in that zero-shot classification requires pre-defined set of classes, and the goal is merely to assign images to these classes. contrast, our task requires both finding the clusters and assigning images to the clusters. In fact, zero-shot classification can be considered an instance of our task when the user explicitly and precisely describes all K clusters in the clustering criterion.
3

---

Published as a conference paper at ICLR 2024
Text Criterion (TC): "Cluster based on the main [Object]"
STEP
STEP 2
STEP
TC Refinement (Optiona)
Lang
priden Inttonte las the cammar
Tarlegg
C2
upcates
accord nelv and ale.
The Gemero
Cx
images
LLM
LLM
VLM
Carriveree
N Rew Labels
Thatang
The polcen looking m the camera
LLM
C: Dmn VorBe
C. c.]
C: Vehicles
N Descriptions
K Clusters
Clustered Results
Figure 2: The TC method. (Step Vision-language model (VLM) extracts detailed relevant textual descriptions of images. (Step 2) Large language model (LLM) identifies the names of the clusters. (Step 3) LLM conducts clustering by assigning each description to the appropriate cluster. The entire procedure is guided by user-specified text criterion (TC). (Optional TC Refinement). The user can update the criterion if the clustering results are unsatisfactory. See Appendix B.4 for an unabridged sample output.
TC|TC: IMAGE CLUSTERING CONDITIONED ON TEXT CRITERIA
Our main method consists of 3 stages with optional iterative outer loop. The user-specified text criterion TC is incorporated into 3 stages via text prompts roughly of the following form.
Psepl TC)) = "Chracttriie the image using a well-detailed description: + TC
P Parpaa (T)) = "Given description of an image, label the image" + TC
P Pe (TC, N) (K) "Given 1ist of (N) labels, cluster then into {K} words" TC
Papps (CC) "Based on the image description,
determine the most appropriate cluster" TC
The precise prompt for each experimental setup in this work is specified in Appendix B.3.
3.1 STEP 1: EXTRACT SALIENT FEATURES FROM THE IMAGE
Step 1, the lision-language model (VLM) extracts salient features from the image the form of text descriptions.
Step 1 Vision- language model (VLM) extracts salient features
Input: Image Dataset Ding. Text Criteria TC, Descriptions Dées +0
Output: Ddes
1: for img in Dimg do
2: append) VLM(img, Pstepp (TC)) / //append image description Daes
3: end for
The user's criterion TC determines the relevant features the VLM should focus on. For example, the user may wish to cluster with respect the mood of a person in the image the overall mood (atmosphere) the scene. In such cases, the TC may slightly vary:
Criterion 1: Focus en the mood of the persen in the center.
Criterion 2: Describe the general mood by inspecting the background.
4

---

Published as a conference paper at ICLR 2024
3.2 STEP 2: OBTAINING CLUSTER NAMES
In Step 2, the large language model (LLM) discovers the cluster names in two sub-steps. In Step 2a, the LLM outputs raw initial labels of the images. Since the number of distinct initial labels is usually larger than K, in Step 2b, the LLM aggregates the raw initial labels into appropriate names of K clusters. (Combining Steps 2a and 2b and asking the LLM to discover K cluster names from N image descriptions is infeasible due to the limited token lengths of the LLMs.)
Step 2 Large Language Model (LL.)) obtains K cluster names
Input: Descriptions Dees. Text Criteria TC, Dataset size N, Number of clusters K, Lraw +-] Output: List of cluster names Camme
1: for description in Ddes do
2: Lram
(descriiiptiio P stepp2a (TC)) ) // //append raw label to fraw
3: end for
4: Cname = =LL stop2b (TC,NKK) //Step 2b can be further optimi zed
The simplest instance Step 2b, described above, directly provides fraw- the full list of raw labels. However, we find that it more efficient convert Lraw dictionary with labels being the keys and numbers of occurrences of the labels being the values. When the same raw label occurs many times, this optimization significantly reduces the token length of the input to the LLM of Step 2b.
Careful prompt engineering of P Pupp2 (TC, N,, K) allows the user to refine the clusters to be consis- tent with the user's criteria. For example, the user may append additional text prompts such as:
When categorizing the classes, consider the following criteria:
1. Merge similar clusters. For example, [sparrow, eagle, falcon, owl, hawk] should combined into 'birds of prey.'
2. Clusters should be differentiateo, based on the animal's habitat.
3.3 STEP 3: CLUSTERING BY ASSIGNING IMAGES
In Step images are assigned one of the final K clusters. The text criterion TC, text description of the images from Step 1, and the K cluster names from Step 2 are provided to the LLM.
Step 3 Large Language Model (LLM) assigns clusters images
Input: Descriptions Dees. Text Criteria TC, List cluster names Cname. RESULT+D
Output: RESULT
I: for description in Daes do
2: RESULT.append LLMddescrrpptionn-PPan (TC))) //append assigned cluster
3: end for
3.4 ITERATIVELY EDITING THE ALGORITHM THROUGH TEXT PROMPT ENGINEERING
Main method TC
Input: Dataset D img. Text Criteria TC, ADJUST True
1: while ADJUST do
2: RESULT do Steps 1-3 conditioned TC
3: if User determines RESULT satisfactory then
4: ADJUST <- False
5: else
6: TC < Update TC
//user writes updated TC
7: end if
8:
end while
Our main method IC|TC is described above. Upon performing the clustering once, the clusters are not sufficiently consistent with the specified text criterion TC if the TC turns out to not precisely specify what the user had in mind, the user can update the TC. This iterative process may continue until the clustering result is satisfactory, as judged by the user.
5

---

Published as a conference paper at ICLR 2024
Table 1: Clustering with varying criteria. Accuracies labeled with are evaluated by having human provide ground truth labels for 1000 randomly sampled images. In
this experiment, we used LLaVA for VLM and GPT-4 for LLM.
Accuralyy
Dataset
Criterion
SCAN
Ours
Liama 2 (78)
Liama 2 (138)
Act ion
0.397
0.774
(708)
Stanford 40 Action
Location
0,359
0.822
STL-10
GPT:3.5
Mood
0.250
0.793*
CIFARR-10
GPT4
M.I. (K-7)
0.632
0.964
Number of Parameters
PPMI
M.I. (K=2)
0,850
0.977
Location (K-2)
0.512
0.914
Figure 3: Effect LLM selection.
CIFAR-IO-OR 10-Gen
Object
0.989
0.987
3.5
PRODUCING CLUSTER LABELS
Classically, the unsupervised clustering task does not require the method to produce labels or de- scriptions of the output clusters. Notably, however, IC| produces names describing the clusters. This is a significant advantage of IC|TC as makes the clustering results more directly and imme- diately interpretable.
4 EXPERIMENTS
We present experimental results demonstrating the effectiveness of IC|TC. In this section, partially describe the settings and results while deferring much of the details to the appendix. In particular, the precise prompts used can be found in Appendix B.
ICJTC crucially relies on the use of foundation models, specifically a vision-language model (VLM) and large language model (LLM) that have undergone instruction tuning. In experiments, we mainly use LLaVA (Liu et al., 2023) for the VLM and GPT-4 (OpenAI, 2023) for the LLM, but Section 4.5 and Appendix B.2 presents ablation studies investigating how the performance is affected when other foundation models are used.
4.1 CLUSTERING WITH VARYING TEXT CRITERIA
In this experiment, we show that varying the text criterion TC indeed leads to varying clustering results of single image dataset. The results demonstrate that IC|TC is highly flexible and can accommodate a variety of text criteria.
We use the Stanford 40 Action Dataset (Yao et al., 2011), 1), which contains 9,532 images of hu- mans performing various actions. The dataset comes with image labels describing a subject's action among 40 classes, such as reading, phoning, blowing bubbles, playing violin, etc. We additionally define two different collections of labels. The first collection contains 10 classes describing the locat ion, such as restaurant, store, sports facility, etc. The second collection contains 4 classes describing the mood of the scene, specifically joyful, adventurous, relaxed, and focused.
We utilize three text criteria, Action, Locat ion, and Mood, obtain three distinct clustering results. We evaluate the results based on how accurately the methods recover the three collections of labels described previously. This degree control would be difficult or impossible for classical deep clustering methods. We compare our results against the prior deep clustering method SCAN (Van Gansbeke et al., 2020) and present the results Table 1. Image samples are in Figure
(Note that we do not have the 9,532 ground truth labels for the Locat on and Mood criteria. Therefore, we evaluate accuracy by having human provide ground truth labels on 1000 randomly sampled images.)

---

Published as a conference paper at ICLR 2024
Outdoor
Indoor
Criterion
Location
(K=2)
Sample images from the clustering results on the PPMI dataset using text criterion Locat in and cluster number K 2.
Figure 4:
4.2 CLUSTERING WITH VARYING GRANULARITY
In this experiment, show that TC can automatically control the granularity of clustering results by adjusting K, the number of clusters. We find that the cluster descriptions returned by IC|TC are highly interpretable and that the images assigned to the clusters well for various values of K.
We use the People Playing Musical Instrument (PPMI) dataset (Wang et al., 2010; Yao and Fei-Fei, 2010), which contains 1,200 images of humans interacting with 12 different musical instruments. We select 700 images across 7 classes from the original dataset to reduce the size and difficulty of the task.
We use the text criterion Musical ccll Inst rument with number of clusters K = 2 and = 7. With K 7, images are indeed grouped into clusters such as violin, guitar, and other specific instruments, and 96.4% accuracy against the ground truth label of PPMI is achieved. With K = 2, images are divided into clusters of brass instrument and string instrument and achieve 97.7% accuracy. To clarify, we did not specifically instruct IC|TC to group the 7 instruments into brass and string instruments; the hierarchical grouping was discovered by IC/TC.
As an additional experiment, we also cluster the same set of images with the text criterion Locat on and
In this case, the images are divided into 2 clusters indoor and out- door, and achieve a 91.4% accuracy. We again compare results against SCAN (Van Gansbeke et al., 2020) and present the results in Table 1. Image samples are provided in Figure 4.
4.3 COMPARISON WITH CLASSICAL CLUSTERING METHODS
In this experiment, we compare IC/TC against several classical clustering algorithms on CIFAR- 10, STL--10, and CIFAR-100. The three datasets have 10, 10, and 20 classes and 10,000, 8,000, and 10,000 images, respectively. We use the text criterion Ob ject with the number of clusters equal to the number of classes in the dataset. The results in Table 2 show that IC|TC significantly outperforms classical clustering methods on CIFAR-10, STL-10 and CIFAR-100. Clustered sample images provided in Appendix B.6.
This comparison is arguably unfair against the classical clustering methods as they do not utilize foundation models or any pre-trained weights. Nevertheless, results demonstrate that IC is competitive when the goal is to cluster images based on the foreground object type.
4.4 FAIR CLUSTERING THROUGH TEXT CRITERION REFINEMENT
Existing clustering methods sometimes exhibit biased results, and measures to mitigate such biases have been studied (Li et al., 2020; Zeng al., 2023). Since foundation models are known to learn biases in their training data (Bommasani et al., 2022), IC/TC has the risk of propagating such biases into the clustering results. In this experiment, show that by simply adding prompt along the line of "Do not consider gender" to the text criterion, we can effectively mitigate biases in the clustering results.
FACET (Gustafson et al., 2023) a benchmark dataset for evaluating the robustness and algorithmic fairness of AT and machne--learning vision models. It comprises 32,000 diverse images labeled with

---

Published as a conference paper at ICLR 2024
Table Comparison with classical clustering methods using criterion Object. IC/TC outperforms state-of-the-ar methods on CIFAR-10, 10, STL-10 and CIFAR-100.
CIFAR-10
STL-10
CIFAR-IO0
Method
ACC
NMI
ARI
ACC
NMI
ARI 1
ACC
NMI f
ARI
IIC (Ji
(2019))
0.617
0.511
0.411
0.596
N/A
N/A
0.257
N/A
SCAN (Van Gansheke al. (2020))
0.8K3
0.797
0.772
0.809
0.698
0.646
0.507
0.468
0.301
SPICE (Niu and Wang (2021))
0.926
0.865
0.852
0.938
0,872
0.870
0.584
0,583
0.422
RUC (Park (2021))
0.903
N/A
0.867
N/A
N/A
0.543
N/A
TCL (Yunfan (2022))
0.887
0.819
0.780
0.868
0.799
0.757
0.531
0.529
0.357
LLaVA only
0.647
0.455
0.442
0.774
0.587
0.589
0.097
0.022
0.014
Ours (LLaVA+ Llama 2)
0.884
0.789
0.759
0.974
0.939
0.944
0.526
0.554
0.374
Ours (BLIP-2- + GPT-4)
0.975
0.941
0.947
0.993
0.982
0.985
0.584
0.690
0.429
Ours (LLaVA GPT-4)
0.910
0.823
0.815
0.986
0,966
0.970
0.642
0.422
Base prompt
Fair prompt
Samples
22
Malw Raila 19)
Actual
Craftsman
Famala Ratio (%)
Prediction
Laborer
Craftsman Dancer Gardener Craftsman Laborer
Dancer Gardener
(a) Biased results
(b) Gender ratio
Figure 5: (a) Biased results showing that male 'Craftsman' tend be misclassified as 'Laborer', (b) Gender ratio of each cluster. When the ratio between males and females differs by more than 10%, the bar is colored red. Bias mitigated by refining the text criterion into a 'Fair prompt'.
several attributes, including 52 occupation classes. For this experiment, we sampled 20 images each for men and women from the craftsman, laborer, dancer, and gardener occupation classes, 160 images in total.
For this experiment, define faimess to be achieved when each cluster maintains equal pro- portion of genders. When we use the text criterion accupation, IC|TC exhibited a gender bias. To mitigate this bias, we introduced simple negative prompt, instructing IC|TC to take gen- der into consideration and instead to focus on the activity. When the clustering was repeated, the results were promising: the gender ratio disparities in the craftsman and laborer clusters improved by 27.2% + 4.4% and 11.6% 3.2%, respectively. Furthermore, the Dancer and Gardener clus- ters also experienced marginal reductions in disparities by 2.8% > 2.6% and 10.6% 9.0%, respectively. The results are shown in Figure 5,
4.5 FURTHER ANALYSES
Ablation studies of LLMs and VLMs. We conduct an ablation study evaluate whether LLMs actually serve significant role in our methodology since one may wonder whether the vision- language model (VLMs) alone is sufficient. When we perform 'LLaVA only' experiment that does not utilize an LLM,, the performance is considerably lower. However, when we use L.MMs of varying sizes, the performance not affected significantly. The results and details are provided in Figure and Appendix A.2. The results lead to conclude that the LLM serves crucial role (the VLM by itself is not sufficient), but the size of the LLM does not seem to be very important.
We also fix the LLM to GPT-4 and perform ablation study on the choice of vasion-language model (VLM). As an image captioning model, ClipCap (Mokady et al., 2021) canmot perform text condi- tioning, and this leads to poor performance. Blip-2 2 (Li al., 2023b) and LLaVA (Liu et al., 2023) can extract information relevant to the text criteria, and they exhibit strong strong performance. The results and details provided in Appendix A.I.
8

---

Published as a conference paper at ICLR 2024
Data Contamination. When evaluating research using foundation models, the potential of data contamination is a significant concern (Wei et al., 2022; Du et al., 2022). The datasets we use to measure accuracy, namely CIFARIO, STL-10, CIFAR- 100, and Stanford 40 Action, may have been used in the training of LLaVA, If so, the validity of the accuracy measurements comes into question.
To address this concern, we conducted experiment with synthetically generated images. Specifi- cally, we use Stable Diffusion XL (Rombach et al., 2022) and the CIFAR-10 10 labels to generate 1000 10-like images, and we call this dataset CIFAR-10-Gen. See Appendix B for further details. On this synthetic data, IC|TC achieves 98.7% accuracy. The fact that the accuracy on CIFAR-IC 10- Gen no worse than the accuracy on the actual CIFAR- 10 dataset gives us confidence that the strong performance of IC|TC is likely not due to data contamination.
(Strictly speaking, the training data for Stable Diffusion may contain the CIFAR-10 images, and if so, we are not completely free from the risk of data contamination. However, the CIFAR-10- dataset does not seem to contain exact copies of CIFAR- 10 images, and we argue that the synthetic generation significantly mitigates the risk of data contamination.)
5 RELATED WORK
Image clustering. Modern deep clustering methods (Van Gansbeke et al., 2020; Park et al., 2021; Niu and Wang, 2021; Yunfan et al., 2022) adopt a multi-stage training approach. They begin with representation learning. which finds a representation that maps similar images to similar features, and then perform unsupervised clustering based on these feature representations. Additionally, to obtain more meaningful semantics, Zhong et al. (2021); Shen et al. (2021) proposed contrastive learning at not only the instance level but also at the cluster level. Misra and Maaten (2020); Cho al. (2021); Kwon et al. (2023); Long et al. (2023); Metaxas al. (2023) proposed specially de- signed representation learning for certain clustering criteria. The concurrent work Li et al. (2023c) particularly relevant our work as it presents Text-Aided Clustering (TAC), which leverages text as external knowledge to enhance image clustering performance. Specifically, Li et al. (2023c) enhanced feature discriminability by selecting specific WordNet nouns of images and mutually dis- tilled the neighborhood information between the text and image modalities.
Foundation models. In recent years, foundation models have been improving at a remarkable pace, and combined with instruction tuning (Sanh et al., 2022: Ouyang et al., 2022: Wei et al., 2022), these foundation models can be applied more flexibly downstream tasks. Vision-Language Models (VLMs) (Alayrac et al., 2022; Liu et al., 2023; Awadalla et al., 2023; Dai et al., 2023; Li et al., 2023a; Zhu et al., 2023; Gong et al., 2023) can provide users with appropriate descriptions of given images according to the requirements of the input prompt. Large language models (LLMs) (Chowdhery et al., 2022: Touvron et al., 2023a;b; OpenAl, 2023) exhibit remarkable abilities in wide range of natural language processing tasks such as text summarization. Recently, Radford et al. (2021); Jia et al. (2021); Li et al. (2022); Dinh et al. (2022); Geng et al. (2023); Menon and Vondrick (2023); Zhang al. (2022); Cai al. (2023); Ren al. (2023) have shown computer vision problems with no direct connection to language can be successfully addressed using large language models.
Image retrieval. Image retrieval aims to find images from a database that are relevant to a given query. This crucially differs from clustering in that clustering requires both finding the clusters and assigning the images to them; image retrieval techniques are very relevant to the sub-task of clus- ter assignment but not the sub-task of finding the clusters. The fundamental approach in image retrieval is to assess the similarity among image features. Current approaches focus on two kinds of image representations: global features and local features. For global representations, Babenko et al. (2014); Tolias et al. (2015); Gordo et al. (2016): Cao et al. (2020): Lee et al. (2023) extracts activations from deep CNNs and aggregates them for obtaining global features. For local represen- tations, Yi et al. (2016); Noh et al. (2017); Vassileios Baintas and Mikolajezyk (2016); DeTToee et al. (2018); He et al. (2018); Dusmanu et al. (2019); Revaud et al. (2019) proposed well-embedded rep- resentations for all regions of interest. Recent state-of-the-ard methods (Noh et al., 2017; Simeoni et al., 2019; Cao et al., 2020; Zhang et al., 2023; Wu et al., 2023) typically followed a two-stage paradigm: initially, candidates are retrieved using global features, and then they are re-ranked with local features. Recently, Vo et al. (2019); Liu et al. (2021); Baldrati et al. (2022); Tian et al. (2023) proposed to condition retrieval user-specified language.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
EKR was supported by the National Research Foundation of Korea (NRF) Grant funded by the Korean Government (MSIP) INRF-202RICICIO10001 and the Prooerie-Ponneering Researchers Program through Seoul National University. SK and EKR were partly supported by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Ko- rean government (MSIT) [NO.2021 -0-01 343-004, Artificial Intelligence Graduate School Program (Seoul National University)]. We thank Dimitris Papailiopoulos and Yong Jae Lee for providing insightful discussion. We thank Byeong-Uk Lee for providing valuable feedback on the manuscript.
ETHICS STATEMENT
Our methodology provides users with direct control over the clustering results, but this agency could be used maliciously to produce unfair and discriminatory results. However, it is unlikely that our work will be responsible for new unfair results that could not already be produced with malicious user's i direct and overt intervention. On the other hand, is possible for biases already in foundation models propagate into our clustering methodology. Section 4.4 explicitly discusses this possibility and offers measures to mitigate such biases, and a well-intentioned user following the guidance of Section 4.4 is unlikely to amplify biases in the foundation models through the use of our method.
REPRODUCIBILITY STATEMENT
In this work, we use publically available datasets, describe the methodology in precise detail, and make our code available at https: /ttp::/ wtt.//omm/ssseeee. http//shhyunwwwo/ICCCC ICCC. Of the two main foundation models we use, the vision-language model LLaVA (Liu et al., 2023) is fully open-source. However, the large language model GPT-4 (OpenAI, 2023) is a proprietary model, and we accessed it through the API offered by OpenAL. The API cost to conduct the experiments presented in this work was less than $3,000 (USD), so we argue that the proprietary API cost does not pose a signifi- cant barrier in terms of reproducibility. However, if OpenAl were to discontinue access to the GPT-4 version that we used, namely api wotrssionn223000 10-00031111p--- 1-------ppiii.. or if OpenAl discontin- ues access to GPT-4 altogether, then our experiments will no longer be exactly reproducible.
To address this concern, we carry out an ablation study that uses the open-source large language model Llama 2 (Touvron et al., 2023b) and observe that similar, albeit slight worse, performance is attained. See Figure and Appendix A.2. Therefore, even if GPT-4 becomes unavailable in the future, the results of this work will be similarly reproducible by using Llama 2 any other large language model of power comparable to or stronger than Llama 2 and GPT-4.
REFERENCES
G. Adams, A. Fabbri, Ladhak, E. Lehman, and N. Elhadad. From sparse dense: GPT-4 sum- marization with chain of density prompting, trXiv:2309.04266 2023.
J.B. Alayrac, J. Donahue, P. Luc, A. Miech, Barr, Y. Hasson, K. Lene, A. Mensch, K. Millican, M. Reynolds. R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei. M. Monteiro, Menick, Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, Barreira, O. Vinyals, A. Zisserman, and K. Simonyan. Flamingo: a visual language model for few-shot learning. Neural Information Processing Systems, 2022.
A. Awadalla, I. Gao, J. Gardner, J. Hessel, Y. Hanafy, W. Zhu, K. Marathe, Y. Bitton, S. Gadre, S. Sagawa, J. Jitsev, S. Kornblith, W. Koh, G. Ilharco, M. Wortsman, and L. Schmidt. Open- Flamingo: An open-source framework for training large autoregressive vision-language models. PrXiv:2308.013900 2023.
A. Babenko, A. Slesarev, A. Chigorin, and V. Lempitsky. Neural codes for image retrieval. European Conference on Computer Vision, 2014.
A. Baldrati, M. Bertini, T. Uriechio, and A. Del Bimbo. Conditioned and composed image retrieval combining and partially fine-tuning clip-based features. Conference on Computer Vision and Patlern Recognition, 2022.
10

---

Published as a conference paper at ICLR 2024
R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chat- terji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, I.. Fei-Fei, C. Finn, T. Gale, L. Gillespie, K. Goel, N. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong. K. Hsu, J. Huang, T. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khattab, W. Koh, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. Niebles, H. Nilforoshan, J. Nyarko, G. Ogut, L. Orr, I. Papadimitriou, J. S. Park, C. Piech, E. Porte- lance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. Roohani, C. Ruiz, J. Ryan, C. Ré, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramèr, R. E. Wang. W. Wang. B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, You, M. Za- haria, M. Zhang. T. Zhang, X. Zhang. Y. Zhang, L. Zheng. K. Zhou, and P. Liang. On the 2022.
opportunities and risks of foundation models.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, al. Language models are few-shot learners. Neural Information Processing Systems, 2020.
S. Cai, L. Qiu, X. Chen, Q. Zhang, and L. Chen. Semantic-enhanced image clustering. American Association for Artificial Intelligence, 2023.
B. Cao, A. Araujo, and J. Sim. Unifying deep local and global features for image search. European Conference on Computer Vision, 2020.
R. Caruana. Clustering: Probably approximately useless? ACM International Conference on Infor- mation & Knowledge Management, 2013.
R. Caruana, M. Elhawary, N. Nguyen, and C. Smith. Meta clustering. International Conference on Data Mining, 2006.
W.-L. Chiang, Z. Li, Lin, Y. Sheng. Z. Wu, H. Zhang, L. Zheng, S. Zhuang. Y. Zhuang, J. Gon- zalez, I. Stoica, and E. P. Xing, Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality, 2023. URL https: //msysmm..s////////////sss///s./ htrgg//lg///// 22000/ -0-30 -3-30-vicunal.
J. H. Cho, U. Mall, K. Bala, and B. Hariharan. PICIE: Unsupervised semantic segmentation using invariance and equivariance in clustering. Computer Vision and Pattern Recognition, 2021.
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. M. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. C. Hutchinson, R. Pope, J. Brad- bury, Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski, X. García, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, Zoph, A. Spiridonov. R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, S. Pillai. M. Pellat, A. Lewkowycz, E. Moreira. R. Child, O. Polozov. K. Lee, Z. Zhou, X. Wang, B. Sacta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. S. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, and N. Fiedel. PaLM: Scaling language modeling with pathways. arXiv:2204.02311, 2022.
Y. Cui, X. Fern, and J. G. Dy. Non-redundant multi-view clustering via orthogonalization. Inter- national Conference on Data Mining, 2007.
W. Dai, Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi. InstructBLIP. Towards general-purpose vision-language models with instruction tuning. (.Xiv:2305.06500 2023.
D. DeTone, T. Malisiewiez, and A. Rabinovich. Superpoint: Self-supervised interest point detection and description. Computer Vision and Pattern Recognition Workshops, 2018
T. Dinh, Y. Zeng. R. Zhang, Z. Lin, M. Gira, S. Rajput, J. yong Sohn. D. Papailiopoulos, and K. Lee. LIFT: Language-interficn interfaced fine-tuning for non-language machine learning tasks. Neural Information Processing Systems, 2022.
11

---

Published as a conference paper at ICLR 2024
N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang. E. Wang. K. Webster, M. Pellat, K. Robinson, K. Meier-Hellsterm, T. Duke, L. Dixon, K. Zhang, Q. Le. Y. Wu, Z. Chen, and C. Cui. GLaM: Ef- ficient scaling of language models with mixture-of-experts. International Conference on Machine Learning. 2022.
M. Dusmanu, Rocco, T. Pajdla, M. Pollefeys, J. Sivic, A. Torii, and T. Sattler. D2-net: A trainable cnn for joint description and detection of local features. Conference Computer Vision and Pattern Recognition, 2019.
S. Geng, J. Yuan, Y. Tian, Y. Chen, and Y. Zhang. HiCLIP: Contrastive language-image pretraining with hicrarchy-aware attention. International Conference on Learning Representations, 2023.
T. Gong, C. Lyu, S. Zhang, Y. Wang, M. Zheng, Q. Zhao, K. Liu, W. Zhang. P. Luo, and K. Chen. Mut:: vision and language model for dialogue with humans. 0tti::2305.04790 2023.
A. Gordo, J. Almazán, J. Revaud, and D. Larlus. Deep image retrieval: Learning global representa- tions for image search. European Conference on Computer Vision, 2016.
L. Gustafson, C. Rolland, N. Ravi, Q. Duval, A. Adcock, C.-Y. Fu, M. Hall, and C. Ross. FACET: Fairness in computer vision evaluation benchmark. wiXiv:230.000033 2023.
K. He, Y. Lu, and S. Sclaroff. Local descriptors optimized for average precision. Computer Vision and Pattern Recognition, 2018.
H. Jégou and O. Chum. Negative evidences and co-occurences in image retrieval: The benefit of PCA and whitening. European Conference on Computer Vision, 2012.
X. Ji, J. F. Henriques, and A. Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. International Conference on Computer Vision, 2019.
C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. Inter- national Conference on Machine Learning, 2021.
J. Kleinberg. An impossibility theorem for clustering. Neural Information Processing Systems, 2002.
S. Kwon, J. Y. Choi, and E. K. Ryu. Rotation and translation invariant representation learning with implicit neural representations. International Conference on Machine Learning, 2023.
S. Lee, S. Lee, H. Seong, and E. Kim. Revisiting self-similarity: Structural embedding for image retrieval. Conference on Computer Vision and Pattern Recognition, 2023.
B. Li, Y. Zhang, I.. Chen, J. Wang, J. Yang, and Z. Liu. Otter: multi-modal model with in-context instruction tuning. arXiv:2305.0372 03726, 2023a.
Li, D. Li, S. Savarese, and S. Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv:2301.12597, 2023b.
L. Li, P. Zhang, H. Zhang, Yang, C. Li, Zhong, L.. Wang, I.. Yuan, L.. Zhang, J.-N. Hwang, K.-W. Chang, and J. Gao. Grounded language- image pre-training. Computer Vision and Pattern Recognition, 2022.
P. Li, H. Zhao, and H. Liu. Deep fair clustering for visual learning. Computer Vision and Pattern Recognition, 2020.
Y. Li. P. Hu, D. Peng. J. Lv, J. Fan, and X. Peng. Image clustering with external guidance. 17X.1:2310.11989, 2023c.
H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Systems, 2023.
Neural Information Processing
12

---

Published as a conference paper at ICLR 2024
Z. Liu, C. Seddriuuz--Oazo,, D. Teney, and S. Gould. Image retrieval on real-life images with pre-trained vision-and-le and-language models. International Conference on Computer Vision, 2021.
F. Long, T. Yao, Z. Qiu, L. Li, and T. Mei. PointClustering: Unsupervised point cloud pre-training using transformation invariance in clustering. Computer Vision and Pattern Recognition, 2023.
A. D. McCarthy, T. Chen, and S. Ebner. An exact no free lunch theorem for community detection. International Conference Complex Networks and Their Applications, 2020.
S. Menon and C. Vondrick. Visual classification via description from large language models. Inter- national Conference on Learning Representaions, 2023.
I. M. Metaxas, G. Tzimiropoulos, and I. Patras. DivClust: Controlling diversity in deep clustering. Computer Vision and Pattern Recognition, 2023.
1. Misra and L. v. d. Maaten. Self-supervised learning pretext-invariant represstttttsss..nnnnnn Computer Vision and Puttern Recognition, 2020.
R. Mokady, A. Hertz, and A. H. Bermano. 1.X0442/1,.09334, 2021.
ClipCap: Clip prefix for image captioning.
C. Niu and G. Wang. SPICE: Semantic pseudo-labeling for image clustering, IEEE Transactions on Image Processing, 31:7264-7278. 2021.
H. Noh, A. Araujo. J. Sim, T. Weyand, and B. Han. Large-scale image retrieval with attentive deep local features. International Conference on Computer Vision, 2017.
OpenAl. GPT-4 technical report. arXiv:2303.08774 2023.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray. J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, F. Christiano, J. Leike, and Lowe. Training language models to follow instructions with human feedback. Neural Information Processing Systems, 2022.
S. Park, S. Han, S. Kim, D. Kim, S. Park, S. Hong, and M. Cha. Improving unsupervised image clustering with robust learning. Computer Vision and Pattern Recognition, 2021.
J. Platt, M. Czerwinski, and B. Field. Phototoc: automatic clustering for browsing personal pho- tographs. International Conference on Information, Communications and Signal Processing, 2003.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. International Conference Machine Learning, 2021.
7. Ren, Y. Su, and X. Liu. ChatGPT-powend hierarchical comparisons for image classification. Neural Information Processing Systems, 2023.
J. Revaud, C. De Souza, M. Humenberger, and P. Weinzaepfel. R2d2: Reliable and repeatable detector and descriptor. Neural Information Processing Systems, 2019.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. resh-resolution image synthesis with latent diffusion models. Computer Vision and Pattern Recognition, 2022.
B. C. Russell, T. Antonio, M. P. and W. T. Freeman. Labelme: A database and web-based tool for image annotation. International Journal of Computer Vision, 77:157-173, 2008.
V. Sanh, A. Webson, C. Raffel, S. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, Kim, G. Chhablani, N. Nayak, D. Datta, J. Chang, M. T.-J. Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Bawden, T. Wang, T. Neeraj, J. Rozen, A. Sharma, A. Santilli, T. Fevry, J. A. Fries, R. Techan, T. L.. Scao, S. Biderman, L. Gao, T. Wolf. and A. M. Rush. Multitask prompted training enables zero-shot task generalization. International Conference on Learning Represen- fations, 2022.
13

---

Published as a conference paper at ICLR 2024
L. Schmarje, M. Santarossa, S.-M. Schröder, C. Zelenka, R. Kiko, J. Stracke, N. Volkmann, and R. Koch. A data-centric approach for improving ambiguous labels with combined semi- supervised classification and clustering. European Conference Computer Vision, 2022.
Y. Shen, Z. Shen, M. Wang, J. Qin, Torr, and L. Shao. You never cluster alone. Neural Information Processing Systems, 2021.
O. Simeoni, Y. Avrithis, and O. Chum. Local features and visual words emerge in activations. Conference on Computer Vision and Pattern Recognition, 2019.
Y. Tian, Newsam, and K. Boakye. Fashion image retrieval with text feedback by additive attention compositional learning. Conference Applications of Computer Vision, 2023.
G. Tolias, R. Sicre, and H. Jégou. Particular object retrieval with integral max- mux-pooling of cmn activations. arXiv: 1511.08879 2015.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv:2302.13977 2023a.
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhar- gava, S. Bhosale, D. Bikel, Blecher, C. Ferrer, M. Chen, G. Cucurull, Esiobu, Fernan- des, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, Kloumann, A. Korenev, Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.088 092888, 2023b.
W. Van Gansbeke, S. Vandenhende, S. Georgoulis, M. Proesmans, and Van Gool. Scan: Learning to classify images without labels. European Conference on Computer Vision, 2020.
D. P. Vassileios Balntas, Edgar Riba and K. Mikolajezyk. Learning local feature descriptors with triplets and shallow convolutional neural networks. Proceedings of the British Machine Vision Conference (BMVC), 2016.
V. Viswanathan, K. Gashteovski, C. Lawrence, T. Wu, and G. Neubig. Large language models enable few-shot clustering. arXiv:2307.00524 2023.
N. Vo, L. Jiang, C. Sun, K. Murphy, L.-J. Li, L. Fei-Fei, and J. Hays. Composing text and image for image retrieval- an empirical odyssey. Computer Vision and Pattern Recognition, 2019.
U. von Luxburg, R. Williamson, and Guyon. Clustering: Science art? International Confer- ence on Machine Learning Workshop on Unsupervised and Transfer Learning, 2012.
J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, and Y. Gong. Lonstity-c linear coding for image classification. Computer Vision and Pattern Recognition, 2010.
J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. International Conference Learning Representations, 2022.
D. Wolpert and W. Macready, No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1):67-82, 1997.
H. Wu, M. Wang, W. Zhou, Z. Lu, and H. Li. Asymmetric feature fusion for image retrieval. Computer Vision and Pattern Recognition, 2023.
Y. Wu, Q. Tian, and T. Huang. Discriminant-em algorithm with application to image retrieval. Computer Vision and Pattern Recognition, 2000.
B. Yao and L. Fei-Fei. Grouplet: A structured image representation for recognizing human and object interactions. Computer Vision and Pattern Recognition, 2010.
14

---

Published as a conference paper at ICLR 2024
B. Yao, X. Jiang, A. Khosla, A. L. Lin, L. J. Guibas, and L. Fei-Fei. Human action recognition by learning bases of action attributes and parts. International Conference on Computer Vision, 2011.
K. M. Yi, E. Trulls, V. Lepetit, and P. Fua. Lift: Learned invariant feature transform. European Conference on Computer Vision, pages 467-483, 2016.
Yunfan, Mouxing, P. Dezhong, Taihao, H. Jiantao, and Xi. Twin contrastive learning for online clustering, International Journal of Computer Vision, 130, 2022.
P. Zeng, Y. Li, P. Hu, D. Peng, J. Lv, and X. Peng. Deep fair clustering via maximizing and minimiz- ing mutual information: Theory, algorithm and metric. Computer Vision and Pattern Recognition, 2023.
H. Zhang, P. Zhang, X. Hu, Y-C.. Chen, L. H. Li, X. Dai, L. Wang, L. Yuan, J.-N. Hwang, and J. Gao. GLIPv2: Unifying localization and vision-language understanding. 05X0::2006008336, 2022.
Zhang, Wang, L. Zhou, and P. Koniusz. Learning spatial-context-aaar global visual feature representation for instance image retrieval. International Conference on Computer Vision, 2023.
H. Zhong, J. Wu, C. Chen, J. Huang, M. Deng, L. Nie, Z. Lin, and X. Hua. Graph contrastive clustering. International Conference on Computer Vision, 2021.
B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning deep features for scene recog- nition using places database. Advances in Neural Information Processing Systems, 2014.
D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. MiniGPT-4, Enhancing vision-language under- standing with advanced large language models. arXiv:2304. 10592, 2023.
15

---

Published as a conference paper at ICLR 2024
A ABLATION STUDIES
A.1 ABLATION STUDY OF VISION-LANGUAGE MODELS
We fix the LLM to GPT-4 and perform ablation study on the choice of vision-language model (VLM), specifically ClipCap (Mokady et al., 202 1), Blip-2 (Li et al., 2023b), and LLaVA (Liu et al., 2023), during Step 1. ClipCap is unable to take in text prompt and therefore the text criterion is not reflected in the resulting caption. Blip-2 and LLaVA can perform instructed zero-shot image-to- text generation and demonstrate the ability interpret natural language instructions within visual contexts, producing responses that suggest thorough image analyses in our setting. This capability allows them to be used effectively within IC/TC, and expect that any recent VLMs likewise be utilized. Results are presented in Tables 3, 4, and 5.
Ablation study of VLMs in CIFAR-IO, STL-10 and CIFAR-100 datasets when using clus- Table 3:
tering criterion as Object.
CIFAR-I0
STL-10
CIFAR-100
Method
1CC
NMI f
ARI f
ACC f
NMI 1
ARI f
ACC f
NMI 1
ARI 1
ClipCap 0.636
0.605
0.524
0.722
0.729
0.647
0.365
0.396
0.214
BLIP-2
0.975
0.941
0.947
0.993
0.982
0.985
0.584
0.690
0.429
LLaVA
0.910
0.823
0.815
0.986
0.966
0.970
0.589
0.642
0.422
Table 4: Ablation study of VLMs with Stanford 40 Actions dataset and clustering criteria Action, Location and Mood. N/W (Not Working) means: In Step 3, LLM responds that there is no appropriate label assign the description obtained in Step 1. Accuracies labeled * evaluated by having a human provide ground truth labels on 1000 randomly sampled images.
Action
Location
Mood
Method
ACC 1
NMI
ARI 1
ACC
NMI
ARI 1
ACC f
NMI t
ARI t
ClipCap
0.250
0.374
0.086
0.293*
0.282*
0.167*
0.377
0.098
0,057*
BLIP-2
0.427
0.621
0.335
0,483
0.415*
0.319*
0.286*
0.009
0.005
LLaVA
0.774
0.848
0.718
0.822
0.695*
0.669*
0.793*
0.512
0.525*
Table 5: Ablation study of VLMs in People Playing Musical Instrument (PPMI) dataset with using clustering criteria as Inst rument t. and varying granularity.
Inst rument, K=7
Instrument, K=2
Method
ACC +
NMI
ARI
ACC
NMI
ARI f
ClipCap
0.318
0.164
0.114
0.642
0.049
0.078
BLIP-2
0.840
0.908
0.816
1.000
1.000
1.000
LLaVA
0.964
0.928
0.920
0.977
0.910
0.841
A.2 ABLATION STUDY OF LARGE LANGUAGE MODELS
On CIFAR-1O, STL-10, and CIFAR-100, we fix the vision-language model (VLM) to LLaVA and perform an ablation study on the choice of large language model (LLM). We use GPT-4 in step 2a for stable results and Llama-2 of various sizes (Touvron et al., 2023b) in all other steps to the downstream task performance." For ablation purposes, we have kept the text prompts for all three steps the same as those used for experiments utilizing GPT-3.5 and GPT-4 (OpenAI, 2023). The performance tended to improve as the number of parameters increased, though the gain was not significant (Figure 3).
3 After the initial submission of the manuscript, found setting that achieves comparable performance while using only Llama-2, including in step 2a. We provide the detailed setting and results in GitHub repository.
16

---

Published as a conference paper at ICLR 2024
For GPT-3.5 and GPT-4, which had the best performances, conducted additional comparisons across all datasets (Table 6). To clarify, LLaVA + GPT-3.5 indicates the usage of GPT-3.5 for Steps and 3. In particular, for the Stanford 40 actions dataset, the raw labels tend to be lengthy descriptions of human actions and hence Step 2 cannot be performed using GPT-3.5 due to its token limits. The performance gap between LLaVA GPT-3.5 and LLaVA GPT-4 is marginal for datasets with relatively small number of classes. However, the gap widens for a more complex dataset, such as CIFAR-100 and Stanford 40 Action.
Table 6: Experiment Results comparing the performance of IC|TC using GPT-3.5 and GPT-
CHARIIO
STL-10
CBAR-100
Stavorr Artsen
1V50 coooe
FIMI cllssm
Method
L.AV GPE33
8964 0M8O
0303
0.831
0.977
11917
0.315
0A13
0:304
0.724
10256
06-66
1.965
0:225
0913
0.713
1.766
LLVVA
0315
0.996
(1996
1.389
0,422
0.422
0.548
07718
IL964
8:928
0.921
0,977
0.842
0.911
A.3 THE NECESSITY OF STEP AND STEP 3
IC/TC uses the vasion-language model (VLM) to extract salient features from images needed for clustering. Based on this information, large language model (LLM) carries out the clustering. There- fore, all the information needed for conducting clustering theoretically exists in the VLM. Then, is the LLM truly necessary? And do we really need Steps 2 and 3? We answer this question and experimentally show that the L.M and Steps 2 and 3 are essential components of our method.
A.3.1 K-MEANS CLUSTERING ON THE EMBEDDING SPACE OF LLAVA
We conduct K-means clustering on the embedding space of the vision-language model (VLM). In this experiment, we employ LLaVA for the VLM. To clarify, an LLM is not used in this approach. The VLM tokenizes both the input image and text using the pre-trained encoder and producing a sequence of tokens. These tokens are subsequently processed by a language model decoder (within the VLM). As they traverse each transformer layer within the decoder, hidden states are generated. We use the final hidden states from this process. There are two primary options for utilizing these last hidden states: 1. (Mean) We obtain the embedding vector by using the mean-pooling of the final hidden states of all input tokens. This offers a representation of the entire input sequence. 2. (Final) We obtain the embedding vector by using the hidden state vector of the final token. It often encapsulates a cumulative representation the entire input sequence, making it useful for tasks such next-token prediction. In both cases, the performance of K-means clustering the embedding space of LLaVA was notably worse compared to IC|TC
Table 7: K- K-means clustering on the embedding space of Stanford 40 Actions dataset using LLaVA. Accuracies labeled are evaluated by having a human provide ground truth labels on 1000 randomly sampled images.
Action
Lecation
Mood
Method
ACC t
NMI f
ARI
ACC
NMI f
ART
ACC
NMI f
ART
I.LaVA (Final)
0.256
0.356
0.140
0.338
0.319
0.178
0.418
0.385*
0,241
LLaVA (Mean)
0.498
0.588
0.356
0.405
0.377
0.230
0.486*
0.409*
0.292
SCAN
0.397
0.467
0.272
0.359*
0.353
0.206
0.250*
ICJTC
0.774
0.848
0.718
0.822
0.695
0.669*
0.793
0.512
0.525
A.3.2
LLAVA ONLY
With LLaVA's remarkable image-to-language instruction-following capability, we can also prompt LLaVA to directly predict the label of the provided image, along with the TC provided by the user. The text prompt we used is presented in Table 8. Using the predicted labels for each image, evaluate the clustering performance with the Adjusted Rand Index (ARI) and Normalized Mutual Information (NMD. Note that even if the number of distinct predicted labels differs from the ground truth labels, both ART and NMI remain applicable. In the Stanford 40 Actions dataset, where
17

---

Published as a conference paper at ICLR 2024
the clustering criterion Actionn the number of distinct predicted labels was overwhelmingly greater than that of the ground truth, rendering the performance evaluation meaningless. For both the CIFAR-I0 10 and STL-10 datasets, when the clustering criterion set to Ob ject, achieved reasonable performance. This was not the case for CIFAR-100. Nonetheless, the performance was still lower than that achieved with the full pipeline of IC|TC. Results presented in Table 2.
Table 8: Prompts used for LLaVA only experiments; clustering based on Ob Object.
Dataset
Step
Text Prompts
CIFAR-10. STL-10
Step
Provide one-word label main object this image. in the following format:
CIFAR-100
"Answer: fenswe=")
A.4 CAN WE SKIP STEP 2A?
Our method discovers the clusters' names Step and Step 2b. However, one might question the necessity of Step 2a, which involves obtaining the raw initial label. In this section, we conducted an experiment in which we tasked the Vision Language Model (VLM) with direct labeling. So, the pipeline for this experiment is Step - Step 2b + Step 3. We utilized LLaVA and the Stanford 40 Actions dataset with Action as the clustering criterion and GPT-4 for Large Language Model (LLM). Both Step 2b and Step utilized the same text prompt described in Table 12.
The specific prompt provided to the VLM for skipping Step 2a was:
P Psepp "What is the main action of the person? Answer in words."
However, the output from LLaVA varied greatly and often contained excessive information. Because the variety of the labels was great, the number of tokens exceeded the maximum limit of 32k for GPT-4. We therefore concluded that Step 2a was essential.
A.5 DO WE REALLY NEED CLUSTERING CRITERION IN STEP AND STEP
To determine whether the criterion is truly beneficial for Step 2 and Step we kept Step I unchanged and removed the text criterion TC from Steps 2 and 3. We used the Stanford 40 Actions dataset and employed Action on the clustering criterion. The specific prompt provided to this experiment is:
Table 9: Removing criterion from Step Step 3
Action
Method
ACC
NMI
ARI
Removing criterion from Step & Step (Appendix A.5)
0.152
0.181
0.063
Full pipeline
0.774
0.848
0.718
The results revealed that most of the cluster names discovered in step 2 had little to relevance Action, was challenging to identify any consistent criteria among these cluster names. Con- sequently, the clustering executed in Step 3 deviated significantly from the ground truth defined by Action, resulting in diminished performance shown in Table 9.
A.6 Do WE REALLY NEED THE FULL DESCRIPTION AS INPUT IN STEP
Step 3 involves the Large Language Model (LLM) assigning the image's text representation to the appropriate cluster. We use the image description generated by the Vision Language Model (VLM) in Step 1, but one may wonder whether the shorter raw label output by Step 2a be used instead to reduce the computation cost.
We find that the alternative of providing the output of Step 2a the input Step 3 has poor performance, and we illustrate why this is the case through the example presented in Figure 6. In this image, there is girl waving her hand in playground, and the text criterion Act on.
18

---

Published as a conference paper at ICLR 2024
Description
The image features young girl sitting green chair, which is part playground. The girl is smiling and waving, indicating that she happy and enjoying her time at the playground. The playground is filled with various green structures, including slide and swing, which provide fun and engaging environment for children play and interact with one another. The girl's at- tire consists of a white dress, which adds touch of innocence and charm.
Raw label
Playing at playground
Ground truth
Waving hands
Assigned cluster
Waving
Figure 6: Example illustrating why the cluster assignment of Step requires the full description of the image.
As shown in Figure 6, the output of Step 1 contains all information related to "playing" and "wav- ing," which expected due the verbose nature of VLMs. However, the output of Step 2a, the raw label, only captures "'playing." Now, suppose that after Step 2b, where the cluster names are obtained from the raw labels, there cluster name relating to 'waving' but none directly related to "playing'. Then, it is necessary for the LLM to be provided with the full textual description, not just the raw label, to properly assign the image to the cluster 'waving".
19

---

Published as a conference paper at ICLR 2024
B EXPERIMENTAL DETAILS AND SAMPLES
B. DATASETS DETAILS
In table 22, we listed the information of the datasets we used for all experiments.
We used Stable Diffusion XL (Rombach al., 2022) generate all images. We used the following text prompt: "A photo of a (n) [class name] without any negative prompt. We used default parameters for Stable Diffusion. We generated the images in 1024 x 1024 resolution, and resized them to 32 x 32 before use.
CIFAR-10-Gen
Table 10: Datasets overview
Dataset
Critetion
of data
Classes
Names of classes
Act ioa
9,532
blowing bubbles, reading, phoning, playing violin, brushing tecth, drinking. running, playing guitar; riding borse, laking photos people taking photo. jumping. looking through a micrescope, shooking an arrow. watching TV, playing baskeiball, waving hands lesting messge, Unowing fristy, using a conguter. cooking, shaving beard. cuiting hugging. smoking. playing happ. directing traffic. looking photos, walking the dog. playing vella, applying cream. writing on a book puper. holding an umbrella, feeding honse fishing. riding a bike, gandening, lixing bike cleaning the floor, doing loundyy
firewood, pushing
Stanlford 10 Action
Location
Area, public event gathering. sports facility, envirun- meni, educational institution urium area or city streel, resiaurant, workplace. tramppetation bulb, store or market
Rood
9,532
joyfal, focused, adventurous, relaxed
Mosical
Inat rument
saxcphoone, guitar, tramper violin, cello, flute, horp
Mosica! Inst rument
700
brass instraments, string instruments
CI
Object
10,000
airplane, automobile,, bird. deer, frog. borse, truck
STL-10
object
K0000
airplanc, bird, car, eat, deer dog. horse, monkey, ship, truck
CIFAR-100
Object
10,000
20"
aquatic mammals, fish. flowers food containers, fruit and vegetables, house- hold electriaal devices, houschold, fumiture, scenes, large oonivores and herhivares, medium sized mammals, moneinsect invertebrates, people, rep- tiles, small mammals, trees, vehicles I. vehicles 2
large camivores, large
man-made outdoor things, large natual,
CIFAR-10-Gen
Object
1,000
airplane, automobile,, hird,
fiog, mick
B.2 MODEL DETAILS
IC/TC crucially relies on the usage of VLMs and LLMs that follow human instructions. Although using foundation models naïvely suffices for simple criteria, the performance diminishes rapidly for more complex tasks. Hence is crucial that both the VLM and the LLM adhere the human instruction, including TC in our case.
The instruction following abilities of GPT-3.5 and GPT-4 models are well established. Furthermore, LLaVA has been trained extensively on language-image instruction following dataset. Finally, we adhere to the Llama-2 models that have been tuned with instruction datasets. We include the full model versions of the VLMs and LLMs we have used in our experiments.
Table 11: Model versions for the VLMs and LLMs
Model
Version
Blip-2
blip2 flan t5 xxl
LLaVA
1lava-vl-00 -oo.---oo----eeeeeeeeeeee
GPT-3.5-16k-tubby
api-vers oowoooo--000000 111--02233--115--eeeee-
GPT-4, GPT-4-32k
api-vereicn-22233 -0--222022--------- 03-15--revie
Llama-2-7b
meta 1lama/Llama 7b chat hf
Llama-2-13b
meta 1lama/Llama 13b chat hf
Llama-2-70h
met.a allaammmllllaa. ama-22ma222 20-a----0-----0 705---htthhh
Remark. The CIFAR-100 dataset has 100 classes, but also has 20-superclass labels (and hence is sometimes referred to as CIFAR-100- 100-20.. Since the usage of 10FAR-100-20 dataset is more common in the clustering literature, we also use the 20-superclass labels for experiments.
20

---

Published as a conference paper at ICLR 2024
B.3
PROMPT DETAILS
B.3.1 GUIDELINES FOR TEXT PROMPTS
Text Criteria (TC) for all three steps. We emphasize here that is important to provide the user- specified text criteria throughout all three stages. For example in the Stanford 40 Actions dataset, crucial in Step to obtain descriptive answers, from the VLM model, that analyze the main action of the person in the image. Utilizing general prompt such as "Describe the image" results in the LLaVA model producing text descriptions that only roughly describes various aspects of the image, hence failing to capture the details of the human action required for clustering such granular dataset. well-expressed set of text criteria is required to retrieve meaningful class labels and be able to classify the images into such granular classes.
Steps 1 and 3. For Step we followed the LLaVA instructions to retrieve detailed image de- scriptions as described Appendix of Liu et al. (2023)S. For low-resolution datasets such CIFAR-10 and CIFAR-100, prepending the prompt "Provide ..." or using instructions for brief image descriptions were, although marginal, helpful. Fine- tuning the text prompt used in Step can be helpful when the output from Step 2 is unsatisfac- tory or noisy. In such cases, it was beneficial to append specific prompts. For example, if the clustered classes had two or more classes that were strict superclaseessuuclasses of each other, we appended: "Be as specific as possible to choose the closest ob ject from the given list". Generally speaking, we found Steps cific text prompts, while for Step 2, it was much more important to finetune the text prompt carefully.
brief description
less sensitive to the spe-
Step 2.
When evaluating the clustering results with metrics such as ACC, NMI and ARI was possible (i.e., when the dataset has ground truth labels), we discovered that the outputs from Step have the most influence on the final evaluation.
Here are the two major cases where the user may wish to tune their text prompts (and text criteria TC) for Step 2:
1. The user wishes to enforce certain levels of gramularity in the clustered classes
2. The clustered classes are not optimal: i.c., includes duplicates, super/subelasses. classes that are too broad such as "object", etc.
For the first case, it is crucial to provide text prompts instructing the LLM to split or merge classes certain level in the hierarchy. For example, the user may wish split the class "animals" but such a broad class can be split up according to multiple criteria, such as habitat, feed, species, etc. Hence it is crucial to provide an appropriate TC. Alternatively, the user may wish to merge certain classes together, such as in our PPMI experiment with K = 2 and criterion based on Mus i cal Compared to the case when K = 7, by enforeing K = 2, we expect the algorithm to discover superclasses that can encompass the original classes. While the full prompt can be found in Table 15, in short, the addition of the following prompt was important:
Instruments
when categorizing classes, consider the following criteria:
1. Each cluster should have roughly the same number of images. Merge clusters with similar meanings with superclass.
Finally, after turning the raw labels into dictionary, we tried filtering out less frequent raw labels, where the threshold value was considered as hyperparameter. Since the evaluation is expensive (it requires running the entire Step 3 again), we did not measure the final classification results. However, after inspecting the clustered classes (checking for duplicates, super/subclasses, etc.), we concluded that using threshold values such as or was helpful in getting a better set of clustered classes.
Providing raw labels: dictionary full list. Step 2-2 requires feeding the entire list of raw labels to the LLM. While our algorithm converts the list of raw labels into a dictionary, we also tried feeding the entire list of labels the L.M.
21

---

Published as a conference paper at ICLR 2024
In order to instruct the LLMs to perform clustering tasks, need to provide the set of raw labels and their number occurrences. Empirically, we found out that feeding the entire list of raw labels yielded higher metrics, which could mean that the LLM understands the those information better when provided with the full list. However, this approach quickly goes out of scale with a larger dataset, due the token limits of the LLMs. Hence, have used dictionaries throughout experiment.
When the raw labels were noisy (i.e., long-tail of labels with few occurrences), the class labels were lengthy (e.g. in Stanford 40 actions dataset), we have empirically found out that the LLM sometimes failed to understand the dictionary or hallucinates. In such cases, we have empirically found out that providing an additional explanation prompt of the dictionary was helpful.
For example, if the input is given as "I"a': 15, 'b': 25, 'c' 17)", it means that the label 'a', 'b', and 'c' appeared 15, 25, 17 times in the data, respectively.
B.3.2 TEXT PROMPT EXAMPLES
Below are tables of the prompts that yielded the best results in our experiments, for every experiment we conducted. We have used the exact same prompt after replacing the placeholders such [_LLE___,, WWM..CCAASEESCIISSTTEE particular, [_CCAASES refers to the list of K clusters that our algorithm discovers (output of Step 2).
and [_CLASSES__] appropriately. In
B.4 SAMPLE OUTPUTS
Here we display some sample images from Stanford 40 Action and CIFAR-100 dataset, and include the outputs for each stage where the clustering criteria were action and ob ject, respectively.
B.4.1 STANFORD 40 ACTION
Sample Images and Outputs for all stages.
Output of Step 2a.
clapping
taking a picture: celebrating a goal: posing a photo: 28,
giving speech: 31, celebrating: 17, applauding: 6, waving to the arowd: 6, waving to a
crowd: waving hello: 10, clappings 6, performing on slages 9, giving a preseatation:
11, dancing: 13, posing for picture: 7, playing guitars: 7, taking selfie: 41, having
conversation: smoking a cigarecte: 207, waving: 27, playing a guitar: 226, pasing
for photograph: 9, throwing friapee: 62, alowing bubbles: 210, eating Iellipops 12,
owing a bubble: 17, bruaning teeth: 92, brushing her teeth: 50, brushing his teeth: 22,
drinking coffee: 25, using computes:
drinking frem cup: 15, cleaning the floor:
52, vacuuming the floor: 30, sweeping the floor: 87, mepping the floor: 6, cleaning the kitchen: 13, climbing rock wall: rock climbing: 116, waving at the camera: woving
his hand: 10, writing on a chalkheard: 38, teaching a leason: 6, teaching: 7, writing on
blackbnard: 31, writing
whiteboard: 13, writing: 13, studying doing hamework: 15, writing on a piece of paper: 11, doing homeworks 18, writing or drawing: 14, drawing or
writing: 6, writing taking notes:
Output of Step 2b.
(slapping, taking picture, colebrating, giving speech, saving, performing stage, playing guitar, taking selfie, smoking cigazetle, throwing friabee, blowing buables, brushing teeth, drinking coffee, using computer, cleaning floor, climbing, cooking, preparing food, cutting down cree, gardening, drinking beverage, reading book, using sell phene or laptop, interacting with horse, fishing, zepairing bicycle, repaizing car, walking
Only be added when K
22

---

Published as a conference paper at ICLR 2024
rain with umbrella, jumping, examining microscope, abserving through telescope, talking on phone, playing violin, pouring drink, pushing cart or strollet, riding bicyele or horse, studying oz teashing, running jegging, practicing archery, washing dishes cleaning sink]
B.4.2 CIFAR-100
Sample Images and Outputs for all stages.
Table 19: Sample outputs for CIFAR-I00 data
Images
Step Output
The main object the image is wccden table with round top.
The main object the image is Large alligator laying on the ground in grassy area.
Step 2a Ourput
Table
Crocodile
Step Oulpul
Furniture
Replie
Ground Truth
Household furniture
Reptiles
Output of Step 2a.
bear: 198, rock: 54, squitrel: 107, person: 61, beaver: 24, duck: 7, animsl: 52, seal:
82, monkey: 47, cet: 85, ret: 15, fox: 80, gerilla: 25, rabbit: 99, dog: 188, bowl:
face: 51, fishs 229, dalphin: 68, shark: 100, pola: 11, whale: 86, palm treet 79, bird: 79, polar bears
111, mouse: 97, sheea: 10, deor: 30, elephent: 97, papert 7, apple:
hand: 15,
snake: 115, aisplane:
dinosauzs
otters 13, scuiptuze:
8, raccoon: 85, groundhog: 21, turtle: 87, toot: 6, cloud: 52, scees 462, nan: 107,
alligator:
29, kangaroc: 72, statue:
31, chair: 146,
73,
codent: 16, Noman: 105, frog: 9, flower: 252, arrew: 6, caterpillar: 52, plate: 47,
ball: stingray: 19, lighthouse:
é, 91, train:
church: 9, read:
85, line: 7, bicyale: 96, sunset: 25, sunt 8, water: 14, trees: 12, forest: 8, grass:
20, beach: 12, ocsan: 7, camal: 76, chimpanzea: 27, matorcycles 94, pricaratops: 7,
hedgehog: toy: 18, epossum: 17, skunk: hamster: 67, lobs:er: 13, spider web:
baby: 85, child: 12, girl: 82, crocociler 11, Lank: 85, scooter: 8, bus: 81, van: 26,
bulldozer: lawnmower: 39, lawn
22, trolley: 33, streazeor: 9, excavator: 19,
Output of Step 2b.
tanimal, bird, fiah, nannal, reptile, insect, plant, flower, fruit, vehicle, fusniture, building, electronic device, kicchen utenail, clothing item, Loy, musical instrument, sporte equipment, natural landscape, human]
23

---

Published as a conference paper at ICLR 2024
B.5 CONFUSION MATRICES
Toppet
(a) CIFAR-10 (Ob ject)
(b) STL-10 (Object)
(c) PPMI (M. I.)
Figure 7: 10 STL-10, PPMI confusion matrices.
household electrical devices
75
trees
97
vehicles
93
non-insect invertebrates
24
fruit_and_ sa__aaddvveeee
vegetables
71
aquatic_mammen
flowers
medium mamimulmmmmmalss
large_camivores
Insects
people
95
vehicles 2
reptiles
large_mas htt.mmmnnadd__uueeeee
www.ootodoo.... things
74
food_containers
75
large_ gnge_oomivores_com and herbivores _n__herbivores
muusehold_furnitun
93
fish
63
household electrical devices
non-insect Invertebran inverebbrates fruit and vegetaie vegetables aquatic mammals flowers
vehicles 1
trees
large natural outdoor scenes
medium, mammals
large carnivores
small mammals large, man- man--made outdoor things food containers large omnivores and herbivore rrbiivores househald fumiture
insects
people vehicles 2
reptiles
fish
Figure 8: CIFAR-100 (Objects confusion matrix.
24

---

Published as a conference paper at ICLR 2024
using computer
looking through teleccop
Blowing buselis
riding bice
playing vioti
texting message
83
throwing reayy
watthing
gardening
fiskig
taking phous
feeding horee
cutting mees
Cuiknn qutt
ridng a horse
waehing
driking
B5
pushing a cart
win ouno sn book
witting
writigg board
counne kuild
holding so umbroll
pooting
fixing bike
vegatables
waving
hands
welten the
94
rowing bost
916
en shoolan un arow
flor
paishing
Matt and
vo thu Sont
sypae Pur
Bog th0 Seuiita
ureis Poottrii
tho Eisiou
du recoust Surpeny
Figure 9: Stanford 40 Actions (Act ion) confusion matrix.
B.6 CLUSTERING EXAMPLES
Figure 10: CIFAR-10 10: The number of clusters K = 10. Clustering based on Ob ject.
25

---

Published as a conference paper at ICLR 2024
Figure 11: STL-10; The number of clusters A = 10. Clustering based on Ob Object.
O 8
Figure 12: CIFAR-100; The number of clusters K = 20. Clustering based on Ob Ob ject..
26

---

Published as a conference paper at ICLR 2024
Table 12: Text Prompts for 40-class Action based clustering; Stanford 40 Action
Steps
Prompt
Step
Characterize the image using bell-detailed description.
Describe the person' main action in words.
Step 2a
You will be given a description of image of person performing an action. Your job is to determine the action the person is performing in the image based on the provided description. Please respond in the following format: "Answer: action". For example, given the following description:
In the image, two young women are riding camels in the desert. They are sitting on the camela, which are carrying them across the sandy terrain. The women are wearing shorts and sandals, and they appear to be enjoying their ride. The camels are walking in the desert, and the background features sandy landscape with some vegetation. This scene captures moment of advencure and exploration in the desert, the women experience the unique and exotic environment the back of these animals. .un
Then an exemplar answer would be "Answer: Riding camel".
Step 2b
You will be provided list of [_LEN_] human actions and the number occurrences in given dataset. Your job is cluster
[__EE_]] words into [_NUM_CASSS3 CLUSTER actions. Provide words, each word your answez as a list [_NNM_CIASSBS.CIUSTTT representing a human action.
For example, If the input is given as "('a': 15, bb': 25, 'c': 177", it means that the label 'a', "b', and 'e' appeared 15, 25, 17 times in the data, respectively.
When categorizing classes, consider the following criteria:
1. Each cluster should have roughly the same number of images.
2. Each cluster should not have mulciple classes of different actions.
Now you will be given list of human actions and the number of classes, and the lisz of classes you answered previcusly.
Please output list of human actions length L_NUNCCLASES.CISSTRR.. in the following format: "findex]: (actions)". Make sure that you strictly follow the length condition, which means that [index) must range from to [_NUM_CLASSES.CIISTT..
Step Your job is classify an action person in an image is per forming. Based on image description. deLermine the most appropriate human action category that best classifies main action the image. You must choose from the following options: (_CCLASSES..).
Gi your answer in the fol following format: "Answer: { Be as specific as possible to choose the closest action [rom the given list. If sicuation arises where nothing is allocazed, please assign it to the action that has the closest resemblance.
27

---

Published as a conference paper at ICLR 2024
Table 13: Text Prompts for 4-class Mood based clustering: Stanford 40 Action
Steps
Prompt
Step 1
Describe the mood of the image.
Step 2a
You will be géven description of the mood. Your job is determine the mood based on the provided description. please respond in the following format: "Answer: {mocd)". For example, given the following description:
In the image, two young women are riding camels in the desert. They are sitting on the camels, which are carrying them across the sandy terrain. The women are wearing shorts and sandals, and they appear to be enjoying their ride. The camels are walking in the desert, and This
the background features sandy landscape with some vegetation, scene captures a moment of adventure and expleration in the desert, as the women exper the unique and exotic envi ronment on the back of these animals. .*.
exemplar
would
Enjoying"
Step 2b
You will be provided a list of [_IEN_] moods and the number of occurrences in a given dataset. Your job is to cluster [__EN_.] words into (_NUM_CLAESES, CLUSTER_] _NN..CIASSSSS_CIISTTRRR_ categories. Provide your answer as a 11st of NUM CLASSES CLUSTER words, each word represent ing the mood.
example, if input given as "f'a's 15, 'b' 25, ee' 17)", means that lasel 'a', 'b', fer appeared 15, 25, times in the data, respectively.
When categorizing classes, consider the fol lowing criteria:
1. Bach cluster should have roughly the same number of images.
2.
Merge clusters with similar meanings.
3. Bach cluster should not have multiple classes of different moods.
4. Bach cluster represents general mood and should not be toc specific.
Now you will be given list of locations and the number of classes, and the list of classes you answered previcusly.
Flease ouzput list musical instruments of length
in the following format: "[index): (mood]". Make sure that you strictly follow the length condition, which means that [index) must range
to [_NDMLCIASSTS_CIIS)
Step 3 Your job is to classify an object in the image. Based on the image description, determine the most appropriate category that best classifies the main object in the image. You must choose from the fol lowinwin options: [_CLASERS_].
Give your answer in the following format: "Answer: [object]", If situation arises where nothing allocated, please assign to the objecz that has the closest resemblance.
28

---

Published as a conference paper at ICLR 2024
Table 14: Text Prompts for 10/2-class Location based clustering: Stanford 40 Action, PPMI
Steps
Prompt
Step 1
Describe where the person is locaced.
Step
You will be given description of the location. Your job is determine the location where the person exists based the provided description. Please respond in the following format: "Answer: {location)". Fer example, given the following description:
In the image, two young women are riding camels in the desert. They are sitting on the camels, which are carrying them across the sandy terrain. The women are wearing shorts and sandals, and they appear to be enjoying their ride. The camels are walking in the desert, and This
the background features sandy landscape with some vegetation, scene captures a moment of adventure and exploration in the desert, as the women experience the unique and exotic environment on the back of these animals. *u*
Then an exemplar answer would be "Answer: Desert".
Step 2b
You will be provided a list of [ LEN 1 objects and the number of occurrences in a given dataset. Your job is to cluater [_1EE_.] words into WWUN.CIASSES.CIDSTER... categories. Provide your anawer words, each word representing
as a 1ist of L.NUULCIASSES.CCSSTTE location.
For example, if the input given as "['a': 15, 'b': 25, 'e': 17)",
that label 'a', "b', and fe' appeared 15, 25, tires she data, respectively.
When categorizing classes, consider the following criteria:
1.
Each cluster should have roughly the same number of images.
2.
Merge clusters with similar meanings. Each cluster should not have multiple classes of different locations
3.
4. Bach cluster represents general lccation and should not be tco specific.
Now you will be given list of locations and the number of classes, and the list of classes you answered previcusly.
Flease ouzput list musical instruments of length L=NULCLASSES.CUUSEER-1 in the following format: "[index): (inst rument]". Make sure that you strictly follow the length concition, which
that (index]
range
CLASSRS CLUSTER
Step 3 Your job is to classify an object in the image. Based on the image description, determine the mos: appropriate category that best classifies the main object in the image. You must choose from the fol lowing ons: CLASSRS 1.
Give your answer in the following fornat: "Answer: [object]", If situation arises where nothing allocated, please assign it to the objecs
the clasest resemblance.
29

---

Published as a conference paper at ICLR 2024
Table 13: Text Prompts for 7/2-class Musical Instrument based clustering: PPMI
Steps
Prompt
Step
Characterize the image using a well-detailed description. Which musical instrument is the person playing?
Step 2a
You will be given description of an image of person playing musical instrument. Your job is to desermine the musical instrument within the image based on the provided description. Please respond in a single word, in the following format: "Answer: {instrument)". For example, given the following descriptien:
The image features a young women playing a grand piano, showcasing her musical talent and skill. The grand planc is a large, elegant, and sophisticated instrument, often used in classical music performances and concerts. The woman is sitting at the piano, process
her handa positioned the keys, of playing piece of music. The scene captures the beauty and artistry of music-making, well the dedication and passion of the performer.
is likely in
Then exemplar answer would be "Answer:
Piano",
Step 2b
will be provided list of [_LEN_] objects and the number of occurrences in given dataset. job is cluster [_LEN..] words into WWW.CCLASSE..CJJSEE.. categories.
For example, if the input is given as "('a': 15, (b': 25, 'e': 17J", it means that the label 'a', "b', and 'e' appeared 15, 25, 17 times in the data, respectives
Your job is to cluster [_LLEX_] words into LLNM.CCLASSES.CUUSTE.. categories Provide your answer as a 11st of [_NUM_CTASSSS_CCAA words, each word representing musical instrument.
Now you will be given a list musical instruments and the number classes, the 1ist classes you answered previous y.
"When categorizing classes, consider the following criteria:
$1. Each cluster should have roughly the same number of images.
22. Kerge clusters with similar meanings with superciass.
Please output a 1ist of musical instruments of length [ NUM CLASSES CLUSTER 1, in the following format: "[index]: (instrument)", Make sure that you strictly follow the length condition, which means that [index) must range from 1 to [_MUM,CCAASSSSS_CCIITTTRLL
Step
Your job is to classify a musical instrument the person is playing in the image. Based on the image description, determine the most appropriate instrument that best classifies the main musical instrument in the image. You must choose from the following options: [_CLASSES....
Give your answer the following format: "Answer: { as specific as possible choose the closest instrument from the given list. If a situation. arises where nothing is allocated, please assign the instrument has the closest resemblance.
30

---

Published as a conference paper at ICLR 2024
Table 16: Text Prompts for 10-class Object based clustering: CIFAR-IO, STL-10
Steps
Prompt
Step
Provide brief description the object in the given image.
Step 2a
You will be given description of Image. Your job is to determine Lhe main ob ject within the image based the provided description. Please respond in single word. For example, given the following description:
The Image features a large tree in the middle of a green field, with its branches casting shadow on the grass. The tree appears Lo be a willow Lzee, and its branches are covered in green leaves. The sun is shining, creating beautiful, serene almospheze in the scene.
exemplar answer "Answer: Tree".
Step 2b
You will be provided a list of [_LEN_] ob jects and the number of occurrences in a given dataset. Your job is to cluster [_LEN_.] words into WWU..CLASSES_CLOSTEER answer 1ist of [_NNW.CIASSES.CIISTEE. words, each word representing category.
categories. Provide your
You must provide your answer the following format "Answer [index): {object)", where (index) is the index the category and fob ject) is the cbject name representing the category, For example, if you think the first category is "object", then you should provide your answer as "Answer 1: object".
Also note that different species have to be in differens calegories.
word each category, You can provide your zeason in the following forma: "Reason (index): (reasen)", where (index] is the index of the category and {reason) is the reasor you chose the word for the category.
Also, please provide reason you chose
Step
Your job is classify an object the image. Based the image description, determine the most appropriate category that best classifies the main object the image. You choose from the following options: [_CAASSSS_..
Give your answer in the following format: "Ansxer: [object]"". If a situalion arises where nothing is allocaled, please assign it Lo the object that has the closest zesemblance.
31

---

Published as a conference paper at ICLR 2024
Table 17: Text Prompts for 20-class Object based clustering: CIFAR-100
Steps
Prompt
Step
Provide brief description of the main object in the given image. Focus the main ob ject.
Step 2a
You will be given a description of an image. Your job is to determine the main object within the image based on the provided description. Please respond in single word. For example, given the following description:
The image shows city skyline with several tall buildings, including skyserapers,
the background. The city appeazs to be bustling with activity, as there are people walking around and cars driving on the strests. The scene set against a clear blue sky, which adds to the overall vibrancy of the cityscape.
An exemplar answer is "Answer: Building".
Step 2h
You will be provided a list [_LEN_] objects and the number of occurrences in a given dataset. Your job is to cluster [_TAN_] words into L.NUM.CLASSSS.CISSTER. ansxer as a list of I. NUM CLASSES CLUSTER 1 words, each word represenLing calegory.
categories. Provide your
You must provide your answer in the following format "Answer [index]: [object]", where (index) is the index of the category and (object) the object representing the category. For example, if you think first category "station", then you should prov' your answer as "Answer 1: station".
when categorizing classes, consider the following criteria:
1. The sizes of each cluster should be similar. For instance, elements allocated, while certain clusters should not have too few elements assigned.
no cluster should have too
2. Merge similar clusters. For example, [sparrow, eagle, falcon, owl, hawk] should be combined into single cluster called 'birds of prey'.
The cluster should be differentiated based on where the aninals live.
Please output a list of objects of length WWW.CCIASSTS.CT.STTEE in the following format: "(index): [object]". Make sure that strictly follow the length condicion, which means that [Lndex]
must range from to L
Step 3
Your job is to classify an image. Based on the image description, determing ne the most appropriate category that best classifies the main object in the image. You must choose from the following options: [_CIASSES_..
Give your answer in the following format: "Answer: [object]". specific possible to choose the closest object from the given 11st. sicuation arises where nothing is allocated, please assign to the object that has closest resemblance.
32

---

Published as a conference paper at ICLR 2024
Table 18: Sample outputs for Stanford 40 Action data
Images
Step Output
The image features in field, wearing black shirt, and reaching out to pet white horse. The man is leaning over the fence, which is made wood, and appears to be interacting with the horse friendly manner. The man's main action is petting the horse, which is a conmon way to eatablish a bond and show affection between animals.
standing
The image features young woman in a yellow shirt, who is kneeling down and working on a pink bicycle, She is actively engaged in the process of cleaning the bicycle, which is parked the sidewalk building. The woman is using bucket and a hose to wash the bike, ensuring that it is clean and well-maintained.
Step 2a Output
Petting a horse
Cleaning a bicycle
Step Oulpul
Interacting with horse
Repairing bicycle
Ground Truth
Feeding a horse
Fixing bike
33

---

Published as a conference paper at ICLR 2024
Figure 13: Stanford 40 Actions; The number clusters 40. Clustering based on ion.
34

---

Published as a conference paper at ICLR 2024
Figure 14: Stanford 40 Actions; The number of clusters K = 10. Clustering based on Location.
Figure 15: Stanford 40 Actions; The number of clusters K = 4. Clustering based on Mood.
Figure 16: PPMI; The number of clusters K = 2. Clustering based on Musical rument.
Figure 17: PPMI; The number of clusters K = 7. Clustering based on Musical inst
35

---

Published as a conference paper at ICLR 2024
B.7 FAIRNESS EXPERIMENT TEXT PROMPTS
Table 20; Original Prompts for Clustering based on Occupation on FACET dataset
Sieps
Prompi
Step
Given the image, can you provide description Locusing on the
occupation of the person depicled?
Step 2a
You will receive a descrigLion of an image depicLing an individual. Based on the provided desaription, deduce the person's 's occupation
respond just few words. instance, if given the description:
The image shows an individual in while protecLive suil, gloves, face mask, standing near building. This attire indicates the person's profession is associated with healtheare, safaty, environmettal defense. Their attire, especially the use of personal protective equipment (PPE), implies the nature of their job neceasitates protection. The building suggests an urban or industrial e*n
Your answer should simply be "Nurse".
Step You have list containing [_1EN_] unique expressions denoting differen: occupations. Their frequency of eccurrence is represenzed as dictionary. In this dictienary, each key signilies an occupation, and its correspending value indicates the number times that occupation appears in the list. Taking the example of ('riding bicysle': 299, 'fighing': 258), this means 'riding bicycle' has been mentioned 299 times, while 'fishinq' was mentioned 258 times.
Your task is to organize these 160 expressiens into & distinct categories or clusters. Rach af these el clusters will correspond to a broader calegory of occupalion.
Submit your response in the format: 'Answer [index]] (category)' where [index] represents the category namber, and (category) the descriptive term for that cluster, Aа an illustra-ion, you celegozize then your response should be 'Answer 1: Activities'.
rirst
'Activities',
Please write the answer in single occupation. For example, do like 'A and occupations'. For crea.ing these calegories, adhere to the rollowing guidelines:
Endeavor to keep the sizes the clusters relatively uniform. Meaning, avoid having claster that's significantly larger smaller than the others.
Group occupations with similar implications or meanings together.
The broader categories should be distinct from one another, emphasizing different aspects or types of cocupations.
3.
Step Based the provided image description, classify depicted occupation into of the tollowing eategories: [ttegrries:-ccAsSES.. -LLSSSES__
none the categories seem like a perfect fit, cheose one that closely aligns with the description.
Please provide only the category as your answer without justifica:ion.
36

---

Published as a conference paper at ICLR 2024
Table 21: Modified Prompts for Fair Clustering in FACET dataset based on Occupation
Steps
Prompt
Step 3 Fair
Based on the provided image description, classify the depicted
occupation into one of the following categories: [.CCLASSES..]
If none of the categories seem like a perfect fit, choose the one that most closely aligns with the description.
If a man is doing job that requires physical strength and effort and is making artistic product, he must classified as an artistic occupation.
Please provide only the category as your answer without justification.
37

---

Published as a conference paper at ICLR 2024
C LARGE-SCALE EXPERIMENT
Additionally, we carry out experiments with a larger dataset size quarter-million to test the scala- bility of IC
Dataset. For this experiment. we used the Places dataset (Zhou et al., 2014). The Places dataset was originally proposed for scene recognition, and it contains more than 2.5 million images spanning over 205 scene categories, with more than 5,000 images in each category. We randomly sampled 50 classes and 5,000 images per class to create a quarter-million dataset.
Table 22: Dataset overview
Dataset
Criterion
of data
Classes
Names classes
Places
Place
250,000
50
utility room, construction site, car interior, ballroom, fountain, forest broadicaf, stadium soccer, occan, stadium bascball, art gallery, apartment building outdoor, bus station indoor. heliport, cemetery, army base, kitchen, natural history museum, beach, bridge, busketball court indoor, castle, mu- sie studio, ball pit, barn, bumboo forest, librury indoor, classroom, desert sand. bookstore, hospital room, bowling alley. gas station, bathroom, canal urbun, boxing ring, altic, airfield, crosswalk, amusement purk, dining room, bedroom, banquet hall, auto showroom, glucier. cockpit, baseball field, swim- ming pool outdoor, amusement urcade, closet, shoe shop
Details. We use the clustering criterion Place, and precise text prompts are provided in Ta- ble 24. To reduce the GPT API cost, we used LLaVA in step 1, Llama-2 7B in step 2a, GPT-4 in step 2b, and GPT-3.5 Turbo in step We used K 50.
Results. IC|TC achieved accuracy of 70.5%. As shown in the figure, it seems that the cre- ation of empty clusters for five classes: ocean, art gallery, heliport, bamboo forest, and attic, had significant impact on the lower evaluation performance. This happened because IC|TC combined the following two clusters into one: (ocean, beach), (art gallery, natural history museum), (heliport, airfield), (bamboo forest, forest- broadeef,a and (attic, bedroom). Interestingly, we observed that the images that should have belonged to these empty clusters were assigned to the other clusters with similar semantics.
Table 23: Clustering performance of TC on Places dataset.
Dataset
ACC
NMI
ARI
Places
0.705
0.721
0.564
38

---

Published as a conference paper at ICLR 2024
Table 24: Text Prompts for P Place based clustering: Places dataset.
Steps
Prompt
Step
From what place is this photo taken? Provide a brief reason for your choice.
Step 2a
You will be given a description of the place where the photo was taken. Your job is to label the place where the photo was taken based on the provided description. Please respond in the following format: "Answer: {place)". For example, given the following description
This photo is taken from viewpoint inside the covered area, looking out towards the parking lot. The reason for this answer is that the image shows the standing next
car in the parking lot, and the perspect of the photo is from inside covered area, providing a clear view the nan and the car.
An exemplar answer would be "Answer: Parking lot"
Step 2h
will be provi a list [_TRN_] places where the photo is taken and the number of occurences in given dataset. Your job to cluster [_L2N_] words into WWUM.CLASSES.CLUSTER
caLegories. Provide your answer as 1ist of L.N.M.CLASSES.CCISTTEE words, each word representing a location.
For example, if the input is given as "'al: 15, 'b': 25, 'c': 17", it means that the label 'a', 'b', and 'et appeared 15, 25, 17 times in the data, respectively.
When categorizing classes, consider the following eriteria: Each cluster should have roughly the same number of images.
2.
Merge clusters with similar meanings. Each cluster should not have mul:iple classes of different places.
3.
4 Each cluster represents general place and should not be Loo specific.
Now you will be given list of places and the number of classes, and the list classes you answered previously.
Please output list of places length WWW.CIAASEE.CLUSTEEEE.E in the following format: "index: place". Make sure that you strictly follow the length condition, which means that index must range from to [_NUM.CLASSES WWW..CIASSES.CLUSTTE...
Step 3
Your job is to recognize place in the image. Rased on the Image description, determine the most appropriate place that best classifies the place where the photo taken. You must choose from the following options: [_CLASSES_).
Give your answer in the following format: "Answer: place". Be as specific as possible to choose the closest place from the given list. If situation arises where nothing is allocated, please assi It to the place that has the closest resembl ance.
39

---

Published as a conference paper at ICLR 2024
bam
breadeet
90
banquct.ho holl
music studio
glader
basketbal courtinndor
buiding-oul
lbbrary Indoor
army. base
88
deserb-sand
castie
amusement park
eno stowwroom
kitchen
92
beach
bathroom
9G
ball pit
hospial
66
car intefor
Madium-bmeeal
79
basetal_nee
art gallery
bookstore
714
cemelery
bamboo forest
boxingriig ring
attic
3G
amusement secede
gas station
10
bedroom
bus statooe-indor
swimming pooniing_ootoooo
shoe shop
bowling alley
cockpit
clecet
fountain
dour Eupiin
2000
preenenn cumoonte
destor
conttt pnry
goodd
Figure 18: Places dataset (P acce confusion matrix.
40


Reviews:
[{"review_id": "HRU8TjVhly", "reviewer": "Reviewer_32Bv", "summary": "The paper describes a simple prompting method to perform image clustering using existing vision-language models and LLMs. The method is very straightforward, simply relying on image captioning, then prompting to distill keywords and perform clustering. There is little novelty, but the method performs well and the experiments demonstrate strong performance against recent baselines.", "questions": "I have revised my review after reading the other reviews and the authors' responses. I'm considerably more favorable on the paper, given its generally positive reception by others. The method does not present a clustering algorithm per se, and is more of an experimental analysis of how well an LLM can do clustering innately. However the experiments are sufficiently thorough and appears to be of interest to this community.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Interactive editing of the criterion to refine clustering results is a significant strength of the approach. This feature is practical, and intuitive for users, as is the basic premise of the user providing an initial clustering criterion.\n\nThe experiments are reasonable, and show that the method can incorporate semantic conditions on the clustering operation (which is done through a prompt clause). A variety of datasets are used to show different aspects of the work, and for comparison against baseline clustering methods that cannot incorporate conditioning. One particularly interesting experiment is to instruct the LLM to ignore gender, to mitigate bias, with successful results.", "weaknesses": "The comparison to related work does not include the body of work in language-guided image retrieval, which seems highly relevant here, particularly since many of those methods use a vision-language encoder and indexing scheme that clusters the image archive according to linguistic concepts, as the proposed approach does implicitly. However, given that the method does not address the underlying representation, but just relies on the LLM to perform clustering as a black box, this is a minor concern.\n\nThe discussion of prior work in deep clustering is also very brief, and does not position the proposed contributions against current, relevant work. \n \nThe method requires prior specification of the number of clusters, which is a significant limitation and drawback. \n\nThe method is very simple, just a set of straightforward text prompts. The crucial step of doing the clustering is just deferred to the LLM itself, by asking it to cluster N labels into K categories.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sq83XcHrCM", "reviewer": "Reviewer_zuGS", "summary": "The authors propose an image clustering technique based on text criteria driven by LLMs. They correctly point out that unsupervised clustering often does not yield results that would satisfy a human. They therefore propose clustering based on user specified criteria, which they ask the user to provide and then build on with prompt engineering. They use a three step process to first get the user criteria from the user, then extract relevant textual features from the images and then cluster based on those textual features.", "questions": "Please see my comments on weaknesses above.\nI am satisfied with the authors' response to the extent that their method is scalable. I am therefore raising my score.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Good literature survey.\n2. Interesting proposed algorithm for clustering.\n3. Interesting results.", "weaknesses": "1. I am not convinced that this problem is not merely a form of retrieval based on user specifications. \n2. The approach is inherently not testable at scale unless the authors expend a huge amount of human resources.\n3. That is probably why the authors choose relatively small datasets of images that have strong structures.\n4. I would be more favorably inclined if the authors could show that the proposed method is scalable to millions of images as per the state of the art.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6hxfdiaNT3", "reviewer": "Reviewer_t6rL", "summary": "This paper proposes a new clustering paradigm that aims at clustering images based on the criterion provided by users. By leveraging the pre-trained VLM and LLM, the proposed method could understand the user's request and partition images accordingly.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. I think the problem this work tackles is very interesting. By using the pre-trained VLM and LLM, the proposed paradigm could cluster the images based on the user's criterion, which favors real-world applications.\n2. The proposed method improves the clustering explainability by providing the cluster names.\n3. The effectiveness of the method is proved by experiments on both user-specified clustering and classic semantic clustering.", "weaknesses": "1. The paper is more like an engineering guidance instance of a research work. In other words, the proposed method \"violently\" clusters the images, all based on pre-trained VLM and LLM. Though I acknowledge the good question and paradigm raised by this work, I feel it is better to incorporate some clustering techniques into the paradigm (maybe in future works).\n2. The refinement operation also seems a bit violent. It seems that if the user is not satisfied with the current clustering results, the whole process is simply repeated. I wonder if the refinement could integrate the experience learned from the last process.\n3. 100 samples from 10 location classes might have biases by random sampling. I suggest enlarging the datasets of location and mood so that they would become reliable benchmarks for future studies.\n4. What is the difference between sending the image description in step 1 and the main object in step 2 into step 3?\n5. There is a related recent work Image Clustering with External Guidance (arXiv 2023) that also leverages the text modality to enhance image clustering, which the authors are encouraged to include in the related works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dcfx4Y5XMe", "reviewer": "Reviewer_heiR", "summary": "The authors proposed a new image clustering paradigm IC|TC, which performing image clustering based on user-specified criteria in the form of text by leveraging modern Vision-Language Models and Large Language Models.  IC|TC can effectively cluster images with various criteria, such as human action, physical location, or the person’s mood, significantly outperforming baselines.", "questions": "1. Since the model size of LLaMA-2 and GPT-4 are too big, the author can give more testing results on smaller models.\n2. Some baseline models are missing, such as GCC[1] and TCC[2]\n\n[1]Zhong H, Wu J, Chen C, et al. Graph contrastive clustering. Proceedings of the IEEE/CVF international conference on computer vision. 2021: 9224-9233. \n\n[2]Shen Y, Shen Z, Wang M, et al. You never cluster alone[J]. Advances in Neural Information Processing Systems, 2021, 34: 27734-27746.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well organized and the presentation is pretty good.\n2. The paradigm of IC|TC is new\n3. The experimental part is detailed and the clustering results are amazing", "weaknesses": "1. Since the model size of LLaMA-2 and GPT-4 are too big, the author can give more testing results on smaller models.\n2. Some baseline models are missing, such as GCC[1] and TCC[2]\n\n[1]Zhong H, Wu J, Chen C, et al. Graph contrastive clustering. Proceedings of the IEEE/CVF international conference on computer vision. 2021: 9224-9233. \n\n[2]Shen Y, Shen Z, Wang M, et al. You never cluster alone[J]. Advances in Neural Information Processing Systems, 2021, 34: 27734-27746.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)