Image Clustering via the Principle of Rate Reduction in the Age of Pretrained Models
Authors: Tianzhe Chu
Shengbang Tong
Tianjiao Ding
Xili Dai
Benjamin David Haeffele
Rene Vidal
Yi Ma
Venue: ICLR 2024 poster (2024)
Keywords: image clustering
pretrained models
rate reduction
principled methods
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper proposes a novel image clustering pipeline that integrates pre-trained models and rate reduction, enhancing clustering accuracy and introducing an effective self-labeling algorithm for unlabeled datasets at scale.

Abstract: The advent of large pre-trained models has brought about a paradigm shift in both visual representation learning and natural language processing. However, clustering unlabeled images, as a fundamental and classic machine learning problem, still lacks an effective solution, particularly for large-scale datasets. In this paper, we propose a novel image clustering pipeline that leverages the powerful feature representation of large pre-trained models such as CLIP and cluster images effectively and efficiently at scale. We first developed a novel algorithm to estimate the number of clusters in a given dataset. We then show that the pre-trained features are significantly more structured by further optimizing the rate reduction objective. The resulting features may significantly improve the clustering accuracy, e.g., from 57\% to  66\% on ImageNet-1k. Furthermore, by leveraging CLIP's multimodality bridge between image and text, we develop a simple yet effective self-labeling algorithm that produces meaningful text labels for the clusters. Through extensive experiments, we show that our pipeline works well on standard datasets such as CIFAR-10, CIFAR-100, and ImageNet-1k. It also extends to datasets without predefined labels, such as LAION-Aesthetics and WikiArts.

Published as a conference paper at ICLR 2024
IMAGE CLUSTERING VIA THE PRINCIPLE OF RATE REDUCTION IN THE AGE OF PRE-TRAINE MODELS
Tianzhe Chu
Shengbang Tong" 1*
Tianjiao Ding
Xili Dai
Benjamin D. Haeffele
René Vidal
Yi Ma
ABSTRACT
The advent of large pre-trained models has brought about a paradigm shift in both visual representation learning and natural language processing. However, clustering unlabeled images, as a fundamental and classic machine learning prob- lem, still lacks an effective solution, particularly for large-scale datasets. In this paper, we propose novel image clustering pipeline that leverages the powerful feature representation of large pre-trained models such as CLIP and cluster images effectively and efficiently at scale. We first developed a novel algorithm to estimate the number of clusters in given dataset. We then show that the pre- pre-trained features are significantly more structured by further optimizing the rate reduction objective. The resulting features may significantly improve the clustering accuracy, e.g. from 57% to 66% on ImageNet- Ik. Furthermore, by leveraging CLIP's s multimodality bridge between image and text, we develop simple yet effective self-labeling algorithm that produces meaningful captions for the clusters. Through extensive experiments, we show that our pipeline works well standard datasets such C1FAR-10, CIFAR- 100, and ImageNet- k. also extends to datasets that are not curated for clustering, such LAION-Aesthetic and WikiArts. We released the code in https: : httpt/ttttuuubee com wow. ieerue//PP.
MOTIVATION
Clustering is fundamental problem in machine learning, with many common methods emerging as early as the 1950s (Lloyd, 1957; Forgey, 1965; Jancey, 1966; McQueen, 1967) along with numerous modern developments. Nevertheless, there is significant discrepancy separating the recent advance of large-scale methods from that of clustering performance. Namely, there are datasets with millions (Russakovsky et al., 2015) even billions (Schuhmann al., 2022) of images and thousands of classes and classification methods with close to 100% accuracy, yet existing clustering approaches typically either fail on natural images (Bradley al., 1996; Bahmani et al., 2012: Souvenir & Pless, 2005; Elhamifar & Vidal, 201 1; 2013; Patel & Vidal, 2014; Lu et al., 2012: Liu et al., 2013; Heckel & Böleskei, 2015; You al., 2016a; Tsakiris & Vidal, 2017), or have been tested only with datasets of small number of clusters (~ 1 and images (~ 10²) (Park et al., 2021; Li et al., 2021; Yaling Tao, 2021; Deshmukh et al., 2021; Niu & Wang, 2021; Li et al., 2022; Sadeghi al., 2022) with some exceptions (Van Gansbeke et al., 2020; Adaloglou et al., 2023). So far, few methods have achieved above 50% clustering accuracy on ImageNet-Ik 1k or TinylmageNet- (2020); Li et al. (2021); Niu & Wang (2021); Sadeghi et al. (2022); Ding et al. (2023) all achieve accuracy smaller than 50%.
c.g., Van Gansbeke et al.
Classic clustering methods often build on assumptions about the geometry of data from each cluster, such as modeling each cluster as centroid (Lloyd, 1957: Forgey, 1965; Bradley et al., 1996; Arthur & Vassilvitskii, 2006), a linear or affine subspace of low (Elhamifar & Vidal, 2013; Heckel & Bölcskei, 2015; You al., 2016a) or high dimension (Tsakiris & Vidal, 2017: Ding et al., 2021: 2024), manifold from known families (Patel & Vidal, 2014), sampled densely (Souvenir & Pless, 2005), or locally approximated by affine subspaces (Elhamifar & Vidal, 2011). Although effective on relatively simple datasets such as COIL. (Nene et al., 1996) or MNIST (LeCunn 1998), these methods are either not accurate (the geometric assumptions are drastically violated) or not scalable (computing a neighborhood graph is expensive) when confronted with more complicated or large-scale datasets such as CIFAR (Krizhevsky et al., 2009) and ImageNet (Russakovsky et al., 2015).
"Equal contribution, Universitty of California, Berkeley, 'ShanghaiTech University, University of Pennsyl- vania, ' Hong Kong University of Science and Technology (Guangzhou), 'Johns Hopkins University. "Hong Kong University

---

Published as a conference paper at ICLR 2024
Key to the recent advance in clustering is teaching an old dog new tricks, i..., using deep networks to learn features with desirable geometric properties that can be used for clustering (and other downstream tasks). Recent clustering pipelines (Van Gansbeke et al., 2020; Ding et al., 2023; Niu et al., 2022) proceed by 2 steps: i) learning an initial representation by self-supervised learning, e.g., the joint-embedding approaches (Chen al., 2020; He et al., 2020; Tong et al., 2023), and ii) gradually refining the representation and clustering membership after the initialization. One family of methods (Li al., 2022; Ding et al., 2023) base their design on the principle of Maximal Coding Rate Reduction (MCR?, Yu et al. (2020)). In short, these methods aim to learn a representation such that features within the same cluster tend span a oow-dimensional subspace (i.e., within-cluster diverse), and subspaces from different clusters tend to be orthogonal (between-cluster discriminative). Promising as it may seem, clustering methods initialized by self-supervised learning highly depends on the initial representation, and thus clustering performance is far from the supervised classification baselines on CIFAR-I00 or ImageNet-Ik. lk.
In parallel with these developments, the advent of large-scale pre-trained models such as Contrastive Language-Image Pre-training (CLIP, Radford et al. (2021)) and Self-Distillation with No Labels (DINO, Caron et al. (2021); Oquab et al. (2023)) have showcased an impressive capacity to learn rich representations from a wide variety of datasets. In particular, CLIP trained by pairing diverse set of images with natural language descriptions. It has been shown to serve as foundation model that scales up to training data, making it highly suitable for tasks that require a nuanced understanding of visual information.
Contributions. To address the challenges inherent in clustering large-scale and uncurated data and really push the limit of clustering, we leverage the advance in both pre-trained models and principled clustering approaches to develop novel pipeline, named CPP (Clustering via the Principle of rate reduction and Pretrained models). In particular, this paper makes the following contributions:
We propose to integrate the powerful image encoder from CLIP into the clustering frame- work MLC, and demonstrate that such combination leads to state-of-ther art clustering performance on standard datasets CIFAR-10 and -20 (Krizhevsky et al., 2009), and further on large-scale datasets CIFAR-100 and ImageNet- (Russakovsky et al., 2015); the latter two are often ignored by prior works.
2. While prior clustering methods typically assume the number of clusters is given, it is often unknown for large and uncurated datasets. Therefore, we provide model selection mechanism suitable for MLC that estimates the optimal number of clusters without any costly retraining. We validate this mechanism on CIFAR-10, -100, and further apply it to MS-COCO (Lin et al., 2014), LAION-Aestheti (Schuhmann et al., 2022), and WikiArt (Saleh & Flgammal, 2015).
3. To further label the obtained clusters with semantic descriptions that can be comprehended by human, we propose simple yet effective self-labeling algorithm utilizing the vision-text binding provided by CLIP. We show that the entire pipeline yields semantically meaningful clusters MS-COCO, LAION-Aesthetic, and WikiArt.
2 RELATED WORK
In this section, we broadly review some of the work that has inspired our research. We begin with the recent progress in pre-trained models, and then discuss the recent trend of image clustering via pre-trained models.
Pre-trained Models in Vision. In recent years, have witnessed the rapid development of vision pre-trained models in the field. Pure vision models have benefited from advances in joint-embedding self-supervised learning (Chen al., 2020; He al., 2020; Bardes al., 2021; Caron al., 2021; Grill et al., 2020) and masked self-supervised learning (He et al., 2022; Zhou et al., 2021; Bao et al., 2021). These models have shown great promise in learning semantically meaningful features on large scale of datasets.
Another line of work focuses on learning meaningful representation from images with the guidance of text information. Inspired by the progress in contrastive learning (Chen et al., 2020), CLIP proposes contrastive language-iman pre- pre-training. The work demonstrates incredible scalability and very strong performance. Follow-up reproduction openCLIP (Ilharco et al., 2021) has verified that the method's ss scalability with the largest vision transformer model (Dosovitskiy et al., 2020) and the
2

---

Published as a conference paper at ICLR 2024
Trainingg
2 Clustering:
3 Captioning:
Initialize representations X pre trained visual model
Obrain structured representations and member II from trineed model
Obcain M. clusters from previous stops and assign semantic lahels each cluster
Visual Model
xb
Images from each cluster
Updating optimizing objective
11st 2
CLIIP Encoder
Spectral Chastering into k Iyors K clasters
Pre-featare layer
Cosine similarities
Fenture head
?
Cluster bead
L(Z)
CLIP Text Enender
photo a cledi consiilate D to photo ot stext condioiate 2>"
1
proja, (CTT
Seashore"
Voting
MLCIZ, II)
Optimal clusters
Figure Overall pipeline of CPP. Left: In the training stage, CPP initializes the features and cluster membership II from large pre-trained model, and updates 2 and II by optimizing the (MLC) objective. Middle: Once training is done, CPP selects the optimal number of clusters via the coding length L()) criteria. Right: CPP assigns semantic captions to each cluster via computing cosine similarities between text candidates and images and voting for the most suitable caption.
largest dataset containing more than 5 billion text-image pairs (Schuhmann al., 2022). As result, in this work, we adopt CLIP as our pre-trained model to design truly scalable and effective clustering learning algorithm.
Image Clustering via Pre-trained Model. The advance in vision pre-trained models have led to major breakthroughs in image clustering. SCAN (Van Gansbeke et al., 2020) proposes three-step image clustering pipeline, starting with a self-supervised SimCLR (Chen et al., 2020) model, training preliminary cluster head on the pre-trained features, and fine-tuning the trained cluster head via confidence- based sseudo-labeling Subsequent research such as RUC (Park et al., 2021). TSP (Zhou Zhang, 2022) and SPICE (Niu et al., 2022) has further enriched the field by exploring robust training, alternative network architecture, and different finetune methods. A few works (Tong et al., 2022a; Kim & Ha, 2021) also explored this paradigm in generative models, showcasing some promising downstream applications.
These pioneering methods have undoubtedly made remarkable progress. However, they often involve a degree of intricate engineering and parameter tuning. While these complexities are inherent their design and have enabled them to achieve their goals, they may present challenges when implementing and scaling these methods to larger datasets. Recently, approaches like NMCE (Li et al., 2022) and MLC (Ding al., 2023) have connected manifold clustering and deep learning via the MCR principle (Yu et al., 2020; Ma et al., 2007a). particular, MLC (Ding et al., 2023) has demonstrated that this principled approach has shown promise in terms of efficiency, scalability, and capability to handle imbalances present larger datasets (Schuhmann et al., 2022; Lin et al., 2014) and real-life data. Consequently, draw inspiration from ML_C develop truly scalable and effective image clustering pipeline capable of handling the scale and natural imbalances present in real-world data.
OUR METHOD
We begin in $3.1 with a brief review of Manifold Linearizing and Clustering (MLC) (Ding et al., 2023), framework that learns both a representation with desirable geometric properties and clustering membership. In $3.2, we discuss how to effectively train MLC leveraging CLIP features initialization. When the number of clusters is further not known, describe how identify the optimal number of clusters without costly retraining in $3.3. few applications are left to $3.4: captioning the clusters and image-to-image search.
3

---

Published as a conference paper at ICLR 2024
3.1 REVIEW OF MANIFOLD LINEARIZING AND CLUSTERING
Given a dataset X' = [z),..., Fn. E RDx= of n points lying on k unknown manifolds, how to i) cluster the points in X' and ii) learn a representation for X that has desirable geometric properties?
Diverse and Discriminative Representation. A fruitful line of research (Yu et al., 2020; Chan et al., 2020; Dai et al., 2022; Baek et al., 2022; Han et al., 2022; Li et al., 2022; Tong et al., 2022b), including MLC (Ding et al., 2023), considers learning representation by using the principle of Maximal Coding Rate Reduction (MCR²). Roughly speaking, an ideal representation pursued by MCR should have the following properties.
Within-cluster diversity; The features of each cluster spread well in a low-dimensional linear subspace. This naturally leads to a notion of (non-trivial) distance within a cluster, benefiting downstream tasks such as image retrieval (as shown $4.2), generation, or interpolation.
Belween-cluster discrimination: Features (or subspaces) from different clusters are orthogonal. This is typical goal in representation learning (Bengio et al., 2013), e.g., as pursued by the cross- entropy objective (Papyan et al., 2020; Zhu et al., 2021). Note that when each cluster modeled by a low-dimensional linear subspace, this property further aids denoising in the feature space.
In partieular, MLC seeks find both representation 2 E TRdxn of the data and doubly stochastic clustering membership II E _ where each entry Had 2 O measures the similarity between the i-th and j-th points, Here, Z = (x)),,,,....... , foy (x.)) where fn -R= + I denotes neural network with parameters n, while II is produced by network parameterized by w and a doubly stochastic projection (detailed below). MLC finds Z and II by maximizing
max 7,0
R(Z(n): E) (,) E),
(MI.C
R(2; e) = log I+ d 22
where
(1)
x² n
I+ (I+ Z diag [II )2)
d
Re I1: =
log det
(2)
Broadly speaking, R(2; E) measures the volume of points in Z up to a > 0 rate distortion coding precision. Likewise R, II; 5) measures the sum of volumes of the clusters encoded by II. Thus, ML.C proposes to maximize the difference between the volumes, i.e., expand the volume of the embedded data globally, while compressing the volume of the embedded data within a cluster. This has been shown in (Yu et al., 2020) to provably Iearn a representation where features in each cluster (for a fixed II) spread well in oiw-dimeniional linear subspace, and subspaces from different clusters are orthogonal to each other.
Doubly Stochastic Membership. To learn membership of the data for clustering, MLC draws inspiration from the success of doubly stochastic clustering (Lim et al., 2020; Ding et al., 2022)
and computes doubly stochastic membership from some latent codes of data. Specifically, MLC
adopts a neural network Iw RD + IRd with parameters w to first obtain latent codes C
[gw (x11, (a01),...99e) (x,,)) for each datapoint. Then, II is given regularized projection proja, (C C) onto Q, where Ω denotes the set of doubly stochastic matrices
Ω = [II E R III = -T = =1; 2220, Vi,j)-
(3)
Here, (-) is defined as argminnes - C, + H.g log II which can be efficiently computed (Hisenberger et al., 2022; Sander et al., 2021), and y is a regularization strength that controls the entropy of II; in short, the larger y is, the more uniform II is. Note that roughly speaking less uniform II solutions result in fewer false connections between different clusters at the expense of less inter-cluster connectivity, We highlight here the dependency of Z and II on their respective network parameters 77, w, which we will typically omit for simplicity of notation.
3.2 TRAINING MLC: LEVERAGING AND REFINING CLIP FEATURES
Structured Feature and Membership Initialization via CLIP. Since (MLC) is non-convex, its initialization and optimization dynamics play vital role in performance. Prior work (Li et al., 2022; Ding et al., 2023) used image superviee learning to initialize the features, which often fail to capture nuanced semantics, and result, they only reach, e.g., 33.5% clustering accuracy on
4

---

Published as a conference paper at ICLR 2024
Tiny-Imageti (Ding et al., 2023). We describe in the sequel how to initialize the representation and membership leveraging a pre-trained CLIP (Radford et al., 2021) model. Recall that CLIP has an image encoder and text encoder, which maps imput image and text respectively to a joint feature space. These encoders are trained utilizing billions of image-caption pairs. Motivated by the remarkable performance of CLIP in doing zero-shot tasks unseen datasets, we take its pre-trained image encoder as our backbone (or feature extractor). The parameters of the backbone are henceforth fixed. Equivalently, we are taking the input data X' be CLIP features rather than raw images.
Refining CLIP Features via MI.C. To allow fine-tuning of both 2 and II, it is natural to add extra trainable layers after the backbone, as seen by the feature head and cluster head in Figure 1. However, these added layers could be arbitrary due to random initialization, undermining the quality of Z and II Moreover, seen in Figure 3, the pre-trained features from CLIP often have moderate wise similarity even for those from very different classes. Toward this end, we propose to diversify the features 2 simply via
max R(Z(n): e),
(4)
which is precisely (I); similar ideas have been explored also in et al., 2022; Tong et al., 2023). In practice, we find that updating (4) only 1-2 epochs suffices to diversify Z, making it a lightweight initialization step. To initialize II, we copy 1 to w (equivalently, assigning C = Z) without extra training effort, which is benefit of using doubly stochastic membership. With both Z and II initialized, we proceed and optimize all the added layers in (MLC). Once the training process is done, one can use the feature head and cluster head to obtain 2, C for any set of images, seen or unseen; C in turn gives II following $3.1. When the number l of clusters is determined or given, one can simply run spectral clustering (von Luxburg, 2007) on II to get clusters.
3.3 DETERMINING NUMBER OF CLUSTERS WITHOUT RETRAINING
In many scenarios, is impossible for one know the number of clusters. Two issues arise: i) one must have a mechanism to guess the number of clusters, and ii) to obtain an accurate estimate, one typically runs the entire deep clustering pipeline multiple times, which is computationally costly. In this regard, we propose to estimate the number of clusters without expensive retraining. This flexibility which alleviates ii) is attributed to the following fact: Note that the membership II E R merely signals pairwise similarity (recall $3.1), so the number i of clusters is not part of the training pipeline of (MLC) whatsoever in contrast to the more common way of using n l matrix which encodes & explicitly (Li et al., 2022). That said, given a II E Raxx, how can one know what is reasonable number of clusters?
Towards this end, we again leverage the minimum coding length (or rate) criteria (Ma et al., 2007a),
this time including not only the cost of features but also that of the labels. Recall the definition of coding length from (Ma et al., 2007a)
L(z) = (n + d)R(2; s).
(5)
We now present Algorithm estimate the number of clusters. Assume that the training is done and one already has 2 and II. For each & E (1..... K} where K is the max possible number of clusters, we do spectral clustering on II to obtain & clusters. Let 2μg denote the features from the i-th cluster and |2aq] denote the number of features in i-th cluster. Then, (7) gives the cost LA of coding k clusters, which includes the cost of the features of each cluster as well as the labels, which are the first and second terms in the summation. Finally, one can choose the optimal number of cluster that gives the lowest cost L.k-
3.4 CLUSTER CAPTIONING AND IMAGE-TO TO-IMAGE SEARCH
We end the section by noting that since MLC refines the CLIP features as well as performs clustering, several interesting applications could be done, such as captioning the learned clusters, as well as image-to-image search. For the interest of space, we showcase these applications in $4 and detail the procedures in Appendix H and Appendix G.
4 EXPERIMENTS
In the sequel, we empirically verify the effectiveness of the CPP pipeline. We first compare CPP with state-of-the-art alternatives CIFAR-10, -000 (Krizhevsky et al., 2009) and ImageNet-
5

---

Published as a conference paper at ICLR 2024
Algorithm 1: Clustering without knowing the number of clusters
Input: Learned features 2 E Rdx= and membership IT E RR*x,, max. # of clusters K E Z4
For <- 1,.... K:
[1 ... 1k Spectral clustering II to get & clusters for 2;
(6)
(-log ( [
L
(( + 12
(7)
Output: Optimal number of clusters argming T
Russakovsky et al. (2015). In $4.1, we show that CPP achieves higher clustering accuracy than deep clustering methods. $4.2, we show that CPP learns more structured representation than CLIP. We further challenge CPP deal with large uncurated datasets, where the number of clusters is unknown, and the clusters are not annotated with captions. We report our findings on MS-COCO (Lin et al., 2014) and LAION-Aestheties (Schuhmann et al., 2022) in $4.3 and WikiArt (Saleh & Elgammal, 2015) in $4.4. The full training details are left Appendix C.
4.1 COMPARISON WITH DEEP CLUSTERING METHODS
Methods. We compare CPP with state-of-the-arr deep clustering methods. These methods follow similar two-stage approach as CPP, in that they leverage pre-trained self-supervised or language- supervised network initialization, and fine-tune the network with some clustering loss. In this subsection, all methods are given the ground-truth number of clusters. We refer the readers to Appendix A.2 for full comparison with many more methods and their backbone architectures.
Metries. For each method, given its estimated clusters, we compare them with the ground-truth ones to compute the clustering accuracy (ACC) and normalized mutual information (NMI).
CIFAR-
CIFAR-20
CIFAR-I00
ImageNet- 1k
Method
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
MLC (Ding al., 2023)
86.3
76.3
52.2
54.6
49.4
68.3
SCAN (Van Gansbeke al., 2020)
88.3
79.7
50.7
48.6
34.3
55.7
39.9
IMC-SWAV (Ntelemis et al., 2022)
89.7
81.8
51.9
52.7
45.1
67.5
TEMI (Adaloglou et al., 2023)
96.9
92.6
61.8
64.5
79.9
64.0
CPp*
97.4
93.6
64.2
72.5
74.0
81.8
66.2
86.8
Table 1: Comparison with state-of-the-art deep clustering models. asterisk (*) uses pre-training from CLIP.
Methods marked with an
Results. From Table 1, it is evident that CPP outperforms alternatives in terms of ACC and NMI across all datasets; the same true when CPP is compared with more methods in Appendix A.2. We observe that methods incorporating pre-training from external data have displayed significant improvement compared to their predecessors. This underlines the importance of integrating pre- trained models for image clustering. When compared with TEMI (Adaloglou al., 2023), which also employs external pre-training data, CPP not only attains superior training accuracy but also drastically reduces training epochs: CPP converges within a maximum of 50 epochs, whereas previous methods typically require more than 100. This endows CPP with the capacity to efficiently and effectively scale up to even larger datasets in this era of treitrained models. We will delve deeper into this in $4.3.
4.2 COMPARISON FEATURES LEARNED CLIP AND CPP
As CPP uses CLIP features initialization, it is natural to ask: does CPP really improve the representation of CLIP? Below we answer this question in the affirmative: compared to CLIP features, CPP-learned features are i) qualitatively better structured. ie.. being within-cluster diverse and between-cluster discriminative, and ii)more amenable to tasks such as clustering and image-to-ing tomi search.

---

Published as a conference paper at ICLR 2024
CIFAR-10
CIFAR-100
ImageNet-Il
CLIP
CLIP
CPP
Figure 2: Structured representations learned by CPP. Left: An example of image-to-image search on ImageNet, using representations provided by CLIP (Top) and CPP (Bottom). Right: Cosine similarity 2 2| visualization. Clear block-iiagonal structures emerge in CPP-lcarned representations (Bottom), while the ones learned by CLIP show strong sample-wist wise correlation (Top).
Singular Ours CLIP
Singular
Singular
Compotwats
Comprsents
Components
(a) All Samples
(b) Within Cluster (CLIP)
(c) Within Cluster (CPP)
Figure 3: Normalized singular values of CIFAR-100 features. Left: full dataset features. Middle: cluster-wise features from CLIP, membership given by KMeans. Right: cluster-wise features from CPP, membership given by spectral clustering upon membership matrix.
Visualizing Cosine-Similarrtt of Features. CPP aims at learning a union- of wrthogonal-subsppe structure optimizing the MLC (Ding et al., 2023) objective. To validate the effect of the training stage, we visualize the cosine similarity matrix of learned representations given by | 2| in the same manner of prior literature (Ma et al., 2007b). We observe that CPP's learned representations yield block-diagonal matrices in CIFAR- 10/-0-1001100 and ImageNet-1k kk In this setting, each block corresponds to one cluster, which qualitatively validates the effectiveness of CPP, i.c. representations within the same cluster are relatively similar, while those from different clusters are relatively dissimilar. In contrast, also in Figure 2, off- shell CLIP representations do not exhibit similar structures.
Visualizing Spectrum of Features by Cluster. To better analyze the space structure of representa- tions, we conduct PCA the learned representations of CPP and CLIP. Specifically, apply SVD on two groups of representations and visualize the normalized singular values. From Figure 3 (a), we observe that CLIP learns highly-compacted space with features mainly spread around one direction (blue curve). In contrast, space learned by CPP shows to be more diverse with multiple dimensions (orange curve). Similar patterns appear in cluster-wis wise representations Fiiuure (b)(c)) Our results suggest that CPP learns holistically more diverse structures compared to the original CLIP.
Measuring Clustering Performance. Besides visualization, we evaluate the effectiveness of CPP through clustering tasks. As demonstrated in prior literature, CLIP (Radford al., 2021) has already learned discriminative representations that can be employed for clustering. To assess the quality of these clusters without the application of CPP, and to validate the necessity of CPP, we apply KMeans, subspace clustering methods (EnSC (You et al., 2016a) and SSC-OMP (You et al., 2016b)), and spectral clustering on the representations learned through CLIP, comparing their results with CPP. From Table 2, observe that CPP has achieved significant improvement in cluster accuracy across all four datasets.
Visualizing Image-to-Image Search Performance. We further demonstrate the effect structured representations on image-to-image search tasks. In this setting, we first establish image repository
7

---

Published as a conference paper at ICLR 2024
CIFAR-10 10
CIFAR-20
CIFAR-100
ImageNct- Ik
Method
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
KMcans
83.5
84.0
47.3
51.3
52.3
67.7
49.2
81.3
EnSC
85.8
89.2
61.6
69.3
66.6
77.1
56.8
83.7
SSC-OMP
85,4
84.6
60.9
65.3
64.6
72.8
49.6
80.5
Spectral Clustering
73.6
75.2
52.4
56.2
67.2
76.1
55.8
83.4
CPP
97.4
93.6
64.2
72.5
74.0
81.8
66.2
86.8
Table Clustering accuracy and normalized mutual information of classic clustering methods applied to CLIP features (Top) and the proposed CPP using CLIP as pre-trained features (Bottom). CLIP's ViT-L/14 is used as the backbone. See Appendix D for hyperparameter settings.
Stmbber Clumm
Nanaber
of
(a) CIFAR-10 (10)
(b) CIFAR-100 (20)
(c) MS-COCO (150)
(d) LAION-AeS. (300)
Figure 4: Model selection for clustering without knowing the number of clusters using Algorithm 1.
For each dataset, the elbow point of the curve indicates the optimal number (in parenthesis) of clusters.
The model selection is done efficiently without any retraining. See Appendix F for more results.
comprised set of images from the dataset, and then compute their respective representations. yielding our feature set. To search for a target image, we compute its representation and identify the 64 images from our feature set that have the closest cosine similarity the target. We display one example in Figure 2, with additional results provided in Appendix G. Our results suggest that CPP representation entails a more comprehensive understanding, as demonstrated by the shampoo bottle example. Image-to-image search based on CPP recognizes the semantic meaning of shampoo bottle as a daily hygiene product, while the search based on CLIP predominantly returns images of cylindrical objects.
4.3 CLUSTERING AND CAPTIONING ON LARGE UNCURATED IMAGE DATASETS
Recall that CPP is a complete clustering pipeline on large-scale datasets with specific designs for measuring the number of clusters ($3.3), and ii) captioning learned clusters ($3.4). In this subsection, we verify the designs on standard datasets CIFAR- 10-RR-00-110 100 and study their effectiveness on datasets with larger scales such as MS-COCO and LAION-Aesthetics.
Finding Optimal Number of Clusters. We apply Algorithm 1 on the feature head representations after the training stage of CPP and report the results in Figure 4. Algorithm 1 identifies cluster numbers of 10 for CIFAR-10 and 20 for CIFAR-100, which aligns the finding in prior works (Van Gansbeke al., 2020; Niu et al., 2022). Figure 4 also shows that the optimal number cluster for LAION-Aesthetic and MS-COCO are 300 and 150 respectively. We will use these two numbers to cluster LAION-Aesthetie and MS-COCO in the next paragraph.
Appendix H on LAION-Acstherr and MS- COCO. Notably, ideal text candidates for the stage would be static high-quality vocabulary. In our setting, we utilize a mixture of LLM--eenratte with details in Appendix H. Visualization of learned clusters and the corresponding captions can be found in Figure 5 and Appendix H. These results suggest that CPP's learned clusters can be summarized via consistent semantic meanings. CPP shows promising qualitative performance on LAION-Acstheties, even if it's large, diverse and imbalanced (see Figure 7 for details).
Captioning Clusters. We apply Algorithm
labels and ImageNet- labels for efficiency,
4.4 WIKIART: CASE STUDY IMAGE CLUSTERING IN THE AGE OF TRAIIIIGG
In the age of pre-training, now have easy access general visual representations that span broad spectrum of distributions. As a result, it is no longer necessary to undertake dataset-specific pre-training for tasks like image clustering. Building on this trend, CPP offers efficient solution
8

---

Published as a conference paper at ICLR 2024
Windsor tie"
"church"
"surfboard"
'valley'
"comic"
*xmss tree"
Figure 5: Examples of cluster captioning on MS-COCO (Left) and LAION-Aesthetics (Right). More visualization can be found in Appendix H.
Dataset:
WIKIART VISUAL ART INCYCIOFIOIA
Optimal # of clusters: 20
Coding Length
Clusters
Analyticai cubist Painting
'Mrittmme Pointing"
Text candidates:
LLM
"Give
common objects'
"Give me 120 words of ort styles"
"Giv me 100 words describing the content of pointings"
"Ukiyo-e Pointing"
Fin de siècle Pointing"
Figure 6: Example of full CPP pipeline on WikiArt. CPP clusters art pieces based on artistic and scenic styles and assigns them with meaningful captions from ALM--eeneated text candidates.
for clustering and captioning images from some rare distributions. We demonstrate this through a case study on WikiArt, collection of art pieces from Wikipedia. Using CPP, we determined that 20 clusters optimally represent the dataset. We then captioned each cluster with tailored description. Our findings reveal that CPP successfully categorized WikiArt into 20 distinct painting styles, including both scenic types like maritime and specific artistic movements such analytical cubism. This capability makes CPP highly valuable for further analysis and application.
5 CONCLUSION AND DISCUSSION
This paper proposes a pipeline to do representation learning and clustering on large-scale datasets. The pipeline, dubbed CPP, takes advantage of CLIP pre-trained model. CPP achieves state-of-the-oe clustering performance on CIFAR-10, -20, -100, and ImageNet-Ik. Further, when the number of clusters unknown, we give a mechanism for CPP that estimates the optimal number of clusters, without any costly retraining of deep networks. Finally, CPP refines the CLIP model, by giving more accurate clustering, as well as more diverse and discriminative representation, allowing better image-to-imane search.
As for future work, we find it fascinating to explore the continual learning setting since real-world big data come only in streaming fashion with new modes continuously showing up. It is also of interest to leam diverse and discriminative representation and to cluster data with both text and image input.
ACKNOWLEDGEMENT
This work was partially supported by the Northrop Grumman Mission Systems Research in Appli- cations for Learning Machines (REALM) initiative, DARPA Grant HR001 12020010, NSF grant 1704458, and ONR MURI 503405-78051. Yi Ma acknowledges support from the joint Simons Foundation- NSF DMS grant #2031899, the ONR grant N00014-22-1-2102. TBSI, and support from the University of Hong Kong.

---

Published as a conference paper at ICLR 2024
REFERENCES
Nikolas Adalogiou, Felix Michels, Hamza Kalisch, and Markus Kollmann. Exploring the limits of deep image clustering using pretrained models. arXiv preprint arXiv:2303.17886 2023.
David Arthur and Sergei Vassilvitskii. k-means++: The advantages of careful seeding. In the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms. Society for Industrial and Applied Mathematics, June 2006.
Christina Back, Ziyang Wu, Kwan Ho Ryan Chan, Tianjiao Ding, Yi Ma, and Benjamin D Haeffele. Efficient maximal coding rate reduction by variational forms. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 500-508, 508, 2022.
Bahman Bahmani, Benjamin Moseley, Andrea Vattani, Ravi Kumar, and Sergei Vassilvitskii. Scalable K- M.ans++. Proceedings VLDB Endowment, 5(7), March 2012.
Hangbo Bao, Li Dong. Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.086 08254, 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: seff-supervised learning. arXiv preprint arXiv:2105. 04490, 2021.
regularization for
Yoshua Bengio, Aaron Courville, and Paseal Vincent. Representation learning: a review and new perspectives. IEEE Trans, Pattern Anal. Mach. Intell., 3588))1798-1888 1828, August 2013. ISSN 0162-8828, doi: 10.1109/ TPAMI.2013.50, URL http: / dx doi wrg/10........o..... /00 1109/ TEAMI. TFFAM.2211 2013.50.
Paul Bradley, Olvi Mangasarian, and W Street. Clustering via concave minimization. In Advances in neural information processing systems. 1996.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin.
Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pp. 9650 9660, 2021.
Kwan Ho Ryan Chan. Yaodong Yu, Chong You. Haozhi Qi. John Wright,, and Yi Ma. Deep networks from the principle of rate reduction. arXiv preprint arXiv:2010.14765, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Niv Cohen and Yedid Hoshen. Dataset summarization by k principal concepts. arXiv preprint arXiv:2104.0395.. 2021.
Xili Dai, Shengbang Tong, Mingyang Li, Ziyang Wu, Michael Psenka, Kwan Ho Ryan Chan, Pengyuan Zhai, Yandong Yu, Xiaojun Yuan, Heung- Yeung Shum, et al. Ctrl: Closed-loop transcription to an ldr via minimaxing rate reduction. Entropy, 24(4):456, 2022.
Aniket Anand Deshmukh, Jayanth Reddy Regatti, Eren Manavoglu, and Urun Dogan. Representation learning for clustering via building consensus. Springer Machine Learning Journal arXiv preprint arXiv:2105.0 2021.
Tianjiao Ding, Derek Lim, Rene Vidal, and Benjamin D Hacffelc. Understanding doubly stochastic clustering. In the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5153-5165. PMLR, 2022.
Tianjiao Ding, Shenghang Tong, Kwan Ho Ryan Chan, Xili Dai, Yi Ma, and Benjamin D. Haeffele. Unsupervised manifold linearizing and clustering. In Proceedings of the IEEE/CV International Conference on Computer Vision (ICCV), pp. 5450-5461, October 2023.
Tianjiao Ding, Liangzu Peng, and René Vidal. HARD: Hyperplane ARrangement Descent. In Conference on Parsimony and Learning, pp. 134-158. PMLR, January 2024. URL htt.httas//prrccediingg mlr. press / v234 /ddng/inaaa...m html. ISSN: 2640-3498.
Tianyu Ding, Zhihui Zhu, Manolis Tsakiris, René Vidal, and Daniel Robinson. Dual Principal Component Pursuit for lcarning a union of hyperplanes: Theory and algorithms. 2021. URL https: /www. t iinyuding.
Conm///peessAAstATS221--1 .dd..
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov. Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. arXiv preprint 20 1999, 2020.
10

---

Published as a conference paper at ICLR 2024
Marvin Eisenberger, Aysim Toker, Laura Leal-Taixé, Florian Bernard, and Daniel Cremers. A unified framework for implicit sinkhorn differentiation. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 309-518,70 2022.
Ehsan Elhamifar René Vidal. Sparse manifold elustering and embedding. Advances neural information processing systems, 24, 2011.
Ehsan Elhamifar and Rene Vidal. Sparse subspace clustering: Algorithm, theory, and applications. IEEE transactions on pattern analysis and machine intelligence, 35(11):2765 2781, 2013.
Edward Forgey. Cluster analysis multivariate data: Efficiency vs. interpretability classification. Biometrics, 1965.
Ccan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires. Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33: 21271-21284, 2020.
Xiaotian Han, Zhimeng Jiang, Ninghao Liu, Qingquan Song, Jundong Li, and Xia Hu. Geometric graph representation learning via maximizing rate reduction. In Proceedings of the ACM Web Conference 2022, pp. 1226 1237, 2022.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 9738, 2020.
Kaiming He. Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Reinhard Heckel and Helmut Böleskei. Robust subspace clustering via thresholding. IEEE transactions on information theory, 61(11):6320 6342, 2015.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongscok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https:: /wwiiiiooo// org/ 10/ 5281/2211// 02228/2200 zenodo. 5143773.
R C Jancey. Multidimensional group analysis. Australian Journal of Botany, 14:127-130, 1966.
Yunji Kim and Jung- Woo Ha. Contrastive fine-grained class clustering via generative adversarial networks. arXiv preprint arXiv:2112. 14971, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exil/nnnist/ 1998.
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Jocy Tianyi Zhou, and Xi Peng. Contrastive clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8547-8555, 2021.
Zengyi Li, Yubei Chen, Yann LeCun, and Friedrich Sommer. Neural manifold clustering and embedding. arXiv preprint arXi::201..10000 2022.
Derek Lim, René Vidal, and Benjamin D Hacffele. Doubly stochastic subspace clustering. arXiv Jcs.LGI, November 2020.
Tsung-Yl Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Caawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Guangean Liu, Zhouchen Lin, Shuicheng Yan, Ju Sun, Yong Yu, and Yi Ma. Robust recovery of subspace structures by low-rank representation. IEEE transactions pattern analysis and machine intelligence, 35(1): 171-184, January 2013.
Stuart Lloyd. Least squares quantization in PCM. Technical report, Bell Laboratories, 1957.
Can-Yi Lu, Hai Min, Zhong- Qiu Zhao, Lin Zhu, De-Shaang Huang, and Shuicheng Yan. Robust and efficient subspace segmentation via least squares regression. In European conference computer vision, pp. 347-360. Springer, 2012.
11

---

Published as a conference paper at ICLR 2024
Yi Ma, Harm Derksen, Wei Hong. and John Wright. Segmentation of multivariate mixed data via lossy data coding and compression. IEEE transactions on pattern analysis and machine intelligence, 29(9):1546 1562, 2007a.
Yi Ma, Harm Derksen, Wei Hong. and John Wright. Segmentation of multivariate mixed data via lossy data coding and compression. IEEE trunsactions on puttern analysis and machine intelligenne,, 2999):1446-1562 2007b. ISSN 0162-8828. doi: 10.1109TPPAM.2200.085. URL http:://xx..o..oo.p....pp.. wrg/10 1109/ TPAMI , 2007 1085.
James B McQueen. Some methods for classification and analysis of multivariate observations, Fifth Berkeley Symposium on Mathematical Statistics and Probability, pp. 281-297, 1967.
Sameer A Nene, Shree K Nayar, Hiroshi Murase, Columbia object image library (coil-20) 20). 1996.
Chuang Niu and Ge Wang. Spice: Semantic pscudo-labeling for image clustering, 2021.
Chuang Niu, Hongming Shan, and Ge Wang. Spice: Semantic pseudo-labeling for image clustering. IEEE Trunsactions on Image Processing, 31:7264-7277 2022.
Foivos Ntelemis, Yaochu Jin, and Spencer A Thomas. Information maximization clustering via multi-view self-tabelling. Knowledge-Based Systems, 250: 109042, 2022.
Maxime Oquab, Timothée Dareet, Théo Moutakanni, Huy Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Danicl Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint X31::2304.07193 2023.
Vardan Papyan, X Han, and David L Donoho. Prevalence neural collapse during the terminal phase of deep leaminn training. Proceedings of the National Academy of Sciences of the United States of America, 117(40); 24652-2466 24663, October 2020.
Sungwon Park, Sungwon Han, Sundong Kim, Danu Kim, Sungkyu Park, Seunghoon Hong, and Meeyoung Cha. Improving unsupervised image clustering with robust learning. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 12278-12287. 2021.
Vishal M Patel and Rene Vidal. Kernel sparse subspace clustering. In 2014 IEEE International Conference on Image Processing (ICIP). IEEE, October 2014.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-888 8763. PMLR, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenct large scale visual recognition challenge. International journal of computer vision, 115:211-252, 2015.
Mohammadreza Sadeghi, Hadi Hojjati, and Narges Armanfard. C3: Cross-instance guided contrastive clustering. arXiv preprint arXiv:22/1.001 1,07136, 2022.
Babak Saleh Ahmed Elgammal. Large-scale classification fine-art paintings: Learning the right metric on the right feature. arXiv preprint 150iv::505.000 008500 2015.
Michael Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Sinkformers:
Transformers with doubly stochastic attention. In International Conference on Artificial Intelligence and Statistics, October 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman. Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, training next generation image-text models. arXiv preprint arXiv:2210.084000 2022.
Laion An open large-scale dataset for
Souvenir and R Pless. Manifold clustering. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, volume 1, pp. 648-653 Vol. 1, October 2005.
Shengbang Tong, Xili Dai, Yubei Chen, Mingyang Li, Zengyi Li, Brent Yi, Yann LeCun, and Yi Ma. Unsuper- vised learning of structured representations via closed-loop transcription, arXiv preprint arXiv:2210.16782. 2022a.
Shengbang Tong, Xili Dai, Ziyang Wu. Mingyang Li, Brent Yi, and Yi Ma. Incremental learning of structured memory via closed-loop transcription. arXiv preprint (XXiv:22220054111 2022b.
Shengbang Tong, Yubei Chen, Yi Ma, and Yann Lecun. Emp-ssl: Towards self-supervised learning in one training epoch, arXiv preprint 2x4142234..33977, 2023.
12

---

Published as a conference paper at ICLR 2024
Manolis C C'Tsakiris and René Vidal. Hyperplane Clustering via Dual Principal Component Pursuit. In Doina Precup and Yec Whye Teh (eds.), Proceedings of Machine Learning Research, volume 70, pp. 3472-3481. PMLR, 2017. URL https: : /pppcceedingg wwtpppppwaa////taaaaaa.aa.ama Abbreviation: Proceedings of Machine Leamingg Research.
html. Journal
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Luc Van Gool, SCAN: Learning to classify images without labels. In European conference on computer vision. Springer, 2020.
Ulrike von Luxburg. A tutorial on spectral clustering. Statistics and computing, 17(4):395 416, 2007.
Kouta Nakata Yaling Tao, Kentaro Takagi. Clustering-rriente representation learning via instance discrimination and feature decorrelation. Proceedings of ICLR 2021, 2021. URL https: /ttp://///peeeeeeeeeeeeee net forum? id=e 2NDM7wkEY.
Chong You, Chun Guang Li, Daniel P Robinson, and Rene Vidal. Oracle based active set algorithm for scalable elastic net subspace clustering. In the IEEE conference computer vision and pattern recognition, pp. 3928-3937, 2016a.
Chong You, Daniel Robinson, and René Vidal. Scalable sparse subspace clustering by orthogonal matching pursuit. In JEEE Conference on Computer Vision and Pattern Recognition, June 2016b.
Yaodong Yu, Kwan Ho Ryan Chan, Chong You. Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in Neural Information Processing Systems, 33:9422-9453 2020.
Li Yunfan, Yang Mouxing, Peng Dezhong. Li Taihao, Huang Jiantao, and Peng Xi. Twin contrastive learning for online clustering. International Journal of Computer Vision, 2022.
Jinghao Zhou, Chen Wei, Huiyu Wang. Wci Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832, 2021.
Xingzhi Zhou and Nevin L Zhang, Deep clustering with features from self-supervised pretraining. arXiv preprint wxXiv:207.133366 2022.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li. Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. Ady. Neural Inf. Process. Syst., 34:29820 298344, 2021. ISSN 1049-5258.
13

---

Published as a conference paper at ICLR 2024
A ADDITIONAL QUANTITATIVE RESULTS AND CLARIFICATIONS
A.1 COMPARISON OF DIFFERENT PRE-TRAINED VISUAL MODELS
In our proposed pipeline (Figure 1), CLIP's image encoder is not the only candidate of pre-trained models. To further justify the generalizability of CPP, we evaluate the clustering performance of CPP leveraging visual representations from MAE (He al., 2022) and DINO (Caron al., 2021). For each model, CPP significantly improved the clustering performance and NMI score when compared with KMeans. Empirically,
find that both DINO and CLIP give good initializations for CPP: while MAE features are not suitable for image clustering, i.c. 12.3% accuracy via KMeans and 24.2% accuracy via CPP pipeline ImageNet- 1k, which is aligned with the discussion by Oquab et al. (2023) that MAE as a backbone are great for finetuning with labels, while less competent directly learn discriminative representation.
CPP Pipeline
K.Means
pre-train
Backbone
ACC
NMI
ACC
NMI
MAE
VIT L/16
24.2
69.8
12.3
60.6
DINO
VIT B/16
59.0
84.5
53.5
81.6
DINO
VIT B/8
61.9
86.8
56.0
85.2
CLIP
VIT L/14
66.2
86.8
49.2
81.3
Table 3: Benchmarking various models on ImageNet-1k with CPP and KMeans. MAE and DINO models are pre-trie peetrrined on ImageNet-1k, kk; CLIP model is pre-trained on external data from their official implementation.
A.2 COMPARISON WITH MORE DEEP CLUSTERING METHODS
In addition to Table 1, we list other state-of-the deep clustering methods and report the quantitative performance. The primary purpose of showing Table and Table 4 is not compete with or surpass other deep clustering methods. Instead, we aim to probe the boundaries of image clustering. We clearly list the backbones for reference.
CIFAR-10
CIFAR-20
CIFAR-100
ImageNet-Ik
Method
Backbone
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
MLC (Ding et al., 2023)
ResNet-18
86.3
76.3
52.2
54.6
49.4
68.3
SCAN (Van Gansbeke et al., 2020)
ResNet-18 18
88.3
79.7
50.7
48.6
34.3
55.7
39.9
IDFD (Yaling Tao, 2021)
ResNet-18
81.5
71.1
42.5
42.6
IMC-SWAN (Ntelemis al., 2022)
ResNet-18
45.1
67.5
RUC+SCAN (Park et al., 2021)
ResNet-18 18
990.3
$4.3
SPICE (Niu & Wang. 2021)
ResNet-34
91.7
85.8
58.4
58.3
NMCE (L.i et al., 2022)
ResNet-34
88.7
81.9
53.1
52.4
TCL (Yunfan et al., 2022)
ResNet-34
88.7
81.9
$3.1
52.9
(Sadeghi
2022)
ResNet-34
CC (1.i et al., 2021)
ResNet-34
79.0
70.5
42.9
43.1
ConCURL (Deshmukh al., 2021)
ResNet-50 50
84.6
76.2
47.9
46.8
Single-Noun Prior (Cohen & Hoshen, 2021)
V/T-B/32
93.4
85.9
48.4
51.5
TEMP (Adaloglou et al., 2023)
VIT L/14
96.9
92.6
61.8
64.5
73.7
79.9
64.00
CPP
VIT 1/14
97.4
93.6
64.2
72.5
74.0
81.8
66.2
86.8
Table 4: Comparison with more state-of-the-art deep clustering models. Methods marked with (*) pre-training from OpenAl CLIP, method use (#) pre-training from OpenCLIP LAION-2B.
B ABLATION STUDY
In this subsection, we conduct ablation studies on 2 components of CPP: pre-train datasets and diversified initialization.
The main results of our proposed methods leverage pre-training from OpenAl CLIP (Radford et al., 2021). To validate the generalizability of CPP, we additionally conduct experiments using ViT-L/14 from OpenCLIP (Ilharco et al., 2021), which is pre-trained on LAION-2B (Schuhmann et al., 2022). We report the results Table 5.
14

---

ReLU after Published as a conference paper at ICLR 2024
CIFAR-10
CIFAR-20
CIFAR-100
ImageNet-Ie Ik
Backbone
Pretraining
CPP
KMeans
CPP
KMcans
CPP
KMcans
CPP
KMcans
ViT-L/14"
LAION-2B
96.5
94.0
70.2
58.7
76.7
62.3
66.8
50.2
ViT-L/14
OpenAl
97.4
83.5
64.2
47.3
74.0
52.3
66.2
49.2
Table 5: Ablation study on pre-training. We report the clustering accuracy and observe that CPP consistently outperforms KMeans on two different pretrained models. (*) leverages open source pretraining from OpenCLIP, (#) leverages pretraining from official CLIP.
We then conduct ablation study the contribution of diversified initialization (described in Section 3.2). We diversify the representation via optimizing the objective equation (4). This procedure improves the clustering performance on various datasets with results reported in Table 6.
CIFAR-10
CIFAR-20
CIFAR-100
ImageNet-1k
Initialization
ACC
NMI
ACC
NMI
ACC
NMI
ACC
NMI
Random
87.6
90.6
57.4
69.9
67.9
78.3
63.7
84.7
Diversified
97.4
93.6
64.2
72.5
74.0
81.8
66.2
86.8
Table 6: Ablation study on the contribution of diversified initialization. We randomly initialize pre- feature layer, cluster head and feature head (Top) and compare the performance with the diversified initialization (Bottom) in equation (4).
C TRAINING DETAILS
This section provides the training details network architecture, datasets, optimization and hyperpa- rameters.
Datasets, CIFAR contains 50, 000 training and 10, 000 test images, which are divided evenly into 10, 20 or 100 ground-truie truth classes, which we refer as CIFAR- 10, -20 -00; note that the classes of CIFAR-20 are given by merging those of CIFAR- 100. ImageNet incorporates around 1.2 million training images and 100, 000 test images, spread across 1, 0,0 classes, called ImageNet-1k kk. We process data in manner identical to that used in CLIP (Radford et al., 2021), which involves resizing and center cropping images to dimensions of 224 x 224.
Network Architecture. We VIT L/14 model (Dosovitskiy et al., 2020) pre-trained via CLIP (Radford et al., 2021), with checkpoint from OpenAl', As shown in Figure 1, we freeze the backbone during training and add pre-feature layer composed of -or--aattooo--LL-----
the backbone. For feature head and cluster head, we use Linear layer with mapping from the hidden dimension to feature dimension respectively. Unified architecture is applied on all the experiments across different datasets except adjusted hidden dimension and feature dimension d. Finally, to learn a ideal representation that spans union of orthogonal subspaces as in $3.1, note that the feature dimension d should be larger than or equal to the expected number of clusters. We leave more details in the Appendix C.
Details of Added Layers. For all datasets, utilize simple architecture composed of three parts: pre-feature layer, feature head and cluster head. Pre-feature layer has a structure with Linear- -ew-ooo----LLLLLnaaaa
with detailed setting in Table 7a. For feature head and cluster head, we use a linear layer respectively as is described in Table 7b.
Dimensions. Dimension and dradden for each dataset are provided in Table 8a, note that d should be larger than or equal to the expected number of clusters to satisfy the orthogonal subspace assumption.
Optimizers. We specify two independent optimizers to simultaneously optimize the MLC objective with detailed parameters in Table 8b.
httpp::///t wwwpppoo..oooaaaaaaaaaaaaa /CLIP
15

---

Published as a conference paper at ICLR 2024
(a) Pre-feature layer
Linear: R
R dhdiddnn
(b) Feature head and cluster head
BatchNorm ld (dhidden)
Feature head
Linear: IR Ihhidden _
IR
ReLU
Cluster head
Linear: IRdnzadon
Rd
Linear: Rduidden Rdaidden
ReLU
Table 7: Network Architecture
Sinkhorn Distance. The doubly stochastic membership matrix II is computed by a sinkhorn distance projection on C C, where the parameters y regulate the sparsity of the membership matrix as is described in Section 3.1. Details with this parameter are recorded in Table 8c.
Optimization. As describe in $3.2, we first warmup our network by 1-2 epochs by training R(2; e) alone, then simultaneously optimize both feature head and cluster head using (MLC). For both the feature head and cluster head, we train with SGD optimizer, learning rate set to 0.0001, momentum set to 0.9 and weight decay set to 0.0001 and 0.005 respectively.
Initialization and Training Epochs. Details in initialization (simply optimize R(Z; E)) epochs and total training epochs are recorded in Table 8d.
(a) Model Parameters. We adjust dimension of learned features for the diffcrent expected numbers of clusters.
(b) Optimizers, We optimize objective function using SGD optimizer with unified parameters as
Datasets/Parameterr
dhsiddnn
below:
CIFAR-10
128
4096
CIFAR-20
128
4096
Optimizers
Type
Ir
wd
momentum
CIFAR-100 100
128
4096
Feature
SGD
0.0001
0.0001
0.9
ImageNet-Ik 1k
1024
2048
Cluster
SGD
0.0001
0.005
0.9
MS-COCO
128
4096
LAION-Acsthetics
1024
2048
(d) Initialization epoch, total training epoch, batch size. Batch size doesn't affect too much the performance. All experiments be conducted on a single A100.
Sinkhorn Distance Parameters while Training
Datasets
7
Iter
Datasets
epoch
epochrota!
bs
CIFAR-10
0.175
CIFAR-10
1024
CIFAR-20
0.13
5
CIFAR-20
1
15
1024
CIFAR- 100
0.1
5
CIFAR-100
1
50
1500
ImageNet-Ik lk
0.12
5
ImageNet-IK
2
20
1024
COCO
0.12
5
COCO
1
1200
LAION
0.09
5
LAION
2
20
1024
Table 8: Core hyperparameters selected experiments.
16

---

is the weight Published as a conference paper at ICLR 2024
D SUBSPACE CLUSTERING PARAMETERS
We conduct subspace clustering methods on CLIP features and report the highest accuracy after searching for optimal parameters.
EnSC. Both EnSC and SSC-OMP" estimate membership matrix via solving some convex op- timizations that depend only on CLIP features, and then run spectral clustering on the resulting membership. For EnSC, we use the efficient active-set solvers from (You et al., 2016a) to solve the convex optimization. EnSC has two parameters 7, T. Roughly speaking, T E [0, balances between an & and an & penalty on the membership, with larger giving sparser affinity; of the data fidelity error, aside from the regularizing term.
SSC-OMP. (kmars E) We use the OMP solver for SSC (You al., 2016b). Amaz is the maximum number of non-zero entries of each row of the membership, while controls the allowed data fidelity error.
Spectral Clustering. 7 denotes the parameter for sink horn distance projection, which is the same as the one mentioned in previous sections. For given batch of CLIP's feature C' E pd we first normalize each feature vector and then do inner production plus sink horn distance projection, i.c. HeLI = projn,"(C')) (C)CC)). We then do spectral clustering on this membership matrix HCLIP
Table 9: Parameter search with the following parameters for EnSC, SSC-OMP and spectral clustering. We report the highest performance on Table 2.
Datasets
Parameters
EnSC
[1, 5, 10, 50, 100), [0.9, 0.95, 1.0] kmaz E [3, 10], c E [le
SSC-OMP
4, le 5, 1e 6, 1e 7]
Spectral Clustering
Y [0.2, 0.18, 0.16, 0.1, 0.09, 0.08, 0.07, 0.06]
E EVALUATION ON IMBALANCED DATASETS
We evaluate CPP on imbalanced CIFAR-10 and imbalanced CIFAR-100, where images of odd classes (i.e. 1, ...) are reduced to half of the original. Additionally, some large, uncurated datasets, like -estheti,,, exhibit natural imbalance. To demonstrate CPP's proficiency in identifying minority groups, we also present visualizations of clusters with fewer members in Figure
F MORE RESULTS ON OPTIMAL NUMBER OF CLUSTERS
We additionally measure the coding length for ImageNet as shown in Figure 8. For all datasets, compute the coding length with e2 0.1, which is consistent with the one in MLC objective function.
G IMAGE-TO-IMANE SEARCH
Pipeline. Figure 9 demonstrates the pipeline of image-to-image search. In practice, the image repository is composed of 1.2M images from ImageNet's training split while the target image is randomly picked from ImageNet's validation split. We search the images in the repository via measuring the Euclidean distance and plot the 64 most similar images.
Results. Here, we provide 10 more image-to-image search results in Figure 10. We observe from these results that CPP learned better representation that facilitates image-to- to-image search.
Thh implementations provided by the authors https://github.co htt.:/u...oo/o///ogggeeeee wetppacceecluserreo
17

---

Published as a conference paper at ICLR 2024
(a) Imb. CIFAR-10
(b) LAION-Acsthetic Cluster (i) (c)
Cluster (ii)
Figure 7: Performance of CPP on imbalanced datasets. CPP achieved 97.3% and 71.3% clustering
accuracy Imb. CIFAR-10 and Imb. CIFAR-100 respectively; The confusion matrix demonstrates the prediction results on Imb. CIFAR-10 validation set (Lef).. LAION-Aesthetii is also a natural imbalanced dataset, where two clusters with few members visualized, each composed of 0.73% and 0.47% images respectively from the dataset (clustering on 30k random samples from LAION- Aesthetic) (Middle, Right)
1.00
Colling Dibe
Number of Clestera
Figure 8; ImageNet (200)
H MORE RESULTS ON CLUSTERING AND LABELLING WITH TEXT
Text Candidates Selection. Ideally, an open vocabulary source with tons of highly reliable labels best suits our needs. However, text candidates of this quality and scale usually hard be obtained. In practice, In practice, we leverage powerful I.LMs, such as GPT-4, to generate text candidates as an economical substitute. More specifically, we employ prompts like "generate 2000 names of real-world objects/creatures, generate 100 words of art styles, give me 100 words describning the content of paintings..." " during the generation process. To guarantee the diversity and reliability, we furtherly mix them up with 1000 ImageNet class labels to construct the final 3000+ text candidates. It is noteworthy mention that we did not utilize any label information from MS-COCO, LAION-Aesthere or WikiArt.
Pipeline. We introduce cluster-labeling algorithm after obtain well-trained CPP. First, do spectral clustering upon the membership matrix given by CPP and get clusters images. Then, for images in each cluster, conduct weighted voting for the common labels. The voting algorithm is described in Algorithm 2.
Results. In this section, we visualize more cluster-capiioning results for datasets including CIFAR- 100, ImageNet-Ik, COCO and LAION-Aestheties. We also follow the optimal number of clusters measured in Section 4.3 for each dataset.
18

---

Published as a conference paper at ICLR 2024
Euclidean Distance
Pre-feature layer
Image Repository
Backbone
Feature head
Targel Feature
Top 64
Target Image
Figure 9: Image-to-Image Search Pipeline.
2
123
Figure 10: Searching for similar images in ImageNet's training split. Left: Target image from validation split in ImageNet.Middee searched images via CPP's representation; Right: searched images via CLIP's representation.
19

---

Published as a conference paper at ICLR 2024
Algorithm Captioning one cluster Input:
Images from learned cluster X' E IR IR-Nx33222x221, MM text candidates, 0 initialized voting result vector V E RM
imm f- CLIP: encode images(
Ztxt
CLIP: encode texts (text candidates)
For i f- 1, N: N:
Scores4labels +- Cosine Similarity for for(2ing Zimg Btxt)
(8)
Valid Score +- Score41ahels(top5]
(9)
V V Valid Score
(10)
Output: Caption for this cluster: text candidates(argmax V]
Witecla rike selecion
Servea bear beaas, Uneun wimter 25.W
pollo
red aet brido 16.1
(a) tricycle.
(b) brown bear.
(c) goblet
(d) steel arch bridge
(c) crib
(f) baby
g streetcar.
(h) poppy
$3.25
spute 33
robils kilk STEEEOL Orcnnserce Vabas cored demoddrry Oumelie dreae dartar 90.0
sweet pepper
agaric
whale.
Arabian camel.
(m) lion.
nn) armored vehicle.
(o) chimp.
(p) maple tree
dle telaeaabers dhd phooe
powerr 41.23
Penen 2022
24.0
(q) dial telephone.
lawn
sunflower
(0 bee
Figure 11: Clustering CIFAR-100 into 20 clusters and relabeling them using our pipeline.
20

---

Published as a conference paper at ICLR 2024
(a) head cabbage
(b) parking meter
(c) manhole cover
(d) gibbon
Rhodesian ridgehack
(f) snorkel
(g) lorikeet
(h) keeshond
weevil
buckeye
guenon
geyser
(m) Urutu
(n) guacamole
(o) pill bottle
French hom
(q) entertainment center
(r) African grey
(s) EntleBucher
(t) rose hip
Figure 12: Clustering ImageNet (15k random samples from train split) into 200 clusters and relabeling them using our pipeline. (Randomly selected 20 clusters) Non-square figures represent that images within that cluster not enough fulfill the 8 x grid.
21

---

Published as a conference paper at ICLR 2024
(a) ballplayer
(b) joystick
(c) stove
(d) Giraffe
e motor scooter
!1 trailer truck
g groom
(h) church
(i) cellular phone
ii Police car
(k) Windsor tie
(l) Vegetable garden
(m) desktop computer
(n) surfboard
(0) Tennis racket
(p) Vase
(q) Fire extinguisher
(r) Sun umbrella
(S) soccer ball
(t) dock
Figure 13: Clustering COCO (30k random samples) into 150 clusters and labeling them using pipeline. (Randomly selected 20 clusters)
22

---

Published as a conference paper at ICLR 2024
(a) Dollhouse kit
(b) website
(c) seashore
(d) cocktail
(e) Loaf pan
Vegetable garden
needle fitting kit
(h) fountain
Dollhouse kit
ii) Quiltered tablecloth
(k) suit
gown
(m) accordion
(n) suit
(o) gown
(p) Quiltered tablecloth
(q) Egg
(r) Model
(8) mosque
(t) pickup truck
Figure 14: Clustering LAION-Aestheti into 300 clusters and labeling them using pipeline. (Randomly selected 20 clusters)
23


Reviews:
[{"review_id": "funIzBLHkq", "reviewer": "Reviewer_2xo4", "summary": "This paper studies the image clustering problem in the age of large pre-trained models. Specifically, this paper develops a method to determine the cluster number in a given dataset. Then, this paper validates that the features from large pretrained models, such as CLIP, help achive better custering accuracy than the traditional feature pre-training. Moreover, this paper also develops a self-labeling method to produce text labels for the clusters. Experiments on many image datasets, including ImageNet-1k, demonstrate the effectiveness of the proposed method.", "questions": "See the weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper achieves state-of-the-art results on many image datasets, including ImageNet-1k.", "weaknesses": "Three main technique contributions are developed in this paper, including a method to determine the cluster number, a validation that the features from CLIP push the limits of image clustering, and a self-labeling method to annotate the text-labels for the clusters. I have three concerns about these three technique contributions:\n\n- This paper seems did not discuss the existing methods to determine the cluster number and the difference among them. Do all the existing clustering methods not discuss how to determine the cluster number?\n- The proposed clustering method is a simple combination between CLIP features and MLC optimization method. I realize it is meaningful to validate the superiority of CLIP features in image clustering, but the technique contribution itself is kind of subtle.\n- A self-labeling method to annotate the text-labels for the clusters in *Algorithm 2* simply uses a cosine similarity metric to determine which texts are the closest ones given text candidates, which is a very simple solution. It does not meet my expectations that the proposed self-labeling method strongly relies on the pre-define text candidates. What if the text candidates are not given?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0njcXHDVQk", "reviewer": "Reviewer_hovR", "summary": "This paper proposed a novel method that leverages the rate reduction principle to learn to do image clustering using pretrained models. A technique for automatically select the optimal number of clusters is also proposed based on the same principle. Finally, a self-labeling mechism is proposed to label the clusters with semantic labels.\nExperiments show that the proposed method achieves a good performance, as well as give a good estimation of the optimal number of clusters.", "questions": "1. I would be interesting in how the method perform on fine-grained datasets.\n2. It would be better if the paper could include results of using other variant of CLIP models, such OpenCLIP.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper provides an alternative way of performing image clustering, which seems to be performing well and could be of interesting for the community.\n2. The method enables automatic estimation of the optimal number of clusters in a dataset, from the result the method seems to perform pretty well.", "weaknesses": "1. The main experiments are done on somewhat small datasets like CIFAR, or coarse grained dataset like COCO, the paper would be stronger if it could include finer-grained dataset for clustering like iNaturalist.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tl59uLr7IL", "reviewer": "Reviewer_5jgz", "summary": "The submission introduces a new image clustering pipeline named CPP, which leverages large pre-trained models like CLIP to efficiently and effectively cluster images, particularly on large-scale datasets. The authors propose to estimate the optimal number of clusters in a dataset and optimize the rate reduction objective using pre-trained features, resulting in a notable improvement in clustering accuracy (e.g., from 57% to 66% on ImageNet-1k). Furthermore, by utilizing CLIP's multimodal capabilities, a simple yet effective self-labeling algorithm is developed to generate meaningful text labels for the clusters. The pipeline demonstrates state-of-the-art performance across various standard datasets including CIFAR-10, CIFAR-100, and ImageNet-1k, and extends its applicability to datasets without predefined labels like LAION-Aesthetics and WikiArts.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Leveraging Large Pre-trained Models**: The integration of the powerful image encoder from CLIP into the clustering framework MLC significantly enhances the pipeline’s ability to process and analyze images, leading to state-of-the-art clustering performance on various datasets.\n\n2. **Improvement in Clustering Accuracy**: Through the optimization of the rate reduction objective using pre-trained features, the pipeline achieves a noticeable improvement in clustering accuracy, as demonstrated on ImageNet-1k.\n\n3. **Self-Labeling Algorithm**: The pipeline includes a simple yet effective self-labeling algorithm that leverages CLIP’s vision-text capabilities, resulting in semantically meaningful clusters that are comprehensible to humans.", "weaknesses": "1. **Limited Innovation in Methodology**: The main innovation of the proposed method seems to be centered around utilizing features extracted by CLIP for initialization, but there appears to be a lack of novelty in the algorithmic aspect of the approach.\n2. **Concerns about Stability and Sensitivity**: As depicted in Fig.4, the vicinity of the extreme points in model selection appears quite flat, raising concerns about the algorithm's stability and its sensitivity to perturbations, such as the choice of hyperparameters and network architecture.\n3. **Potential Information Leakage**: Given that CLIP has been trained on vast amounts of data, there is a suspicion of cluster/label information leakage in almost all of the experimental data (CIFAR and ImageNet) presented, which could potentially bias the quantitative results.\n4. **Lack of Adequate Metrics for Text Labeling**: While the automated text annotation aspect of the pipeline is interesting, it seems to be lacking appropriate metrics to adequately evaluate and validate its performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5zZJzVxJwM", "reviewer": "Reviewer_7qGL", "summary": "The paper presents a unified approach for integrating feature learning with clustering processes. Its novel method for selecting the optimal number of clusters enhances the practicality of KNN and similar clustering methods for large-scale data, even when using computationally intensive models like CLIP. Additionally, it achieves state-of-the-art results on various datasets.", "questions": "1- The complexity added by Equation 4 to the optimization process isn't clear. Is it possible for there to be multiple optimal values for K?\n2- Could a simpler method, like the elbow method, be used to determine K?\n3- A brief explanation of what \"more-structured representation\" means in the context of related works would be helpful.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1- SOTA results\n2- I find that enhancing KNN methods – whether by improving their scalability, representation, or explainability – is valuable. \n3- Good visual analysis Fig 3 and 4.", "weaknesses": "1- The paper's flow is somewhat challenging to follow.\n2- The acronym MLC is initially mentioned in the contributions section without prior definition, leading to initial confusion, though its relation to previous works becomes clearer later in the paper.\n3- While the method appears to be a practical extension of MLC, its level of novelty and contribution to the field is not distinctly evident.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FIuO7JwFu2", "reviewer": "Reviewer_DGyy", "summary": "This paper propose a new image clustering workflow that leverages the powerful feature representation capabilities of large pretrained models (like CLIP) to effectively and efficiently perform image clustering. It first develop a new algorithm to estimate the number of clusters in a given dataset. Through extensive experiments, the paper demonstrate that the workflow performs well on standard datasets.", "questions": "How does the choice of pre-trained models affect the clustering performance or the quality of labels?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel image clustering pipeline: The paper proposes a novel image clustering pipeline that leverages the powerful feature representation of large pre-trained models such as CLIP and cluster images effectively and efficiently at scale. The paper also develops a new algorithm to estimate the number of clusters in a given dataset, and a simple yet effective self-labeling algorithm that generates meaningful text labels for clusters.\n2. State-of-the-art performance: The paper demonstrates that the proposed pipeline achieves state-of-the-art clustering performance on standard datasets such as CIFAR-10, CIFAR-100, and ImageNet-1k. The paper also shows that the pipeline works well on datasets without predefined labels, such as WikiArt", "weaknesses": "1. Dependence on pre-trained models: The paper heavily relies on the pre-trained models such as CLIP to provide the initial feature representation and the text candidates for labeling. The paper does not explore how the choice of pre-trained models affects the clustering performance or the quality of labels. The paper also does not consider the potential biases or limitations of the pre-trained models, such as their data sources, or domains.\n2. Limited evaluation and comparison: The paper also does not report enough ablation studies or sensitivity analysis to show the impact of different components or hyperparameters of the pipeline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)