Customized Multiple Clustering via Multi-Modal Subspace Proxy Learning
Authors: Jiawei Yao
Qi Qian
Juhua Hu
Venue: NeurIPS 2024 poster (2024)
Keywords: Multiple Clustering
Multi-modal Model
Large Language Model
Primary area: machine_vision

Abstract: Multiple clustering aims to discover various latent structures of data from different aspects. Deep multiple clustering methods have achieved remarkable performance by exploiting complex patterns and relationships in data. However, existing works struggle to flexibly adapt to diverse user-specific needs in data grouping, which may require manual understanding of each clustering. To address these limitations, we introduce Multi-Sub, a novel end-to-end multiple clustering approach that incorporates a multi-modal subspace proxy learning framework in this work. Utilizing the synergistic capabilities of CLIP and GPT-4, Multi-Sub aligns textual prompts expressing user preferences with their corresponding visual representations. This is achieved by automatically generating proxy words from large language models that act as subspace bases, thus allowing for the customized representation of data in terms specific to the user’s interests. Our method consistently outperforms existing baselines across a broad set of datasets in visual multiple clustering tasks. Our code is available at https://github.com/Alexander-Yao/Multi-Sub.

Customized Multiple Clustering via Multi-Modal Subspace Proxy Learning
Jawwii Yao Qian Juhua Hult
1 School of Engineering and Technology, University of Washington, Tacoma, WA 98402, USA
Zoom Video Communications {juyao, juhuah) edu, qi w...anzzozo..u.
Abstract
Multiple clustering aims to discover various latent structures of data from different aspects. Deep multiple clustering methods have achieved remarkable performance by exploiting complex patterns and relationships in data. However, existing works struggle to flexibly adapt to diverse user-specific needs in data grouping, which may require manual understanding of each clustering. To address these limitations, introduce Multi-Sub, novel end-to-end multiple clustering approach that incor- porates a multi-modal subspace proxy learning framework in this work. Utilizing the synergistic capabilities of CLIP and GPT-4, Multi-Sub aligns textual prompts expressing user preferences with their corresponding visual representations. This is achieved by automatically generating proxy words from large language models that act as subspace bases, thus allowing for the customized representation of data terms specific the user's interests. Our method consistently outperforms existing baselines across a broad set of datasets in visual multiple clustering tasks. Our code is available at https://github..omm www.aooo//eeeaeaeee..... Yao/Multi-Suub
Introduction
Clustering is fundamental technique for analyzing data based on certain similarities, attracting extensive attention due the abundance of unlabeled data. Tra- ditional clustering methods [MaQuuen et al., 1967 Ng al. 2001 Bishop and Nasrabadi (2066) rely on general- purpose handcrafted features that may not suit specific tasks well. Deep clustering algorithms have improved clustering performance by employing
MultiSub
Which
group?
User
would like group by color.
Expert
include yallow green, orange, purple, blue.
Deep Neural Networks (DNNs) [Xie et al. 2016
have learned the embedding of each image the subspace spanned by reference worts.
Guérin and Boots) 2018 Qian et al. 2022
Qian
2023] to learn sask-specific features. However, most of these algorithms assume single partition of the data, while real data can clustered differently ac- cording different aspects, e.g., fruits in Fig. can be grouped differently by color or by species.
Figure 1:
The workflow of Multi-Sub that obtains desired clustering based on the sub- space spanned by reference words obtained from GPT-4 using users' high-level interest.
Multiple clustering algorithms [Bac and Bailey 2006 Hu al. 2017] address this challenge by producing multiple partitions of the data for various applications,
Work done while Alibaha Group. Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

showing the capability of discovering multiple clusterings from a single dataset. For instance, in e-commerce, products can be clustered by category for inventory management or by customer preferences for personalized recommendations. Recently, there has been a growing interest in incorporating deep learning techniques into multiple clustering. These techniques mainly use auto- encoders and data augmentation methods to extract a wide range of feature representations, which enhance the quality of multiple clustering MMiklautz et al., 2020 Ren et al. 2022 Yao et al. 2023].
For real-world applications, a key challenge for end users is efficiently identifying the desired clustering from multiple results based on their interests or application purposes. We observe that users are willing to indicate their interest using succinct keywords (e.g., color or species for fruits in Fig. [ However, difficult to use only a concise keyword to directly extract the corresponding image representations. Fortunately, the recent development of mulli-modal models like CLIP [Radford et al. 202T] that align images with their text descriptions can help bridge this gap. Nevertheless, unlike methods that can use labeled data to fine-tune pre-trained models [Gao et al. 2023, Wang al. 2023] to learn new task-specific representations, multiple clustering often faces scenarios with ambiguous unspecified label categories and quantities. Therefore, given only high-level concept from the user, intractable to fine-tune pre-trained models to capture a particular aspect of the data in an unsupervised manner. Very recently, Multi-MaP (Yao al. 2024] leverages CLIP to learn textual and image embeddings simultaneously that follow the user's s high-level textual concept. However, to achieve better performance, they require the user to provide contrastive concept that is different from the desired concept, which may not be feasible in many real real-world upplications. Moreover, they obtain the new representations first and then apply the traditional clustering method like k-means in a separate stage. This insufficient optimization lacking refinement between stages makes the clustering performance sub-optimal.
To mitigate these challenges, in this work, we first assume that the desired image and textual representations are residing in the same subspace according to the user's wer's specific concept. Thereafter, to capture the desired subspace better, we can ask low-cost experts like Google or large language models (LLMs) (e.g., GPT-4) for common categories under the desired concept, illustrated in Fig. [ Although those returned common categories may not directly capture the clustering targets, they can be applied as the subspace basis to help search the appropriate representations inside. More importantly, during the learning under the desired subspace, we also incorporate the clustering loss to learn the representations and obtain the clustering simultaneously, which significantly enhances the model's clustering performance and efficiency. The main contributions of this work can be summarized follows.
We present novel multiple clustering method, Multi-Sub, that can explicitly capture user's clustering interest by aligning the textual interest with the visual features of images. Concretely, we propose to learn the desired clustering proxy in the subspace spanned by the common categories under a user's interest.
Unlike most existing multiple clustering methods that require distinct stages for represen- tation learning and clustering, Multi-Sub can obtain both the desired representations and clustering simultaneously, which can significantly improve the clustering performance and efficiency.
Extensive experiments on all publicly available multiple clustering tasks empirically demon- strate the superiority of the proposed Multi-Sub, with a precise capturing of a user's interest.
2
Related Work
2.1 Multiple Clustering
Multiple clustering, methodology capable of unveiling alternative data perspectives, has garnered significant interest. Traditional approaches for multiple clustering [Hu and Pei 2018 employ shallow models to identify diverse data groupings. Some methods, such as COALA Bae and Bailey 2006] and [Qi and Davidson 2009), utilize constraints to generate alternative clusterings. Other techniques leverage distinct feature subspaces to produce multiple clusterings, exemplified by [Hu et al.) 2017 and MNMF (Yang and Zhang, 2017]. Information theory has also been applied to generate multiple clusterings, demonstrated by [Gondek and Hofmann 2003 and [Dang and Bailey 2010).

---

Recent advancements have seen the application of deep learning to discover multiple clusterings, yielding improved clustering performance. For instance. (Wei et al. 2020] proposed deep matrix factorization method that utilizes multi-view data to identity multiple clusterings. ENRC [Miklautz] at 2020) employs an auto-encoder to learn object features and optimizes a clustering objective function to find multiple clusterings. iMClusts (Ren et al. 2022 leverages auto-encoders and multi-head attention to learn features from various perspectives and discover multiple clusterings.
AugDMC Yao et al. 2023) uses data augmentation to generate diverse image aspects and learns representations to uncover multiple clusterings. DDMC (Yao and Hu, 2024] employs a variational Dopeetattion-axxiizzzenn
framework with disentangled representations to achieve superior clustering outcomes. However, almost all of these methods necessitate substantial user efforts to understand and select the uppropriate clustering for different application purposes. Recently, Multi-MaP Yao et al., 2024] leverages CLIP encoders to align a user's interest with visual data by learning representations close to the interested concept but far away from contrastive concept, significantly improving the efficiency of capturing user-desired clusterings. However, Multi-MaP requires the user to input contrastive concept for better performance, which is often not applicable. More importantly, it separates the representation learning and clustering as two distinct stages, which may result in sub-optimal performance. These issues will be mitigated in this work.
2.2 Multi-Modal Models
Multi-modaa learning involves acquiring representations from various input modalities like image, text, or speech. Here, we focus on how vision models benefit from natural language supervision. A key model in this area CLIP (Radford et al. 2021], which aligns images with their corresponding text using contrastive learning on a dataset of 400 million text-image pairs.
Fine-tuning adapts lision-language models, such as CLIP, for specific image recognition tasks. This is seen in CoOp [Zhou et al. 2022] and CLIP-Adapter [Gao et al., 2023). the latter using residual style feature blending to enhance performance. TeS [Wang et al.
2023 highlights the efficacy of fine-tuning in improving visual comprehension through natural language supervision. With limited labeled data, zero-shot learning has gained attention. Some approaches surpass CLIP by integrating
other large pre-trained models. For example, VisDesc [Menon and Vondrick [022] uses GPT-3 to generate contextual descriptions for class names, outperforming basic CLIP prompts. UPI. (Huang et al. 2022 and TPT (Shu et al. 2022) utilize unlabeled data to optimize text prompts. InMaP [Qian et al., 2024] and the online variant [Qian and Hu/ 2024) aid class proxies in vision space with text proxies. Recent advancements have significantly improved vision-language pre-training using large- scale noisy datasets. ALIGN [Jia et al. [021] employs over one billion image alt-text pairs without expensive filtering, showing that corpus scale can offset noise. Similarly, BLIP-2 (Li et al. 2023 uses a novel framework to bootstrap captions from noisy web data, enhancing both vision-lane vision-language understanding and generation tasks. While these methods strive to enhance the performance of vision classification tasks, clustering presents a distinct scenario where class names are not available to extract useful information from multi-modal information as in this work.
3
The Proposed Method
Given dataset of images {x:) and user-defined preferences for data grouping (such as color and species), our goal to generate clustering results that are specifically tailored to each preference. Thereafter, end users can directly use them for different application purposes without additional manual selection efforts. This process poses significant challenges, as it requires accurately aligning the complex, multi-dimensional data of images with the subjective and varied textual preferences of users. Traditional clustering methods often fail to capture these nuances, leading to a generic and less informative categorization for specific user applications.
Recently, the CLIP model [Radford al. [021]] facilitated more natural alignment between textual interests and visual representations. Our method, Multi-Sub, extends this alignment through a novel multi-modal subspace proxy learning approach. Fig. Tig-Zouthon outlines the overall framework of Multi-Sub, which is tailored to capture and respond to the diverse interests of users in clustering tasks. Multi-Sub employs two-phase iterative approach to align and cluster images based on user-defined preferences such color and species as described below.
3

---

Fruit dhilaee
Lser
am ouger see the color-based clustening results.
Expent
yellow, green......
Conceus color garemte the clustering of color with the eyferenee wonds are red (Pt yellow grece
Phase
Refferenue red yellow
Input Image
O
O
Prony
Image Enceder
OLL
Prompt
O
with codor
Text Encoder
Leamable Latest Factors
Optimize image projection layer
Aliazed Feauue Subspace
Cosine Objectivve
Priny Word Leaiing Sutopace
Phase II
Cosine similasit
Inpur
X2
Image
Proxy Word
Prompt
Encoder
fruii with color
Text
Claser Colors
Pacudo-lahela
Encouler
Optimize image
projection layer
Clustoring Loss
Figure 2: Multi-Sub framework. Multi-Sub framework, Phase I (Proxy Learning and Alignment) processes each image z; with user-defined textual prompts through partially learnable image encoder (with a learnable projection layer) and a frozen text encoder. The latent factor p, calculates weights {a, K based on the similarity to reference word embeddings {z, k which are then aggregated to form the proxy word embedding w, This proxy word embedding, combined with the image representation x,, establishes the Aligned Feature Subspace for better alignment between the text and image under the user's interest. In Phase II (Clustering), given the learned proxy word embeddings (w;) from Phase I to form pseudo-labels, the projection layer of the image encoder is further refined using the clustering loss. In Phase I, both the latent factor p and the projection layer learn 100 epochs, after which the projection layer further learns 10 epochs using the clustering loss in Phase II. This alternative process repeats until convergence.
3.1 Background: Multi-Modal Pre- Pre-Training in CLIP
Let {æi, } be set of image-text pairs, where z; denotes image and denotes its correspond- ing text description. We can obtain the vision and text representations of each pair by applying two encoders, f()) and h(.), x; f(x.) and t, h(tz). Both f() and h--) are encoders that optimize the vision and text representations, respectively, such that x; and t; are unit vectors. The primary goal during this pre-training phase minimize the contrastive loss, formulated
exp(x; (:/T) log exp(t, xi/T) min Σ log Σ, exp(x; t//T) E, exp(t, x;/T)
where 7 is a temperature parameter. The contrastive loss encourages the aligmment of the image and its description while penalizing the similarity of the image with irrelevant texts [Qian al. 2219)] The efficacy of this contrastive approach is vital for the subsequent phases of proxy word learning and fine-gu grained clustering. as it ensures that the foundational embeddings accurately reflect the inherent content and context of each modality.
3.2 Subspace Proxy Word Representation
We build upon the pre-trained image and text encoders from CL.IP and investigate whether we can leverage the image-text alignment to extract user-specific information. Specifically, given a fruit

---

image [Hu et al. 2017] as illustrated in Fig. 2 different users may have different interests of its attributes, such as color, species, etc. However, the pre-trained image encoder in CLIP can only produce single image embedding, which may not capture a user's interest exactly, not mentioning capturing different aspects. Furthermore, unlike classification tasks, clustering tasks do not come with concrete cluster names numbers. Therefore, we cannot directly use the pre-trained text encoder of CLIP to generate the corresponding text embedding.
To address these challenges, we propose a subspace proxy word learning method to learn new embedding under the preferred aspect provided by the user. Thereafter, the main challenge is, given only high-level concept like 'color' as in Fig. 2 how to effectively represent its subspace. Since the high-level concept itself cannot reflect different details under this concept in different images, it is difficult to do effective alignment between the high-level concept and images to figure out the corresponding vision subspace. Therefore, we propose to figure out the text subspace at first. Concretely, given pre-trained large language models like GPT-4 as low-cost experts, we can quickly gather common categories under a high-level concept using only one query like 'what are the common fruit colors' in Fig. 2 However, we cannot directly use the returned categories to do grouping, since they may not cover all existing categories in the data. Instead, we consider that most categories the data under this concept are residing in the same subspace as the returned ones. Therefore, we can apply suggested categories as basis or reference words in the subspace. Then, each image's 's category under the desired concept be represented by linear combination of these reference words.
Assuming GPT-4 provides K reference words as {= the proxy word of image I; can be calculated as
K
W: (t..()(z))
where ((x)) is the token embedding of reference word zς and {ax,* K are weights corresponding each reference word as basis. A higher weight 0g indicates that the image x; 's category is closer to the reference word zg- Here, we introduce trainable latent factor to learn the weight ag and can be calculated as
exp (p.zk)
ai,k
Σ3 exp (p.z,)
where n₁ = ((2)). Thereafter, w; is representing the token embedding of image xy's proxy word under the preferred user concept. Once p, is well obtained, the image's proxy word representation under the preferred user concept is also obtained. Next, we discuss how to learn p; using CL.IP.
3.3 Multi-Modal Subspace Proxy Learning
As mentioned above, CLIP's text and image encoders were learned by aligning the text prompt with its corresponding image. The standard text prompt of CLIP is designed as "a photo of fruit" for an image containing "fruit". Now, given user's preference "a fruit with the color of ** denoted by to for image Ft, where "*" is the placeholder for the unknown proxy word of image I; under concept 'color' and its token embedding w, can be formulated the linear superposition of reference words' token embeddings as discussed above.
color), we rewrite the prompt as
Thereafter, the prompt text embedding after the text encoder can be formulated as
((((()))))w))
To effectively learn p;, the trainable latent factors, we utilize the alignment capabilities of CLIP by adjusting these factors so that the weighted sum of reference word embeddings closely aligns with the visual representation of the image. This process involves iteratively adjusting p: to maximize the cosine similarity between the image's 's representation x; and its corresponding proxy word embedding w;. The optimization is conducted with the following loss function:
C(w.) - -((x=),,, (((())))(((()))))())))))))))))
(5)
5

---

It should be noted that this optimization procedure can be conducted with both the text encoder and image encoder frozen, which is very efficient. However, the image embedding extracted directly from the pre-trained image encoder may not reflect its representation under the desired user interest. Therefore, during the optimization procedure, we do freeze the text encoder but open the image encoder. Nevertheless, to preserve the strong capacity of the pre-trained image encoder in CLIP, we open only the projection layer of the image encoder, while its remaining parameters are frozen as shown in the Phass I' of Fig. 2
3.4 Clustering Loss
To enhance the clustering performance of Multi-Sub, in 'Phase II', we leverage pseudo-labels assigned using the currently learned proxy word embeddings [w,] and image embeddings [x,] from 'Phase r. Concretely, each image z; can be represented by the concatenation of its currently learned proxy word embedding w; and image embedding x;, denoted as v, = [w, The pseudo-label can be obtained by an offline k-means on {v;), which is however not efficient. Considering that proxy words for data points within the same cluster should show similar relationships to reference words, we obtain the pseudo-labels using the highest cosine similarity between the currently learned proxy word embeddings [w;] and the reference word embeddings [2x] ).
Given the pseudo-labels, the image embeddings be further optimized by opening only the projection layer of the image encoder for improved compactness and separability in clusters. This loss consists of two primary components: intra-cluster loss and inter-cluster loss, aimed at refining cluster cohesion and separation, respectively. It should be noted that to better represent each image under the desired user concept, we define the clustering loss over V; containing both textual and visual information.
Intra-cluster Loss: The intra-cluster loss is designed to minimize the distances between embeddings within the same cluster, encouraging cluster compactness. is calculated using the following formula:
Lintra
Σ - v,1
Nintra
i,jEintra
Here, ||v; - v2 is the squared Euclidean distance between embeddings x; and x; of data points i and within the same cluster, and Ninm denotes the number of intra-cluster pairs.
Inter-cluster Loss: This component aims to maximize the distances between embeddings from different clusters, thus enhancing separability. The inter-cluster loss is defined by a margin-based hinge loss as follows:
Cinter
1
Σ
max(0, m - ||v - v1
Ninter
i.j.inter
where max(0, m I|v; vyll) computes the hinge loss for each pair of embeddings from different clusters, ensuring minimum margin between them. Nimer is the count of inter-cluster pairs.
Total Loss: The overall clustering loss combines the intra- and inter-cluster losses, moderated by balancing factor A:
Lisall X Linta + (1 A) Linter
Optimizing this loss function in 'Phase II helps regularize the embedding space where clusters are both internally dense and well-separated from each other. It should be noted that in this phase we aim to learn a better projection layer only for the image encoder, while all others are fixed as shown in 'Phase II of Fig. 2
Previous methods often use a two-stage strategy that separates representation learning and clustering to simplify the optimization process. This separation, however, can lead to sub-optimal clustering results, since the learned representations may not be fully aligned with the clustering objective without refinement. In this work, we obtain both the proxy word and the clustering alternatively and simultaneously. Concretely, we first learn the proxy word in user pre-prrfereed subspace. Then, we fix the proxy word and refine the image encoder further to obtain better image representations using the clustering objective. These two phases are repeated alternatively until convergence, where "Phase learns 1 100 epochs and 'Phase II' learns 10 epochs each alternating according to the empirical experience summarized in Fig. 2

---

Table 1: Dataset Statistics.
Datasets
# Samples
# Hand-crafted features
# Clusters
Standford Cars
1,200
wheelbase length; body shape: color histogram
4:3
Card
8,029
symbol shapes; color distribution
13:4
CMUface
640
HOG; edge maps
4:20;2;4
Fruit
105
shape descriptors; color histogram
3;3
Fruit360
4,856
shape descriptors; color histogram
4;4
Flowers
1,600
petal shape; color histogram
4;4
CIFAR-10
60,000
edge detection; color histograms; shape descriptors
2:3
Table 2: Quantitative comparison. The significantly best results with 95% confidence are in bold.
Dutasse
Chataring
NMIT
MSC
R11
3041"
MCV
R11
NM1T
ENRC
R3"
IMClusts NMI
3041
AsgDMC
R1-
NML
DDMC
R21
NMH1
MuttiMaaPP
R11
356"
Multi-Su
RIT
036622
0.1627
0.733
1,0000
1,0000
1,000
Color
0.266
0.624
0.375
0.6791
0.4644
0.686
0.4197
0.6841
0.4544
0,7292
0.4551
0.7472
0,6239
03243
0.6644
0
Specka
02184
0.5885
0.2085
06676
04142
0.6964
0.3851
0.6332
0.3139
0,7430
0.5292
0.700
0.5244
0.2552
0,623
0.5594
Orde:
0.792
0.123
0.1225
03313
0.658
05257
0.155
0.5653
0.3921
Enstion
0.1284
0,6736
0.1433
0.5268
0.1592
0.6630
0,0422
0,5932
0.161
0.5367
0.172
0.3503
0.1786
0.7105
0,2653
0.5527
CMurce
Kenity
Gibes
111471
085922
0.7106
0.5774
0.4611
0.1201
86M4
0.004
01449
0.62%
0.7611
11 1979
0.5199
0.780
015627
0.107
08114
0.591
0A100
0.261
0.500
0.766
0.675
0.96
0.2068
0.4879
0.7611
6.9814
Pose
0.5687
0.3244
0.0038
02200
0.5129
0,4437
0.614
0.1320
0.557
0.453
0.4693
0.6634
0.5923
0.5736
Sonfoot Care
Color
01:311
0.1325
0,5336
016151
0.3103
0.163
015802
0.5634
0.365
0,2033
06775
0.6213
012516
0. 19653
016153
0.5643
0.2364
0.31%
0,7555
0.7356
01699
0.695
0.8765
0.7957
0.6355
0.730
0.8309
0MM95
0.753
0.6816
0.938T
0,5792
Color
0.2561
0.5965
0,2935
(15560
0.332
0.214
0.3169
0.027
0.355
0.6911
06327
0.7887
0.6426
1.2454
0,6939
0
Specias
0.326
0.5273
0.156
0.065
0.1894
0.195
0.1857
0.607
0.199
0.627
0.6488
0.332
0.503
0,6224
0.5719
CHRRR-19
Fasinement
0.147
0.136
0.3082
0.1015
0.1379
0.335
0.5346
0.1892
0.1536
0,3469
0,359
0,3140
0 11220
0.3655
0.3664
0.2655
0.3923
0.4516
0.4659
0,5782
0,3547
0.5827
0.8969
0.498
0.6757
0.5271
0.4628
0.7344
6,7096
4 Experiments
Datasets To demonstrate the effectiveness of Multi-Sub, we evaluate the proposed method on almost all publicly available visual datasets commonly used in multiple clustering tasks Yu et al.
2024 including Stanford Cars Yao et al. [2024]. Card Yao et al. [2023]. CMUface Günnemann et al.
2014 Flowers Yao et al. [2024), Fruit Hu et al. [2017] and Fruit360 Yao et al. 2023]. Stanford Cars contains two differentt clustering types, one for car color (c.g., red, blue, black) and one for car type (c.g., sedan, SUV, convertible), comprising 1,200 annotated car images. Card includes 8,029 images of playing cards, with two clustering types: one based rank (e.g., Ace, King, Queen) and another on suit (e.g., clubs, diamonds, hearts, spades). CMUfaae provides 640 facial images with clustering options for pose (e.g., front-facing, side-facing), identity, glasses (with/without), and emotion (e.g., happy. neutral, sad). Flowers comprises 1,600 flower images with two clustering types: one for color (e.g., red, blue, yellow) and another for species (e.g., iris, aster). Fruit includes 105 images of fruits with two clustering criteria: species (e.g., apples, bananas, grapes) and color (e.g., green, red, yellow). Fruit360, similar to the Fruit dataset, contains 4,856 images annotated for species (c.g., apple, banana, cherry) and color.
Additionally, created a multiple clustering dataset from CIFAR-10 Krizhevsky et al. 2009 by organizing the images into clusters based on type and environment. For type, the clusters are transportation and animals. For environment, the clusters are land, air, and water. The dataset characteristics about data size, handcrafted features, and cluster information are also summarized in T
should be noted that some data may face challenges in extraction of meaningful candidate categories from GPT-4, or their labels lack semantic features. Taking the identity clustering on the CMUface dataset Günnemann et al. [2014) as an example, different identities correspond to different individuals, and the names' semantic meanings should not affect clustering outcomes. In such cases, following the Multi-Map setting Yao et al. [2024), we randomly select 10 words from WordNet Fellbaum (0100) as reference categories.
Baselines We compare our Multi-Sui with seven state-of-the-arr multiple clustering methods. These methods are: MSC et al. [0077] is a traditional multiple clustering method that uses hand- crafted features to automatically find different feature subspace for different clusterings; MCV Guérin and Boots (018 leverages multiple pre-trained feature extractors as different views of the same data; ENRC Miklautz et al. (2020) integrates auto-encoder and clustering objective to generate different clusterings; iMClusts Ren et al. [2022] a deep multiple clustering method that leverages the expressive representational power of deep autoencoders and multi-head attention to generate
7

---

Table 3: Variants of CLIP. The significantly best results with 95% confidence are in bold.
Dataset
Clustering
Crl
Multi-Sub
NMI
RIf
NMP
RI1
NMI
RI1
Fruit
Color Species
0.7912
0,9075
0.8629
0,9780
0.9693
0.9964
0,9793
0,9919
1,0000
1,0000
1,0000
1,0000
Fruit360
Color
0.5613
0.7305
0.5746
0,7673
0.6654
0.8821
Species
0.4370
0.7552
0.5364
0.7631
0.6123
0.8504
Card
Order
0.3518
0.8458
0.3518
0.8458
0.3921
0.8842
Suits
0.2711
0.6123
0.2711
0.6123
0.3104
0.7941
Emotion
0.1576
0.6532
0.1590
0,6619
0.2053
0.8527
CMUface
Glass
0.2905
0.6869
0.4686
0.7505
0.4870
0.8324
Identity
0.1998
0.6388
0.2677
0.7545
0.7441
0.9834
Pose
0.4088
0.6473
0,4691
0.6409
0.5923
0.8736
Color
0.6539
0.8237
0.6830
0.8642
0.7533
0.9387
Stanford Cars
Type
0.6207
0.7931
0.6429
0.8456
0.6616
0.8792
Color
0.5653
0.7629
0.5828
0.7836
0.6940
0.8843
Flowers
Species
0,5620
0.7553
0,6019
0.7996
0.6724
0.8719
Type
0.4935
0.6741
0.5087
0.7102
0.5271
0.7394
CIFAR- Environment
0.4302
0.6507
0.4643
0.6801
0.4828
0.7096
multiple salient embedding matrices and multiple clusterings therein; AugDMC Yao et al. 2023
leverages data augmentations to automatically extract features related to different aspects of the data using a self-supervised prototye--bsee representation learning method; DDMC Yao and Hu [024]
combines disentangled representation learning with variational Expectation- Exppetation-Maxiiiztttoo (EM) framework; Multi-MaP Yao et al. (2024) relies on a contrastive user-defined concept to learn proxy better tailored to user's interest. It is worth noting that, in our experiments, we apply both traditional and deep learning baselines. Traditional methods rely on hand-crafted features, while deep learning methods directly utilize the original images as input.
Hyperparameter For each user's preference, we train the model for 1000 epochs using Adam optimizer with momentum of 0.9. We tune all the hyper-parameters based on the loss score of Multi-Sub, where the learning rate is selected from {le-1,5e-2,1e-2,55 1e, 1e-3,5e-4), weight decay is chosen from (5e-4,1e-4,55 1e-5, 0} for all the experiments. Most methods obtain each clustering by applying k-means Lloyd 1982 to the newly learned representations, while ours is end-to-end. The experiments are performed on four NVIDIA GeForce RTX 2080 Ti GPUs
Considering the randomness of k-means for those applicable baselines, we run k-means 10 times and report the average clustering performance using two metries, namely, Normalized Mutual Information (NMI) White et al. [2004] and Rand index (RI) Rand [971]. These metrics range from to I with higher value indicating better performance compared to the groundtruth.
Evaluation metrics
4.1 Performance Comparison
Table 2 reports the clustering results. During the clustering stage, after we obtain the proxy word embedding of each image for a desired concept, we can concatenate the image embedding and the token embedding of proxy word. The results show that Multi-Sub consistently outperforms the baselines, demonstrating the superiority of the proposed method. This also indicates strong generalization ability of the pre-trained model by CLIP, which can capture the features of data from different perspectives.
Our methodology uses the CLIP encoder and GPT-4 to derive clustering results, prompting an evaluation of their performance zero-shot manner. We introduce two zero-shot variants of CLIP: CLIPGrT and CLIPPabel CLIParT uses GPT-4 to generate candidate labels and performs zero-shot classification, while CLIP uses ground truth labels directly, providing an optimal setting. As shown in Table 3 CLIPbba generally outperforms CLIP due to its use of accurate labels, while CLIPGPT introduces noise. Both variants perform equally on the Card dataset as GPT-4's labels match the groundtruth. Multi-Sub surpasses CL.PPGPT and even outperforms CLIP in all cases, demonstrating its ability to capture user-interest--rat based data aspects and confirming its efficacy. This superiority can be attributed to Multi-Sub's 's proxy word learning mechanism, which automatically adjusts textual

---

Table 4: Comparison of differenttext encoders. The significantly best results with 95% confidence are in bold.
CLIP
ALIGN
BLIP
Dataset
Clustering
NMI
RI7
NMP
RI
NMI
RI
Fruit360
Color
0.6654
0.8821
0.7031
0,8925
0.6522
0.8814
Species
0.6123
0.8504
0.6426
0.8565
0.6254
0.8536
Order
0.3921
0.8842
0,4316
0,9922
0,3845
0,8359
Card
Suits
0.3104
0.7941
0.3226
0.8006
0.3151
0.7956
Emotion
0.2053
0.8527
0.2148
0.8553
0,2081
0.8335
CMUface
Glass
0.4870
0.8324
0.4951
0.8351
0.4951
0.8353
Identity
0.7441
0,9834
0.9828
0.6853
Pose
0.5923
0.8736
0.6137
0.8942
0,5732
0.8427
Standford Cars
Color
0.7533
0,9387
0.7624
0.8942
0.5732
0.8427
Type
0.6616
0,8792
0.6712
0.8865
0.6581
0.8731
Flowers
Color
0.694
0.8843
0.6925
0.8812
0.6843
0.8789
Species
0.6724
0.8719
0.6693
0.8691
0.6627
0. 0.8654
Type
0.5271
0.7394
0.5342
0.7456
0,5221
0.738I
CIFAR 10
Environment
0.4828
0.7096
0.4793
0.7064
0.4752
0.7038
Table 5: Ablation study of Multi-Sub. The results that achieved the highest and second highest performance for each clustering are indicated by boldface and underlined numerals, respectively.
clustering with ()) 11
clastering with 1
clustering with dtis,)
Datavet
Chustrrng
Subspace
image
text
concatorvat
5511
concotenato
concattrano
NMI1
RI
NMI
R11
NM1
R11
NM41
R011
NM11
141
NMN
R11
XMIT
0.3649
0.6546
0.-789
0.6607
0.5208
0.7281
0.4586
0.6331
0.4987
0.6933
0.4438
0.6382
0.4996
0.7059
Type
CIFAR-10
0.7211
0.4601
0.5093
0.6821
0.4271
0.6764
0.6531
0.6813
0.6131
0
0.4249
0.6337
0.4149
0.6062
0,6136
0.0691
0.4589
0.6816
Plarionment
0-215
0.6677
0.429
0.6531
0.440
0.56.1
0.4316
0.689
0.4563
0.678)
0.4364
0.6996
0.4514
0.0665
0.4320
0.6837
0.4507
0,6762
0.4656
0,6834
0.4218
0.6541
0.4432
0.6631
0.4586
0.6876
0,4828
0.7096
embeddings based on user-defined interests, creating more accurate proxy word embeddings. This approach reduces noise compared to CLIPGPT. which suffers from label mismatches. Additionally, Multi-Sub's 's iterative learning process refines these embeddings, optimizing alignment between text and image representations.
4.2 Ablation study
Different ways of constructing subspace The subspace of the proposed method can be expanded by different embeddings, i.e., the token embedding of the proxy word ((w:). ), the text embedding of the proxy word h ((:)),, and the text embedding of the prompt tū = (((((()) (4) [[:(w;;) These three kinds of embeddings can also be used to evaluate the clustering results in each case. In addition, we can use different combinations of learned embeddings (e.g., different concatenations of text and image embedding) the final embedding for clustering. The results are shown in Table 5 It can be seen that using word token embedding usually achieves better results. This is expected since the word proxy directly reflects the image's category under the desired concept. The token word embedding subspace is also aligning well with CLIP's training method. In contrast, prompt embedding performs the worst it introduces noise from user interest, dataset, and reference words, which are unnecessary for clustering. Additionally, most methods perform better when the same approach is used for constructing subspace and evaluating clustering results. Combining text and image embeddings generally enhances performance, capturing user interests from both aspects effectively.
(())(()))))
expected
Effect of text encoder Table compares the performance of three text encoders CLIP, ALIGN, and BLIP- various datasets. The results indicate that ALIGN generally outperforms CLIP and BLIP most tasks. This suggests that ALIGN's text encoder effectively captures and aligns textual and visual representations, enhancing clustering performance. ALIGN tends to excel in tasks that require distinguishing subtle visual differences influenced by textual descriptions, such as emotions and accessories in the CMUface dataset, and colors in the Fruit360 dataset. CLIP shows strong tendency in identity-related tasks and complex object categorization, as evidenced by its performance in the CMUface identity task and Standford Cars type clustering. BLIP, while competitive, seems to perform better in categorical distinctions rather than abstract attributes, performing relatively well in species-related tasks across various datasets. These findings underscore the importance of effective text embeddings in multi-modal clustering frameworks.
We conducted an additional analysis using the Maximum Mean Discrepancy (MMD) metric to quantify the differences in the feature spaces generated by different text encoders (i.e., CLIP, ALIGN, and BLIP) in Table 6 The MMD results indicate that although our text prompts are simple, the

---

feature spaces generated by different text encoders exhibit significant distributional differences. The effectiveness of a text encoder can vary depending on the specific clustering task. For example. ALIGN tends to excel in tasks with more abstract attributes, such as colors and emotions, while CLIP shows strong performance in identity-related tasks. This variability underscores the importance of selecting an appropriate text encoder based on the specific application requirements. The difference between text encoders may come from the different corresponding pre-training tasks and this will be an interesting future direction.
Table 6: MMD between different text encoders across datasets.
Dataset
Clustering
CLIP ALIGN
CLIP
BLIP
ALIGN BLIP
Color
0.234
0.211
Fruit360
Species
0.189
0.172
0.183
Order
0.215
0.202
0.219
Card
Suits
0.198
0.184
0.192
Emotion
0.276
0.245
0.263
Glass
0,231
0.217
0,225
CMUface
Identity
0.263
0.249
0.258
Pose
0.245
0.228
0.239
Color
0.238
0.223
0.231
Stanford Cars
Type
0.212
0.198
0.205
Color
0.257
0.244
0.252
Flowers
Species
0.248
0.231
0.242
Type
0.193
0.178
0.186
CIFAR-I0
Environment 0.178
0.162
0.174
Visualization To further demonstrate the ef-
fectiveness of Multi-Sub, we visualize the repre- sentations from CLIPlabet label. CLIP and Multi- Sub for color and species clustering tasks (Fig- ure 3) In species clustering, CLIPitbel shows clear boundaries using ground truth labels, while CLIPGPT introduces noise from reference words. Multi-Sub outperforms both by effectively cap- turing image features and user interests with proxy word embeddings. In color clustering, both CLIP and CLIPGPT focus on species features, resulting in less distinct clusters. Multi- Sub excels by clearly distinguishing colors, leveraging user-specific interests for improved aligmment. Overall, Multi-Sub consistently aligns embeddings with user interests, surpass- ing CLIP and CLIParr, demonstrating its robust multi-modal subspace proxy learning.
(a) CLiPted
CLiPeer
(c)
(d) Species add (c) Species CLIPGRT Species of Mulii Sub
Figure 3: Visualization of feature embeddings and related labels on Fruit dataset. For the visualization of color, red, green, and yellow points indicate the color of red, green, and yellow, respectively. For the visualization of species, red, yellow, and purple points indicate the species of apple, banana, and grapes, respectively.
5 Conclusion and Limitations
In conclusion, study mitigates an important challenge in multiple clustering: effectively identify- ing desired clustering results based on user interests or application purposes. We introduce Multi-Sub, novel approach that integrates user-defined preferences into customized multi-modal subspace proxy learning framework. By leveraging the synergy between CLIP and GPT-4, Multi-Sub automati- cally aligns textual prompts expressing user interests with corresponding visual representations. First, we observe reference words for user's interests from large language models. Given the absence of concrete class names in clustering tasks, our method uses these reference words to learn both text and vision embeddings tailored to user preferences. Extensive experiments across various visual multiple clustering tasks demonstrate that Multi-Sub consistently outperforms state-of-the-atd techniques.
However, our approach has certain limitations. The reliance on large language models like GPT-4 can introduce biases inherent in these models, potentially affecting the clustering outcomes. Additionally, the field of multiple clustering lacks large, diverse datasets, which limits comprehensive evaluation. Although we have annotated CIFAR-10, more extensive datasets needed.
10

---

Acknowledgment
Yao's research was funded in part by J.P. Morgan Chase Co and Advata Gift funding. Any views or opinions expressed herein are solely those of the authors listed, and may differ from the views and opinions expressed by J.P. Morgan Chase & Co. its affiliates. This material is product of the Research Department of J.P. Morgan Securities LLC. This material should not be construed as an individual recommendation for any particular client and intended as recommendation of particular securities, financial instruments or strategies for particular client. This material does not constitute solicitation offer in any jurisdiction.
References
E. Bae and J. Bailey. Coala: A novel approach for the extraction of an alternate clustering of high quality and high dissimilarity. ICDM, pages 53 62. IEEE, 2006.
C. M. Bishop and M. Nasrabadi. Pattern recognition and machine learning, volume Springer, 2006.
X. H. Dang and J. Bailey. Generation of alternative clusterings using the cami approach. In Proceedings of the 2010 SIAM International Conference on Data Mining, pages 118-129. SIAM, 2010.
C. Fellbaum. Wordnet. In Theory and applications of ontology: computer applications, pages 231-243, Springer, 2010.
Gao, Geng, R. Zhang, T. Ma, R. Fang, Zhang, H. Li, and Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, pages 1-15, 2023.
D. Gondek and Hofmann. Conditional information bottleneck clustering. In 3rd ieee international conference on data mining, workshop clustering large data sets, pages 36-42, 2003.
J. Guérin and B. Boots. Improving image clustering with multiple pretrained enn feature extractors. In British Machine Vision Conference 2018, BMVC 2018, 2018.
S. Günnemann, Färber, M. Rüdiger, and T. Seidl. Smve: semi-supervised multi-view clustering in subspace projections. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages
2014.
Hu and Pei. Subspace multi-clustering: review. Knowledge and information systems, 56(2): 257-284, 2018.
J. Hu, Q. Qian, J. Pei, R. Jin, and S. Zhu. Finding multiple stable clusterings. Knowledge and Information Systemx, 51(3):991-1021, 2017.
T. Huang, Chu, and F. Wei. Unsupervised prompt learning for vision-language models. arXiv preprint Stti:22004.064499 2022.
C. Jia, Y. Yang, Y. Xia, Y.TT. Chen, Z. Parekh, H. Pham, Q. Le, Y.-.. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4918 4916. 16. PMLR, 2021.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Li, D. Li. S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning, pages 19742. PMLR, 2023.
Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 28(2): 129-137, 1982.
J. MacQueen et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pages 281-297. Oakland, CA, USA, 1967.

---

S. Menon and C. Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07183, 2022.
L. Miklautz, D. Mautz, M. C. Altinigneli, C. Böhm, and C. Plant. Deep embedded non-redundant clustering. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 5174-5181, 2020.
A. Ng, M. Jordan, and Y. Weiss. On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems, 14, 2001.
Z. Qi and I. Davidson. A principled and flexible framework for finding alternative clusterings. In SIGKDD, pages 717-726, 2009.
Q. Qian. Stable cluster discrimination for deep clustering, In ICCV, pages 16599-16608. IEEE, 2023.
Q Qian and J. Hu. Online zero-shot classification with clip. In ECCV, 2024.
Q Qian, L. Shang, Sun, J. Hu, H. Li, and R. Jin. Softtriple loss: Deep metric learning without triplet sampling. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6450-6458, 2019.
Q. Qian, Y. Xu, Hu, H. Li, and R. Jin. Unsupervised visual representation learning by online constrained k- mens... In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16640-16649, 2022.
Q Qian, Y. Xu, and J. Hu. Intra-modal proxy learning for zero-shot visual categorization with clip. Advances in Neural Information Processing Systems, 36, 2024.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 9748-8763. PMLR, 2021.
W. M. Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association, 66(336):846-850, 1971.
L. Ren, G. Yu, J. Wang. L. Liu, C. Domeniconi, and X. Zhang. diversified attention model for interpretable multiple clusterings, IEEE Transactions Knowledge and Data Engineering, 2022.
M. Shu, W. Nie. D.-A. Huang, Z. Yu, Goldstein, A. Anandkumar, and C. Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems, 35:14274-14289, 2022.
Wang, Y. Xu, J. Hu, M. Yan, J. Sang, and Q. Qian. Improved visual fine-tuning with natural language supervision. In ICCV, pages 11865-11875. IEEE, 2023.
S. Wei, J. Wang, G. Yu, C. Domeniconi, and X. Zhang. Multi-view multiple clusterings using deep matrix factorization. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 6348-6344 0335,635,270 2020.
White, S. Steingold, and C. Fournelle. Performance metrics for group-detection algorithms. Proceedings of Interface, 2004, 2004.
Xie, R. Girshick, and A. Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pages 478-487, 2016.
S. Yang and L. Zhang. Non-redundant multiple clustering by nonnegative matrix factorization. Machine Learning, 106(5)669--712, 2017.
J. Yao and J. Hu. Dual-disentangled deep multiple clustering. In Proceedings of the 2024 SIAM International Conference on Data Mining (SDM), pages 679-687. SIAM, 2024.
Yao, F. Liu, M. Rashid, and J. Hu. Augdme: Data augmentation guided deep multiple clustering. Procedia Computer Science, 222:571-5800 2023.
12

---

J. Yao. Q. Qian, and J. Hu. Multi-modal proxy learning towards personalized visual multiple clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14066-14075, 2024.
G. Yu, L. Ren, J. Wang, C. Domeniconi, and X. Zhang. Multiple clusterings: Recent advances and perspectives. Computer Science Review, 52: 52:100621, 2024.
K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision- laaiunl models. International Journal of Computer Vision, (2179)2333-23448 2022.
13

---

A Appendix
A.1 Further Analysis
Parameters Analysis. To show the sensitivity of the balancing factor that is the only hyper- parameter our proposed method, the experiments were conducted on CIFAR-I0. We varied the value of from 0.1 to 1.0 in increments of 0.1 to observe its effect on the model's performance. As shown Fig. we can observe that different values of can work with our method and the optimal performance for "Type" & "Environment" clustering is achieved when À is set to 0.5, When À is too much on maximizing the distances between different clusters, which can
low, the model focuses lead to less cohesive clusters. Conversely, when À is too high, the model emphasizes compactness within clusters at the expense inter-cluster separation, leading less distinct clusters. Therefore, we set À to be 0.5 for all datasets, which confirms the robustness of our method.
0.75
(Tppl (lasineemer))
(linsinmmmn)
0,70
0.65
Score
0.60
0.55
0.50
0.1
0.2
0.3
0.4
0..
0.66
0.7
0.8
0.9
1.0
Balancing Factor
Figure 4: Sensitivity analysis of balancing factor À on CIFAR-10 dataset.
Model Adaptability. To evaluate how Multi-Sub adapts to new user demands not originally provided in the ground-truth of the dataset, we conducted an additional experiment using the Fruit dataset. Specifically, we introduced new demand based on the shape of the fruits, with the prompt set "fruit with the shape of ** We categorized the fruits into two shapes, that is, round and elongated. Although this specific demand may not be common in practical applications, it serves as an exploratory experiment test the adaptability of our method.
The results in Table 7 demonstrate that Multi-Sub successfully adapted to the new user demand of shape. The model learned to align the textual descriptions of shapes with the visual features, resulting clustering under the new subspace of shape.
Table 7: Clustering performance based on shape demand on the Fruit dataset.
Method
NMI
RI
MSC
0.553
0.762
MCV
0.586
0.787
ENRC
0.603
0.825
iMClusts
0.629
0.821
AugDMC
0.643
0.844
Multi-MaP
0.717
0.875
Multi-Sub (ours)
0.752
0.891
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims in the abstract and introduction are supported by detailed descriptions, empirical evaluations, and theoretical analysis provided in the body of the paper (Sections 1, 3, and 4).
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitation in Section 5, addressing potential biases
from large language models.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper specifies the datasets used, the training and testing details, the hyperparameters, and the evaluation metrics in Section 4, ensuring that the experiments can be reproduced.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The data and code are public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ prijss.cc/ubbliccguu.eee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides comprehensive details on the training and testing setups, including data splits, hyperparameters, and optimizer settings in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Since some baselines involve randomness using k-means, the paper did 95% significance test using 10 repeated results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides details on the computational resources used, including the
type of GPUs and the total amount of compute required for the experiments, mentioned in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https://neuripp: httppppps..ppppppeettttteeee itwcddddeeenndn
Answer: [Yes]
Justification: The research adheres to the NeurlPS Code of Ethics, ensuring transparency, reprodueibility. and consideration ethical implications throughout the study.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses the potential positive impacts of improving clustering techniques for various applications and mentions possible negative impacts such as biases introduced by large language models Section 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not involve the release of data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The paper properly credits the creators of the datasets and models used, and mentions the licenses and terms of use, detailed in Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The new datasets and code introduced the paper are well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "4M5Chj2Hft", "reviewer": "Reviewer_HfG7", "summary": "This paper introduces an end-to-end multi-clustering method that integrates a multimodal subspace proxy learning framework. It combines text prompts expressing user preferences with corresponding visual representations to achieve clustering based on user interests.", "questions": "The evaluation metrics mentioned in the paper require comparing results with ground truth values. How were the multiple clustering ground truth values in the dataset obtained? How is the accuracy of these ground truth values ensured?", "limitations": "The authors only address the limitations of their work and do not discuss broader impacts.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.The clustering task, driven by user interests, aligns better with user preferences and is more applicable to real-world scenarios.\n2.The experimental results are promising, and the methodology is clear and logical.", "weaknesses": "1.The contributions of the paper are not very clear. At first glance, it appears to merely combine CLIP and GPT, lacking innovative architecture.\n2.The baseline methods chosen for comparison are neither cited nor introduced.\n3.Section 5 discusses only limitations, lacking a discussion on broader impacts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **W1: The contributions of the paper.**\n\nThank you for the suggestion. We will carefully emphasize our contribution in the revision as follows:\n\nGiven only a high-level user interest in an unsupervised scenario without any class labels or names, we cannot directly apply CLIP. Instead, we must learn the unknown proxy word in a continuous space, which is challenging. The subspace approach proposed in this work aids effective learning. More importantly, our proposal can simultaneously optimize the proxy word and the clustering result. We will also emphasize this in our framework figure in the revision.\n\n---\n\n### **W2: The baseline methods chosen for comparison.**\n\nThank you for pointing this out. We will provide detailed descriptions in the revision as follows:\n\n1. **MSC** [2] is a traditional multiple clustering method that uses hand-crafted features to automatically find different feature subspaces for different clusterings.\n2. **MCV** [1] leverages multiple pre-trained feature extractors as different views of the same data.\n3. **ENRC** [3] integrates auto-encoder and clustering objective to generate different clusterings.\n4. **iMClusts** [4] is a deep multiple clustering method that leverages the expressive representational power of deep autoencoders and multi-head attention to generate multiple salient embedding matrices and multiple clusterings therein.\n5. **AugDMC** [6] leverages data augmentations to automatically extract features related to different aspects of the data using a self-supervised prototype-based representation learning method.\n6. **DDMC** [5] combines disentangled representation learning with a variational Expectation-Maximization (EM) framework.\n7. **Multi-MaP** [7] relies on a contrastive user-defined concept to learn a proxy better tailored to a user's interest.\n\n---\n\n### **W3: Broader impacts.** \n\nThank you for the suggestion. We will add the discussion that our proposal has potential in various applications like personalized marketing. For example, it can enhance advertisement effectiveness by tailoring clustering results to business preferences.\n\n---\n\n\n### **Question: How were the multiple clustering ground truth values in the dataset obtained?** \n\nAll multiple clustering datasets used in our experiments are public and come with pre-defined ground-truth labels in different aspects for evaluation that have been widely used. For datasets like Stanford Cars and CIFAR-10, the clustering ground truth labels are derived from their existing class labels. Below are the details of the datasets:\n\n1. **Stanford Cars**: This dataset contains 1,200 images of cars annotated with labels for both color and type (e.g., sedan, SUV).\n\n2. **Card**: This dataset includes 8,029 images of playing cards with labels for rank (Ace, King, Queen, etc.) and suits (clubs, diamonds, hearts, spades).\n\n3. **CMUface**: This dataset consists of 640 facial images with annotations for pose, identity, glasses, and emotions.\n\n4. **Flowers**: This dataset includes 1,600 flower images labeled by color and species (e.g., iris, aster).\n\n5. **Fruit**: This dataset comprises 105 images of fruits with labels for species (apples, bananas, grapes) and color (green, red, yellow).\n\n6. **Fruit360**: Similar to the Fruit dataset, Fruit360 contains 4,856 images of various fruits with detailed annotations for species and color.\n\n7. **CIFAR-10**: This dataset is structured to have 60,000 images clustered based on type (transportation, animals) and environment (land, air, water).\n\nThe data size, handcrafted features, and clusters for all datasets we used are also summarized in the table. It is worth noting that, in our experiments, we apply both traditional and deep learning baselines. Traditional methods rely on hand-crafted features, while deep learning methods directly utilize the original images as input.\n\n| **Datasets**    | **# Samples** |        **# Hand-crafted features**                  | **# Clusters** |\n| :-------------: | :-----------: | :--------------------------------------------: | :------------: |\n| Stanford Cars   |     1,200      | wheelbase length; body shape; color histogram  |      4;3       |\n| Card            |     8,029      | symbol shapes; color distribution              |     13;4       |\n| CMUface         |      640       | HOG; edge maps                                 |    4;20;2;4    |\n| Fruit           |      105       | shape descriptors; color histogram             |      3;3       |\n| Fruit360        |     4,856      | shape descriptors; color histogram             |      4;4       |\n| Flowers         |     1,600      | petal shape; color histogram                   |      4;4       |\n| CIFAR-10        |    60,000      | edge detection; color histograms; shape descriptors |    2;3      |\n\n---\n\n#### **References**\n[1]. J. Guérin and B. Boots. Improving image clustering with multiple pretrained cnn feature extractors.In British Machine Vision Conference 2018, BMVC 2018, 2018.\n\n[2]. J. Hu, Q. Qian, J. Pei, R. Jin, and S. Zhu. Finding multiple stable clusterings. Knowledge and Information Systems, 2017.\n\n[3]. L. Miklautz, D. Mautz, M. C. Altinigneli, C. Böhm, and C. Plant. Deep embedded non-redundant clustering. In Proceedings of the AAAI conference on artificial intelligence, 2020.\n\n[4]. L. Ren, G. Yu, J. Wang, L. Liu, C. Domeniconi, and X. Zhang. A diversified attention model for interpretable multiple clusterings. IEEE Transactions on Knowledge and Data Engineering, 2022.\n\n[5]. J. Yao and J. Hu. Dual-disentangled deep multiple clustering. In Proceedings of the 2024 SIAM International Conference on Data Mining (SDM), 2024.\n\n[6]. J. Yao, E. Liu, M. Rashid, and J. Hu. Augdmc: Data augmentation guided deep multiple clustering. In INNS DLIA@IJCNN, 2023.\n\n[7]. J. Yao, Q. Qian, and J. Hu. Multi-modal proxy learning towards personalized visual multiple clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dH43xtx4CS", "reviewer": "Reviewer_Evc4", "summary": "The paper is about Multiple Clustering, which is an interesting topic. The authors propose a novel end-to-end multiple clustering approach that incorporates a multi-modal subspace proxy learning framework. The paper is well written and well organized. However, there are several concerns in the current version of the paper that addressing them will increase the quality of this paper.", "questions": "Considering that the authors did not add an appendix, are there any other discussions or experiments?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 The authors' idea of using large models to aid clustering is novel.\n2 The paper is clearly structured and easy to understand.\n3 The paper has sufficient experiments to support its point of view.", "weaknesses": "1 The authors point out that different clustering results can be given for different customization needs of users. Then it will bring several associations (not necessarily accurate): a. What should be done if the user's demand is exactly opposite to the potential clustering distribution? b. The experiments do give different clustering results for different demand types, if the user proposes a new type of demand, can the model also adaptively adjust?\n2 Figure 2 is well drawn but could be further improved, some icons and fonts need to be adjusted.\n3 The authors point out that their model is capable of outputting clustering results directly, and then there should be a corresponding formula to represent this. In addition, it is hoped that the authors will discuss further why, if it is not a difficult task to output clustering results directly, few previous methods have done so.\n4 Authors should add details about the dataset, such as data size, feature types, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **W1: Several associations.** \n\nThanks for your insightful comments. Multi-Sub works by learning in the user-preferred subspace. Therefore, it is theoretically unlikely that the learned representations are completely opposite to the user's demand under such aligned subspace. We will make it clear in the revision.\n\nTo evaluate how Multi-Sub adapts to new user demands not originally provided in the ground-truth of the dataset, we conducted an additional experiment using the Fruit dataset. Specifically, we introduced a new demand based on the shape of the fruits, with the prompt set as \"fruit with the shape of *\". We categorized the fruits into two shapes, that is, round and elongated. Although this specific demand may not be common in practical applications, it serves as an exploratory experiment to test the adaptability of our method.\n\nThe following results demonstrate that Multi-Sub successfully adapted to the new user demand of shape. The model learned to align the textual descriptions of shapes with the visual features, resulting in a clustering under the new subspace of shape. Thanks for the suggestion and we will include this experiment in the revision.\n\n| **Method**        | **NMI** | **RI**  |\n| ----------------- | ------- | ------- |\n| MSC               | 0.553   | 0.762   |\n| MCV               | 0.586   | 0.787   |\n| ENRC              | 0.603   | 0.825   |\n| iMClusts          | 0.629   | 0.821   |\n| AugDMC            | 0.643   | 0.844   |\n| Multi-MaP         | 0.717   | 0.875   |\n| **Multi-Sub (ours)** | **0.752** | **0.891** |\n\n---\n\n### **W2: Figure 2.** \n\nThanks for pointing this out and we will carefully revise Figure 2 in the revision.\n\n---\n\n### **W3: Clustering Loss.**\n\nThanks for the suggestion and we will make the following clear in the revision. Previous methods often use a two-stage process that they separate representation learning and clustering to simplify the optimization process. This separation, however, can lead to sub-optimal clustering results, as the learned representations may not be fully aligned with the clustering objective. Instead, we obtain both the proxy word and the clustering alternatively and simultaneously. Concretely, we first learn the proxy word in a user-preferred subspace (i.e., Eqn. (5) in the submission). Then, we fix the proxy word and open the image encoder to obtain better image representations considering the clustering objective (i.e., the clustering loss at line 240 in the submission). These two steps are repeated alternatively until convergence.\n\n\n---\n\n### **W4: Datasets.**\n\nThanks for the suggestion. We will make the following descriptions clear in the revision.\n\n1. **Stanford Cars**: This dataset contains 1,200 images of cars annotated with labels for both color and type (e.g., sedan, SUV).\n2. **Card**: This dataset includes 8,029 images of playing cards with labels for rank (Ace, King, Queen, etc.) and suits (clubs, diamonds, hearts, spades).\n3. **CMUface**: This dataset consists of 640 facial images with annotations for pose, identity, glasses, and emotions.\n4. **Flowers**: This dataset includes 1,600 flower images labeled by color and species (e.g., iris, aster).\n5. **Fruit**: This dataset comprises 105 images of fruits with labels for species (apples, bananas, grapes) and color (green, red, yellow).\n6. **Fruit360**: Similar to the Fruit dataset, Fruit360 contains 4,856 images of various fruits with detailed annotations for species and color.\n7. **CIFAR-10**: This dataset is structured to have 60,000 images clustered based on type (transportation, animals) and environment (land, air, water).\n\nThe data size, handcrafted features, and clusters for all datasets we used are also summarized in the following table. It is worth noting that, in our experiments, we apply both traditional and deep learning baselines. Traditional methods rely on hand-crafted features, while deep learning methods directly utilize the original images as input.\n\n| **Datasets**    | **# Samples** |        **# Hand-crafted features**                  | **# Clusters** |\n| :-------------: | :-----------: | :--------------------------------------------: | :------------: |\n| Stanford Cars   |     1,200      | wheelbase length; body shape; color histogram  |      4;3       |\n| Card            |     8,029      | symbol shapes; color distribution              |     13;4       |\n| CMUface         |      640       | HOG; edge maps                                 |    4;20;2;4    |\n| Fruit           |      105       | shape descriptors; color histogram             |      3;3       |\n| Fruit360        |     4,856      | shape descriptors; color histogram             |      4;4       |\n| Flowers         |     1,600      | petal shape; color histogram                   |      4;4       |\n| CIFAR-10        |    60,000      | edge detection; color histograms; shape descriptors |    2;3      |\n\n---\n\n### **Question.** \n\nWe do have some additional experiments as in the **one-page PDF** that we will add in the revision. Those experiments include:\n\n- Clustering performance based on shape demand on the Fruit dataset, demonstrating the adaptability of Multi-Sub for new demands not existing in ground-truth clusterings provided by the data.\n- Sensitivity analysis of the balancing factor $\\lambda$ on the CIFAR-10 dataset, indicating a best value of 0.5, which is expected since when $\\lambda$ is too low, the model focuses too much on maximizing the distances between different clusters, which can lead to less cohesive clusters. Conversely, when $\\lambda$ is too high, the model emphasizes compactness within clusters at the expense of inter-cluster separation, leading to less distinct clusters. Therefore, 0.5 is used for all datasets.\n- Quantifying feature space differences generated by different text encoders using the Maximum Mean Discrepancy (MMD) metric, showing that even with simple text prompts in our work, different pre-trained text encoders vary in their abilities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oeQFctyi24", "reviewer": "Reviewer_vBFT", "summary": "This paper presents an innovative approach for addressing the limitations of existing multiple clustering methods. By leveraging the synergistic capabilities of CLIP and GPT-4, Multi-Sub aligns textual prompts with visual representations to cater to diverse user-specific clustering needs. This method introduces a novel multi-modal subspace proxy learning framework, which automatically generates proxy words from large language models to represent data in terms specific to user interests. The experimental results demonstrate that Multi-Sub consistently outperforms existing baselines across various datasets. Overall, I believe this paper makes a substantial contribution to the field of deep clustering and holds significant practical application value.", "questions": "1. Although the paper mentions the hyperparameters used, a more detailed analysis and discussion on the sensitivity of the method to these parameters would be beneficial.\n2. Given the method's iterative nature and the use of large models, there is a risk of overfitting, especially on smaller datasets. I am curious whether regularization techniques were used to address this issue?\n3. Table 3 compares the impact of different text encoders on performance. Clearly, there are significant performance differences when using different encoders, and the authors have indeed analyzed this issue. However, I believe the reasons behind this phenomenon could be explored in depth. Intuitively, given that the input text is quite simple, the overall performance should not be particularly sensitive to the choice of text encoder.", "limitations": "The limitations are discussed in the paper by the authors. There is no potential negative societal impact", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper offers several notable strengths that contribute to its overall impact and significance in the field of multiple clustering: \n1.\tThe integration of CLIP and GPT-4 for multi-modal subspace proxy learning is novel and effectively addresses the limitations of traditional multiple clustering methods.\n2.\tMulti-Sub excels in capturing and responding to diverse user interests, providing tailored clustering results without requiring extensive manual interpretation. Moreover, the performance gains come at a low cost and seem relatively easy to achieve. \n3.\tThe writing is clear and easy to follow. The figures are well-drawn, allowing for a quick understanding of the research motivation and methodological design.\n4.\tExtensive experiments on a wide range of publicly available datasets demonstrate the robustness and generalizability of the proposed method.", "weaknesses": "Despite its strengths, there are some areas where the paper could be improved to enhance its clarity and applicability:\n1. Although the paper mentions the hyperparameters used, a more detailed analysis and discussion on the sensitivity of the method to these parameters would be beneficial.\n2. Given the method's iterative nature and the use of large models, there is a risk of overfitting, especially on smaller datasets. I am curious whether regularization techniques were used to address this issue?\n3. Table 3 compares the impact of different text encoders on performance. Clearly, there are significant performance differences when using different encoders, and the authors have indeed analyzed this issue. However, I believe the reasons behind this phenomenon could be explored in depth. Intuitively, given that the input text is quite simple, the overall performance should not be particularly sensitive to the choice of text encoder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **W1:  A more detailed analysis and discussion on the sensitivity of the method to these parameters.**\n\nWe greatly appreciate your suggestion. To show the sensitivity of the balancing factor $\\lambda$ that is the only hyper-parameter in our proposal, the experiments were conducted on CIFAR-10. We varied the value of $\\lambda$ from 0.1 to 1.0 in increments of 0.1 to observe its effect on the model's performance. As shown in **Figure 1** of the one-page PDF, we can observe that different values of $\\lambda$ can work with our method and the optimal performance for \"Type\" \\& \"Environment\" clustering is achieved when $\\lambda$ is set to 0.5. When $\\lambda$ is too low, the model focuses too much on maximizing the distances between different clusters, which can lead to less cohesive clusters. Conversely, when $\\lambda$ is too high, the model emphasizes compactness within clusters at the expense of inter-cluster separation, leading to less distinct clusters. Therefore, we set $\\lambda$ to be 0.5 for all datasets, which confirms the robustness of our method.\n\n---\n\n### **W2: Whether regularization techniques were used to address overfitting?**\n\nWe would like to clarify that the primary objective of our method is clustering rather than supervised learning where overfitting is a more prevalent issue. Specifically, clustering focuses on finding inherent patterns and structures in the data rather than fitting to specific target labels. Therefore, the regularization in clustering helps constrain the data subspace to observe data groups rather than alleviate the overfitting of a model. In this work, we have subspace under a user's preference in the proposed method for regularization. We will make this clear in the revision.\n\n---\n\n### **W3: There are significant performance differences when using different encoders, this phenomenon could be explored in depth.** \n\nThanks for your insightful comments. We conducted an additional analysis using the Maximum Mean Discrepancy (MMD) metric to quantify the differences in the feature spaces generated by different text encoders (i.e., CLIP, ALIGN, and BLIP).\n\n| **Dataset**       | **Clustering** | **CLIP vs. ALIGN** | **CLIP vs. BLIP** | **ALIGN vs. BLIP** |\n| ----------------- | :------------: | :----------------: | :---------------: | :----------------: |\n| **Fruit360**      | Color          | 0.234              | 0.198             | 0.211              |\n|                   | Species        | 0.189              | 0.172             | 0.183              |\n| **Card**          | Order          | 0.215              | 0.202             | 0.219              |\n|                   | Suits          | 0.198              | 0.184             | 0.192              |\n| **CMUface**       | Emotion        | 0.276              | 0.245             | 0.263              |\n|                   | Glass          | 0.231              | 0.217             | 0.225              |\n|                   | Identity       | 0.263              | 0.249             | 0.258              |\n|                   | Pose           | 0.245              | 0.228             | 0.239              |\n| **Stanford Cars** | Color          | 0.238              | 0.223             | 0.231              |\n|                   | Type           | 0.212              | 0.198             | 0.205              |\n| **Flowers**       | Color          | 0.257              | 0.244             | 0.252              |\n|                   | Species        | 0.248              | 0.231             | 0.242              |\n| **CIFAR-10**      | Type           | 0.193              | 0.178             | 0.186              |\n|                   | Environment    | 0.178              | 0.162             | 0.174              |\n\nThe MMD results indicate that although our text prompts are simple, the feature spaces generated by different text encoders exhibit significant distributional differences.\n\nThe effectiveness of a text encoder can vary depending on the specific clustering task. For example, ALIGN tends to excel in tasks with more abstract attributes, such as colors and emotions, while CLIP shows strong performance in identity-related tasks. This variability underscores the importance of selecting an appropriate text encoder based on the specific application requirements. The difference between text encoders may come from the different corresponding pre-training tasks and this will be an interesting future direction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oVi8tszObZ", "reviewer": "Reviewer_r2bm", "summary": "This paper incorporates a multi-model subspace proxy learning (Multi-Sub) to design a novel end-to-end multiple clustering approach and utilizes the synergistic capabilities of CLIP and GPT-4 to align textual prompts expressing user preferences with corresponding visual representations. The main contributions of Multi-Sub can be summarized as follows:\n1. Capturing user’s clustering interest: Existing works struggle to adapt to diverse user-specific needs in data grouping. To overcome this limitation, Multi-Sub explicitly captures a user’s clustering interest by learning the desired clustering proxy under a user’s interest and aligning textual interest with corresponding visual features.\n2. Simultaneous optimized framework: The previous methods separated the representation learning and clustering stages. Different from them, Multi-Sub obtains both the desired representations and clustering simultaneously, which significantly improve the clustering performance and efficiency.\n3. Extensive experimental validation: Extensive experiments on all public multiple clustering tasks demonstrate that Multi-Sub outperform other methods. Moreover, a series of ablation studies further verify the effectiveness of Multi-Sub.", "questions": "Please refer to the Weaknesses.", "limitations": "The authors have discussed social impacts and limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. In real world, data may have multiple aspects that they can be grouped into different clusters. However, existing methods solely consider a single partition. So, it is meaningful to propose an effective algorithm to overcome this problem.\n2. The authors leveraged large language models (LLMs), including GPT-4 and CLIP, to align image and textual representations in the same subspace. Then, multi-modal subspace proxy learning is introduced to allow for the customized representation of data in terms specific to the user’s interests.\n3. Experimental results on public datasets show that the Multi-Sub method has a significant improvement, indicating the effectiveness of the propose method.", "weaknesses": "1. To change the two-stage learning approach of previous works, Multi-Sub aims to learn representation and clustering simultaneously. However, Multi-Sub employs a two-phase iterative approach to align and cluster images in training process, including (1) Phase I: Learning and Alignment; (2) Phase II: Clustering and Optimization. I wonder if this is another form of two-stage task.\n2. The description of Clustering Loss is not very clear in Section 3.4, how to determine that samples belong to the same class? By pseudo-label? Where did the pseudo-label come from?\n3. In this paper, the authors introduced large language models (LLMs) to learn representations and bridge the gap of textual and image features. But does the direct use of a pre-trained large language model introduce a priori information about the category, which can lead to unsupervised scenarios being corrupted?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **W1: I wonder if this is another form of two-stage task.**\n\nThanks for your invaluable feedback. We will make it clear in the revision as follows. A two-stage process used by previous methods separates the representation learning and clustering entirely, where the representation learning is fully completed at first. This separation, however, can lead to sub-optimal clustering results, as the learned representations may not be fully aligned with the clustering objective. Instead, we obtain both the proxy word and the clustering alternatively. Concretely, we first learn the proxy word in a user-preferred subspace (i.e., **Eqn. (5)** in the submission). Then, we fix the proxy word and open the image encoder to obtain better image representations considering the clustering objective (i.e., the clustering loss at **line 240** in the submission). These two steps are repeated until convergence. Therefore, these two components can improve each other iteratively, which shows better performance than the separated two-stage strategy in previous methods.\n\n---\n\n### **W2: The description of Clustering Loss is not very clear in Section 3.4, how to determine that samples belong to the same class? By pseudo-label? Where did the pseudo-label come from?**\n\nThanks for pointing this out. During the clustering, the proxy word is fixed, which can be used to obtain the pseudo-labels to determine the inter-cluster and intra-cluster relationships. Concretely, an offline clustering algorithm (e.g., k-means) can be applied to the currently fixed proxy words to group the samples as the pseudo-labels. We will clarify it in the subsequent versions.\n\n---\n\n### **W3: Does the direct use of a pre-trained large language model introduce a priori information about the category, which can lead to unsupervised scenarios being corrupted?**\n\nThanks for your insightful comments. It should be noted that although the category information released by the LLM can help determine a better subspace for the proxy learning, it is hard to cover all ground-truth labels, especially for user-specific domains as in this work. The provided categories by the LLM can be too narrowed, too expansive, or without any overlapping. Therefore, the corresponding categories existing in the data are unknown or uncertain. Moreover, the exact label for each instance is still unknown as in most unsupervised scenarios. We will make it clear in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We sincerely thank your invaluable time and efforts invested in reviewing our submission. Your constructive and insightful feedback are greatly appreciated for improving our revision.

We have carefully responded to all the questions and concerns raised in the individual rebuttal sections. Additionally, we have a one-page PDF that includes all the related tables/figures. Please kindly check if necessary.

Thank you once again for your insightful comments and suggestions, which have greatly contributed to the improvement of our work.

Best regards,

Authors

Meta-review: This paper received 4 reviews from experts in the field. The paper received the following reviews:  1 Accept, 1 Weak Accept and 2 Borderline Accepts. The paper has clearly positive reviews and the decision is to accept. Please incorporate reviewer comments in your final camera ready submission.
Decision: Accept (poster)